OpenClaw Firecrawl Setup

What Is Firecrawl?

Firecrawl is a web scraping API built for AI agents. It loads a page in a real browser, waits for JavaScript to render, strips the ads and navigation, and hands back the article body as markdown — or as JSON matching a schema you describe in plain language.

The reason it matters for an agent is the shape of the output. A raw HTML fetch gives your model a wall of markup that eats context and buries the answer. Firecrawl gives it the part that carries meaning. It is open source (AGPL-3.0) and one of the most-starred AI infrastructure projects on GitHub.

Firecrawl vs Search APIs

Firecrawl overlaps with search tools but solves a different problem. Search answers “which pages exist about X?” Firecrawl answers “what does this page actually say?” — and, since the search endpoint was added, it can do both in one call by searching and then scraping the top results.

A common pattern is to run both: a search API to discover candidate pages, then Firecrawl to read the best ones properly. See the web search guide for how the search providers compare.

Step 1: Get a Firecrawl API Key

1. Sign up at firecrawl.dev
2. Open the API Keys section of your Firecrawl dashboard
3. Copy the key — it starts with fc-

The free tier includes 1,000 credits per month, which covers roughly 1,000 scraped pages plus a separate allowance of search credits. That is enough for personal agent use. Paid tiers start at the Hobby plan (around $16/month billed yearly) and scale up from there — check Firecrawl's pricing page for current numbers.

Step 2: Connect Firecrawl on the Integrations Page

Firecrawl is available in the OpenClaw Launch integrations catalog, delivered through Composio. This is the supported path — you do not edit openclaw.json and you do not install a scraping skill by hand.

1. Sign in and open openclawlaunch.com/integrations
2. Search for Firecrawl and click Connect
3. A small form appears asking for two fields. The Base URL is pre-filled with https://api.firecrawl.dev/v1 — leave it as-is unless you run a self-hosted Firecrawl. Paste your fc- key into the API Key field.
4. Submit. There is no OAuth redirect for Firecrawl — the connection activates immediately and the card flips to connected.

Step 3: What Your Agent Can Now Call

Connecting adds the Composio tool bridge to your running bots. Your agent discovers the available actions itself, so you never have to memorize action names, but these are the Firecrawl tools it gets:

Crawls are asynchronous: the agent starts a job, then polls its status until the pages come back. That is normal, and it is why a whole-site crawl takes longer than a single scrape.

Step 4: Test It From Chat

Message your bot on whichever channel you already use — Telegram, Discord, WhatsApp, or the web chat. Firecrawl is a tool, not a channel, so nothing about your messaging setup changes.

• “Read https://example.com/blog/post and give me the key points.”
• “Pull the pricing table from competitor.com/pricing as JSON with plan name, price, and limits.”
• “Map docs.example.com and list every page under /api.”
• “Crawl our changelog and summarize what shipped this month.”

If the agent says it has no scraping tool, ask it to list its connected apps — that forces it to re-read the bridge instead of guessing from memory.

Where Your Key Lives

The Firecrawl key you paste is stored by Composio against your connection, not written into your container. Bots call Firecrawl through an authenticated proxy, so the credential is not sitting in a config file inside the instance. You can revoke it at any time by disconnecting Firecrawl on the Integrations page, or by rotating the key in your Firecrawl dashboard.

Firecrawl is open source, so you can also run it yourself — the self-hosted Firecrawl guide covers that setup. Point the Base URL field at your own deployment and connect as normal. Note that this removes Firecrawl-the-service from the picture, not the broker: whatever credential you paste into the connect form is still stored by Composio. If you want no third party holding a credential at all, configure your self-hosted endpoint directly on the instance instead of connecting it here.

Use Cases

• Grounding a knowledge base — scrape docs and product pages into markdown your agent can hold onto instead of guessing from training data
• Competitive tracking — extract pricing, features, and changelog entries from rival sites on a schedule
• Structured collection — describe the fields you want and get consistent JSON back, without writing CSS selectors that break on every redesign
• Documentation ingestion — crawl a whole docs site and have the agent answer questions against it
• Reading long pages cheaply — clean extraction cuts the token cost of a page dramatically versus dumping raw HTML into the model

Alternatives Worth Knowing

Firecrawl is a hosted service with a metered key. If you would rather not meter anything, Crawl4AI is an open-source scraper you run yourself, and the browser automation path drives a real browser when a page needs clicking, scrolling, or a logged-in session that a scraping API cannot reach.

Troubleshooting

The Connect Form Rejects the Key

• Check the key starts with fc- and was copied whole — a trailing space is the usual culprit
• Leave the Base URL at the default unless you are pointing at your own Firecrawl instance; a wrong path there fails the handshake
• If your Firecrawl account is brand new, confirm the key is active in the Firecrawl dashboard first

Blocked by Anti-Bot Protection

Firecrawl handles most JavaScript rendering and bot challenges on its own, but some sites still refuse. When that happens:

• Try a more specific URL — article pages are often reachable when a listing page is not
• Fall back to browser automation, which can handle interactive challenges
• Accept that some sites are simply closed to automated reading; Firecrawl respects robots.txt

Crawls Take a Long Time

• Scope the crawl to a section rather than the whole domain
• Use the map tool first to see how big a site is before crawling it
• Single-page scrapes return in seconds; crawl time scales with page count

Out of Credits

• Check remaining credits in your Firecrawl dashboard — the quota is on your key, not on your OpenClaw Launch plan
• Ask the agent to scrape specific pages rather than crawling a site when you only need a few
• Upgrade the Firecrawl plan if agent usage has outgrown the free tier

Frequently Asked Questions

Do I need to edit any config file?

No. Connecting Firecrawl on the Integrations page is the whole setup — bots already running pick the connection up on their next call. There is no JSON to edit and no skill to install by hand.

Does Firecrawl replace web search?

It can cover both now — the search tool searches and scrapes in one step — but a dedicated search provider still gives broader discovery. Firecrawl's strength is reading a page properly once you know which page matters. See the web search guide for the full picture.

Is Firecrawl free?

Firecrawl's free tier includes 1,000 credits per month plus a separate search allowance. Connecting it on OpenClaw Launch costs nothing extra — you bring your own Firecrawl key and its quota is billed by Firecrawl, not by us.

Can Firecrawl get past paywalls or logins?

No. It reads publicly accessible content and respects robots.txt. For anything behind authentication, use browser automation with your own credentials.

Does this work on Hermes Agent too?

Yes. The integrations bridge covers both frameworks, so a Hermes bot on your account gets the same Firecrawl tools from the same connection.

What's Next?

• Integrations — connect Firecrawl and the rest of the catalog
• Web search guide — pair discovery with Firecrawl's extraction
• Crawl4AI — the self-hosted, unmetered alternative
• Browser automation — for pages that need real interaction
• See pricing — managed hosting for your agent, starting at $3/month