OpenClaw Firecrawl Setup

What Is Firecrawl?

Firecrawl is a web scraping API that converts any webpage into clean, LLM-ready content. Unlike search APIs that find pages for you, Firecrawl extracts data from specific URLs you already know about — pulling out the text, tables, and structured data while stripping away ads, navigation, and other noise.

Think of it this way: search APIs answer “what pages exist about X?” while Firecrawl answers “what does this specific page say?” It's not a search engine — it's a content extraction tool that gives your AI agent the ability to read and understand any webpage on the internet.

Firecrawl vs Search APIs

Firecrawl serves a fundamentally different purpose than search APIs. Here's how they compare:

The key takeaway: use search APIs to find pages, use Firecrawl to extract content from pages you already have. They complement each other — many agents use both Tavily for discovery and Firecrawl for deep extraction.

Step 1: Get a Firecrawl API Key

1. Go to firecrawl.dev and create an account
2. Navigate to the API Keys section in your dashboard
3. Copy your API key (starts with fc-)

Firecrawl offers a free tier with 500 pages per month, which is enough for personal use. Paid plans start at $19/month for higher limits and advanced features like batch crawling.

Step 2: Install the Firecrawl Skill

Install the firecrawl skill from ClawHub:

/skill install @anthropic/firecrawl

The skill will prompt you for your Firecrawl API key during installation. Enter the key you copied from the Firecrawl dashboard.

Alternatively, you can configure the API key manually in your openclaw.json:

{
  "skills": {
    "firecrawl": {
      "apiKey": "fc-your-api-key-here"
    }
  }
}

Step 3: Test It

After installation, test the Firecrawl skill by asking your agent to extract content from a specific URL:

• “Extract the pricing table from competitor.com/pricing”
• “Scrape the documentation at docs.example.com/api-reference”
• “Get the main content from this article: https://example.com/blog/post”

Your agent will use Firecrawl to fetch the page, strip away the noise, and return clean, structured content. The results come back as well-formatted markdown that your agent can immediately reason about.

Use Cases

Firecrawl unlocks powerful capabilities for your OpenClaw agent:

• Web scraping — Extract content from any webpage, even JavaScript-rendered pages that simple HTTP requests can't handle
• Price monitoring — Track prices across e-commerce sites by extracting product data on a schedule
• Content extraction — Pull the main article text from news sites, blogs, and documentation without the clutter
• Research from specific URLs — When you know exactly which pages contain the information you need, extract it cleanly
• Documentation parsing — Convert technical documentation into structured data your agent can reference
• Competitive analysis — Extract and compare features, pricing, and content from competitor websites

Advanced Features

Crawl Entire Sites

Firecrawl can crawl an entire website, not just single pages. Give it a starting URL and it will follow links to discover and extract content from every page on the site:

“Crawl docs.example.com and summarize all the API endpoints”

This is useful for indexing documentation, mapping out a website's content, or building a knowledge base from an entire site.

Structured Data Extraction

Firecrawl can extract data into structured formats. Instead of raw text, you can get clean JSON with specific fields — product names, prices, ratings, dates, and any other structured data on the page.

Screenshot Capture

Firecrawl can take screenshots of pages, useful for visual comparison, monitoring UI changes, or capturing content that's difficult to extract as text.

PDF Extraction

Beyond web pages, Firecrawl can extract content from PDF files hosted online. Point it at a PDF URL and get clean, readable text back.

When to Use Firecrawl vs Search

A common question is when to use Firecrawl versus a search API like Tavily. The answer is simple:

• Search — When you need to find information. You have a question but don't know which pages have the answer. Search discovers relevant pages for you.
• Firecrawl — When you need to extract from a known page. You have a specific URL and want to pull out its content cleanly.

Many agents use both: search to find relevant pages, then Firecrawl to extract detailed content from the best results. This two-step approach gives you the best of both worlds.

Troubleshooting

Blocked by Anti-Bot Protection

Some websites use anti-bot measures like Cloudflare challenges or CAPTCHAs that can block Firecrawl:

• Firecrawl handles most anti-bot protections automatically, but some sites may still block requests
• Try using a different URL format or accessing a cached version of the page
• For heavily protected sites, consider using browser automation instead, which can handle interactive challenges

Slow Crawls

Full site crawls can take time depending on the size of the website:

• Start with a smaller scope — crawl a specific section rather than the entire site
• Use page limits to cap the number of pages crawled
• Single-page extractions are nearly instant; crawls scale linearly with page count

API Limits

If you hit Firecrawl's rate limits or quota:

• Check your usage on the Firecrawl dashboard at firecrawl.dev
• The free tier includes 500 pages per month — upgrade if you need more
• Batch multiple extractions together rather than making many individual requests
• Cache results locally when you need to reference the same page multiple times

Use Firecrawl on OpenClaw Launch

On OpenClaw Launch, the Firecrawl skill is available as a one-click install from the Dashboard. Enter your Firecrawl API key in the configurator, and it's automatically configured for your instance. No JSON editing needed.

Frequently Asked Questions

Is Firecrawl a search engine?

No. Firecrawl is a web scraping and data extraction tool, not a search engine. It extracts content from URLs you provide — it does not discover or search for pages. For web search, use Tavily or built-in web search.

Is Firecrawl free?

Firecrawl offers a free tier with 500 pages per month, which is enough for personal AI agent use. The firecrawl ClawHub skill itself is free to install. Paid plans start at $19/month for higher limits, batch crawling, and priority support.

Firecrawl vs Tavily — which should I use?

They serve different purposes. Tavily is for searching the web when you don't know which pages have the information. Firecrawl is for extracting content from specific URLs you already know about. Many agents use both — Tavily to find pages, Firecrawl to extract their content in detail.

Can Firecrawl bypass paywalls?

Firecrawl is designed for extracting publicly accessible content. It does not bypass paywalls, login walls, or other access restrictions. For content behind authentication, you would need to use browser automation with proper credentials.

What's Next?

• Tavily search setup — Pair Firecrawl with AI-optimized web search for complete web intelligence
• Web search guide — Compare all web search options available for your agent
• Browser automation — Go beyond scraping with interactive web browsing
• Explore all skills — Browse 3,200+ skills beyond web scraping — coding, automation, image generation, and more
• See pricing — Deploy with Firecrawl pre-configured, starting at $3/month