Guide

OpenClaw + Crawl4AI: Self-Hosted Web Crawling over MCP

Q: Can OpenClaw connect to Crawl4AI?

Yes. Configure Crawl4AI’s SSE MCP endpoint under OpenClaw mcp.servers, then use the OpenClaw MCP probe command to verify capabilities.

Q: What is the Crawl4AI MCP URL?

The documented default SSE endpoint is http://localhost:11235/mcp/sse. Use a reachable hostname when OpenClaw runs in another container or host.

Give OpenClaw a structured crawling tool without routing every page through a third-party extraction API. Crawl4AI supplies the browser and MCP server; OpenClaw controls when the tools are used.

Architecture

Crawl4AI runs separately and exposes crawling capabilities through MCP. OpenClaw saves the remote definition, discovers its tools, and presents eligible tools to the agent runtime. This separation is useful: browser dependencies and crawl concurrency stay in the crawler service, while OpenClaw keeps channel, model, approval, and agent policy.

1. Start a pinned Crawl4AI server

Follow the official Crawl4AI self-hosting guide and pin the image or package version you tested. Do not substitute :latest in production. The default server listens on port 11235 and documents MCP endpoints for SSE and WebSocket clients.

# Confirm the service is reachable from the OpenClaw host
curl -fsS http://crawl4ai:11235/health

# The documented MCP SSE endpoint is:
# http://crawl4ai:11235/mcp/sse

2. Add the MCP server to OpenClaw

Current OpenClaw configuration uses mcp.servers. Older examples using tools.mcp or agents.defaults.mcp are not the current schema and can cause strict config validation to fail.

{
  "mcp": {
    "servers": {
      "crawl4ai": {
        "url": "http://crawl4ai:11235/mcp/sse",
        "transport": "sse"
      }
    }
  }
}

The OpenClaw configuration reference documents streamable-http and sse as the supported remote transports. If both services run directly on one host, localhost can work. Across containers, use the service DNS name or a private address reachable from the OpenClaw container.

3. Probe before asking the agent

openclaw mcp list
openclaw mcp status --verbose
openclaw mcp probe crawl4ai
openclaw mcp tools crawl4ai

A probe opens a real MCP connection and lists capabilities, separating transport problems from model or prompt problems.

4. Give the agent a bounded task

Start with one public URL, an explicit page limit, the fields you need, and a required citation. Avoid broad instructions such as “crawl this whole domain.” Add depth, concurrency, and same-origin limits at the crawler, not only in the prompt.

Production hardening

Keep Crawl4AI private or require authenticated access through a reverse proxy.
Block cloud metadata and private network destinations unless explicitly required.
Set time, memory, response-size, depth, and concurrency limits.
Respect robots rules, site terms, copyright, and personal-data restrictions.
Log target URLs and tool outcomes without logging secrets or full sensitive page bodies.

Crawl4AI versus Firecrawl

Crawl4AI is attractive when you want a self-hosted, MCP-native crawler with explicit tools. Firecrawl is often easier when your integration already speaks its API and you want managed extraction. Compare this setup with the OpenClaw + Firecrawl guide.

Frequently asked questions

Can OpenClaw connect to Crawl4AI?

Yes. Crawl4AI exposes an MCP server, and current OpenClaw releases store remote MCP definitions under mcp.servers. Use the Crawl4AI SSE endpoint and probe it before an agent run.

What is the Crawl4AI MCP URL?

The default self-hosted SSE endpoint documented by Crawl4AI is http://localhost:11235/mcp/sse. Replace localhost with a hostname reachable from the OpenClaw runtime.