CrawlForge MCP ServerVerifiedFeatured

What is CrawlForge MCP Server?

CrawlForge MCP Server is a Model Context Protocol server that gives AI assistants 28 web tools — scraping, crawling, web search, real Google SERP rank tracking, multi-source deep research, an autonomous agent, document processing and stealth browsing — through a single stdio or Streamable HTTP connection.

It is built for developers and agent builders who want live web data in a shape a model can actually use: clean Markdown, schema-validated JSON, links, metadata and screenshots from one fetch, rather than raw HTML a model has to guess its way through. Every tool is MCP-native and metered by credits, so cost is predictable per call. The server is MIT-licensed, runs on Node.js 18+, and works in Claude Desktop, Claude Code, Cursor, n8n and any other MCP client.

Unlike most hosted scraping APIs, LLM-powered extraction defaults to a local Ollama model — no LLM provider key, no per-token cost, and page content never leaves your machine. OpenAI and Anthropic are opt-in.

How to use CrawlForge MCP Server?

Get a free API key at crawlforge.dev/signup — 1,000 credits, no credit card. Then run the one-step installer, which stores your key and registers the server with every MCP client it detects:

npx crawlforge init

Or configure a client by hand with the Config block above (CRAWLFORGE_API_KEY is the only required environment variable). For n8n or any HTTP-transport client, run the server in Streamable HTTP mode:

export CRAWLFORGE_API_KEY=cf_live_your_api_key_here
npx -p crawlforge-mcp-server crawlforge-mcp --http   # Streamable HTTP endpoint on :10000/mcp

Then just ask in natural language — "research the top 5 competitors in this space and cite sources", "crawl these docs and summarize them", "pull the price and rating from this Amazon listing", "tell me when this pricing page changes" — and the client picks the right tool.

Optional environment variables: OLLAMA_BASE_URL / OLLAMA_DEFAULT_MODEL to steer local LLM extraction, OPENAI_API_KEY or ANTHROPIC_API_KEY for cloud extraction, DATAFORSEO_LOGIN / DATAFORSEO_PASSWORD to enable serp_rank, and CRAWLFORGE_TOOLS / CRAWLFORGE_TOOL_GROUPS to expose only a subset of tools to a given client.

Key features of CrawlForge MCP Server

• 28 MCP-native tools covering the whole pipeline — discover, fetch, extract, analyze, research, monitor — so you rarely need a second server.
• One fetch, many formats — the unified scrape tool takes a formats array (markdown, html, rawHtml, text, links, metadata, screenshot, JSON-schema) and serves every requested format from a single request, with per-format partial-success warnings.
• Local-LLM extraction by default — extract_with_llm and extract_structured run against local Ollama out of the box: $0 per token, and your page content stays on your machine. Cloud providers are opt-in per call.
• Autonomous agent — describe the outcome in plain English with no URLs; it plans, searches, navigates and shapes an answer under orchestrator-enforced hard stops (max steps, max URLs, wall-clock), never limits the LLM can talk itself out of.
• deep_research with stealth recovery — multi-stage research that automatically retries sources returning 403 through a real fingerprinted browser (Camoufox) and re-extracts from the rendered HTML.
• Real Google organic rank — serp_rank returns the actual SERP position for a keyword and domain, the number a search API cannot give you. It costs 0 credits until you configure DataForSEO.
• 11 pre-built site templates — scrape_template returns structured data from Amazon, Shopify, LinkedIn, GitHub, YouTube, X/Twitter, Reddit, Hacker News, Product Hunt, Stack Overflow and npm without writing a single selector.
• Change monitoring — track_changes baselines a page and reports a change percentage plus a diff on every later check, with CSS-selector scoping and webhooks.
• Current MCP spec (2025-06-18) — schema-validated structuredContent on the tools that return records, self-correctable input errors the model can retry from, deterministic tools/list ordering for prompt-cache friendliness, elicitation before expensive runs, and experimental async tasks on the four long-running tools.
• Security-hardened — SSRF validation on every URL and every redirect hop (loopback, RFC1918, IPv6 link-local, cloud metadata endpoints and dangerous ports all blocked), a fail-closed backend allow-list, a 7-action allowlist for browser automation with JavaScript execution off by default, and per-tool credit gating.
• Polite by default — requests identify honestly as CrawlForge/<version>, respect robots.txt, and are cryptographically signed with Web Bot Auth (RFC 9421) so sites can verify who is crawling them.
• Credits that don't expire — the 1,000-credit free grant and paid-plan credits both roll over; nothing is lost at month end.

Use cases of CrawlForge MCP Server

• Competitive and market research — deep_research or agent gathers, dedupes and synthesizes across many sources with citations, from a single natural-language prompt.
• Feeding documentation to a coding assistant — map_site discovers the URL inventory, crawl_deep walks it, and generate_llms_txt produces an llms.txt for the site.
• Price and inventory extraction — scrape_template for known marketplaces, extract_structured with your own JSON schema for everything else.
• Competitor and compliance monitoring — track_changes watches pricing pages, terms, or regulatory notices and fires a webhook when a page meaningfully changes.
• Bulk collection — batch_scrape processes many URLs in one job (async with a webhook for large batches), paginated back through get_batch_results.
• Reaching pages that block scrapers — stealth_mode and scrape_with_actions handle Cloudflare/DataDome challenges, logins, and click-and-wait flows; localization emulates a country and language for geo-specific pricing.
• SEO and content workflows — search_web for ranked results, serp_rank for true organic position, analyze_content and summarize_content for sentiment, entities, keywords and summaries.
• Document pipelines — process_document parses PDFs, Word files and other formats into text a model can reason over.

FAQ from CrawlForge MCP Server

Do I need an API key, and is there a free tier?

Yes to both. Every tool is metered, so a CrawlForge API key is required — there is no anonymous mode. New accounts get 1,000 free credits, no credit card, and that grant never expires. Get one at crawlforge.dev/signup.

How much does each tool cost in credits?

Costs range from 1 to 10 credits per call:

Credits	Tools
1	fetch_url, extract_text, extract_links, extract_metadata, scrape_template, list_ollama_models, get_batch_results
2	scrape, scrape_structured, extract_content, map_site, process_document, localization
3	track_changes, analyze_content, extract_structured, extract_with_llm
4	summarize_content, crawl_deep
5	search_web, reddit_search, serp_rank, batch_scrape, scrape_with_actions, stealth_mode, generate_llms_txt
8	agent
10	deep_research

Two exceptions work in your favour: serp_rank charges 0 until DataForSEO is configured, and stealth_mode bookkeeping operations (configure, enable, disable, get_stats, cleanup) cost 1 credit instead of 5. Paid plans are Hobby $19/mo (5,000 credits), Professional $99/mo (50,000), Business $399/mo (250,000) — all with access to every tool, and credits roll over month to month.

Do I need an OpenAI or Anthropic key for the LLM-powered tools?

No. extract_with_llm, extract_structured and deep_research default to a local Ollama model, so there is no LLM API key, no per-token charge, and no page content leaving your machine. Run list_ollama_models to see what you have installed. Pass provider: "openai" or provider: "anthropic" with the matching key only if you want a cloud model — that usage is billed by your provider, not by CrawlForge.

Which client should I use it with, and how do I connect?

Any MCP client. npx crawlforge init auto-detects and configures Claude Desktop, Claude Code and Cursor in one command; other clients take the standard npx -y crawlforge-mcp-server stdio stanza shown in the Config block. For n8n or anything speaking HTTP, start the server with --http and point the MCP Client Tool node at http://<host>:10000/mcp with a Bearer credential set to your API key.

What happens when a site blocks the scraper?

Escalate. stealth_mode runs an anti-detection browser (Chromium stealth by default, Camoufox for advanced fingerprinting) and clears challenges that a plain fetch cannot, while scrape_with_actions drives real interactions — click, type, scroll, wait, screenshot — before extracting. deep_research does this automatically: sources that return 403 are silently retried through Camoufox and re-extracted from the rendered HTML. Hard IP-reputation blocks still require residential proxies, which CrawlForge does not provide.

Is it safe to point at arbitrary URLs?

Every URL is validated before the request leaves the process — http/https only, with loopback, RFC1918, IPv6 private and link-local ranges, cloud metadata endpoints and dangerous ports (SSH, SMTP, MySQL, Postgres, Redis, MongoDB and others) all blocked, and each redirect hop re-validated up to a cap of 5. Browser automation accepts only 7 vetted action types and executeJavaScript throws unless explicitly enabled at deploy time. Scraped data is never retained — only usage logs.

---

Links

• Website: https://www.crawlforge.dev
• Documentation: https://www.crawlforge.dev/docs
• GitHub: https://github.com/mysleekdesigns/crawlforge-mcp
• npm: https://www.npmjs.com/package/crawlforge-mcp-server
• License: MIT