Turn any website into LLM-ready, and provide SERP
Overview
What is AnyCrawl?
AnyCrawl is a powerful web scraping and crawling tool designed to turn any website into LLM-ready content, providing seamless integration with various LLM clients via the Model Context Protocol (MCP).
How to use AnyCrawl?
To use AnyCrawl, sign up on the AnyCrawl website to receive an API key, then set it as the ANYCRAWL_API_KEY environment variable. You can start crawling by running the AnyCrawl MCP server with the provided command.
Key features of AnyCrawl?
- Web scraping with multiple output formats
- Configurable website crawling with depth limits
- Integration with search engines for scraping results
- Support for multiple engines like Playwright, Cheerio, and Puppeteer
- Flexible output options including Markdown, HTML, and structured JSON
- Non-blocking async operations with status monitoring
- Robust error handling and logging
- Multiple deployment modes available
Use cases of AnyCrawl?
- Extracting data from specific web pages for analysis.
- Crawling entire websites for content aggregation.
- Integrating web search results into applications.
- Automating data collection for research purposes.
FAQ from AnyCrawl?
- How do I get started with AnyCrawl?
Sign up on the AnyCrawl website to receive your API key and follow the setup instructions.
- Is there a free tier available?
Yes! You can sign up for free and receive 1,500 credits to crawl nearly 1,500 pages.
- What output formats does AnyCrawl support?
AnyCrawl supports Markdown, HTML, text, screenshots, and structured JSON.
Server Config
{
"mcpServers": {
"anycrawl-mcp": {
"command": "npx",
"args": [
"-y",
"anycrawl-mcp-server"
],
"env": {
"ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
"ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
"LOG_LEVEL": "info"
}
}
}
}