6 months ago
A powerful Model Context Protocol (MCP) server that intelligently fetches and processes web content with nested URL crawling capabilities. Transform any documentation site or web resource into clean, structured markdown files perfect for AI consumption and analysis.
Overview
What is Better Fetch?
Better Fetch is a powerful Model Context Protocol (MCP) server designed to intelligently fetch and process web content, featuring nested URL crawling capabilities. It transforms any documentation site or web resource into clean, structured markdown files suitable for AI consumption and analysis.
How to use Better Fetch?
To use Better Fetch, clone the repository from GitHub, install the necessary dependencies, and configure your MCP client to point to the Better Fetch server. You can then use the provided tools to fetch web content either through nested crawling or single-page extraction.
Key features of Better Fetch?
- Smart Web Crawling: Nested URL fetching, single-page mode, domain filtering, and pattern matching.
- Intelligent Content Processing: Content cleaning, smart section detection, automatic titles, and table of contents generation.
- Advanced Markdown Generation: Clean formatting, code block preservation, link preservation, and metadata integration.
- Highly Configurable: Crawl depth control, page limits, timeout settings, and error handling.
Use cases of Better Fetch?
- Processing API documentation and technical references.
- Conducting competitive analysis by gathering competitor documentation.
- Creating structured knowledge bases from scattered web content.
- Preparing clean content for AI model training and chatbot knowledge bases.
FAQ from Better Fetch?
- What is the maximum crawl depth?
The default maximum crawl depth is 2 levels, but it can be configured.
- Can I limit the number of pages processed?
Yes, you can set a maximum page limit (default is 50 pages).
- Is Better Fetch free to use?
Yes, Better Fetch is open-source and free to use.
Server Config
{
"mcpServers": {
"better-fetch": {
"command": "node",
"args": [
"/absolute/path/to/better-fetch/dist/server.js"
],
"env": {
"NODE_ENV": "production"
}
}
}
}