MCP Smart Crawler
@loo-y
About MCP Smart Crawler
MCP Smart Crawler is a Model Context Protocol (MCP) server uses Playwright to crawl web content, extract metadata, and download resources such as videos and images.
Config
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"mcp-smart-crawler": {
"command": "npx",
"args": [
"-y",
"mcp-smart-crawler@latest",
"--download-folder",
"C:\\downloads"
]
}
}
}Tools
No tools detected
Fetch the live tool list by running this server in a temporary sandbox using the button above.
Overview
What is MCP Smart Crawler?
MCP Smart Crawler is a Model Context Protocol (MCP) server that uses Playwright to crawl web content, extract metadata, and download resources such as videos and images. It is designed for users who need to programmatically retrieve and save content from Xiaohongshu (小红书) share links.
How to use MCP Smart Crawler?
Configure your MCP client with the provided JSON, setting the command to npx and adding optional arguments like --download-folder to specify a local download directory. Then send a chat message containing a Xiaohongshu share link to trigger crawling and downloading.
Key features of MCP Smart Crawler
- Extracts metadata (title, description, images) from Xiaohongshu posts.
- Downloads videos and images from Xiaohongshu share links.
- Uses Playwright for browser automation.
- Configurable download folder via command-line argument.
Use cases of MCP Smart Crawler
- Automatically fetching and saving Xiaohongshu post images for content aggregation.
- Downloading videos from shared Xiaohongshu links for offline viewing.
- Extracting metadata from Xiaohongshu posts for analysis or archiving.
FAQ from MCP Smart Crawler
What does MCP Smart Crawler do?
It crawls web pages using Playwright, extracts metadata like title and images from Xiaohongshu posts, and downloads associated videos and images.
What dependencies does MCP Smart Crawler require?
It requires Playwright for browser automation, and is run via npx (Node.js package runner). No other runtime dependencies are specified.
Where are downloaded files saved?
Files are saved to the local folder specified by the --download-folder argument. If not provided, the default behavior is not documented.
Does MCP Smart Crawler support authentication or transport security?
No authentication or transport security mechanisms are mentioned in the README.
Can MCP Smart Crawler work with websites other than Xiaohongshu?
The README only describes functionality for Xiaohongshu share links; no other websites are mentioned.
Frequently asked questions
What does MCP Smart Crawler do?
It crawls web pages using Playwright, extracts metadata like title and images from Xiaohongshu posts, and downloads associated videos and images.
What dependencies does MCP Smart Crawler require?
It requires Playwright for browser automation, and is run via `npx` (Node.js package runner). No other runtime dependencies are specified.
Where are downloaded files saved?
Files are saved to the local folder specified by the `--download-folder` argument. If not provided, the default behavior is not documented.
Does MCP Smart Crawler support authentication or transport security?
No authentication or transport security mechanisms are mentioned in the README.
Can MCP Smart Crawler work with websites other than Xiaohongshu?
The README only describes functionality for Xiaohongshu share links; no other websites are mentioned.
Basic information
More Browser Automation MCP servers
mcp-browser-use: MCP server for browser-use
vinayak-mehtaMCP server for browser-use
Playwright Mcp
microsoftPlaywright MCP server
fetch-mcp
egoistAn MCP server for fetching URLs / Youtube video transcript.
Puppeteer
modelcontextprotocolModel Context Protocol Servers
Parse
Parse.botParse allows your agent to access thousands of APIs instantly. You can use Parse to build an API for any action on any website and edit any previously existing API in our catalogue.
Comments