π¦ MCPBench: A Benchmark for Evaluating MCP Servers
@modelscope
About π¦ MCPBench: A Benchmark for Evaluating MCP Servers
The evaluation benchmark on MCP servers
Config
No standard config provided
This server doesn't expose a parseable MCP config block in its README. See the repository for install instructions.
RepositoryTools
No tools detected
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is π¦ MCPBench: A Benchmark for Evaluating MCP Servers?
MCPBench is an evaluation framework for MCP servers. It supports evaluating three types of servers (Web Search, Database Query, and GAIA) and is compatible with both local and remote MCP servers. The framework measures task completion accuracy, latency, and token consumption under consistent LLM and Agent configurations.
How to use π¦ MCPBench: A Benchmark for Evaluating MCP Servers?
Install Python β₯3.11, nodejs, and jq, then run pip install -r requirements.txt. Write a configuration file specifying the MCP server (remote SSE URL or local stdio with run_config). Optionally launch local servers with sh launch_mcps_as_sse.sh <config>. Then evaluate with sh evaluation_websearch.sh <config>, sh evaluation_db.sh <config>, or sh evaluation_gaia.sh <config>. The tool configuration is auto-detected.
Key features of π¦ MCPBench: A Benchmark for Evaluating MCP Servers
- Evaluates Web Search, Database Query, and GAIA server types.
- Works with both remote (SSE) and local (STDIO) MCP servers.
- Measures accuracy, latency, and token consumption.
- Provides built-in datasets for WebSearch and Database Query.
- Automatically detects server tools and parameters.
- Compatible with multiple MCP servers (e.g., Brave Search, DuckDuckGo).
Use cases of π¦ MCPBench: A Benchmark for Evaluating MCP Servers
- Benchmark Web Search MCP servers under the same LLM and agent.
- Compare Database Query MCP servers on car-related questions.
- Evaluate GAIA task performance of MCP servers.
- Assess latency and token efficiency of different MCP implementations.
- Validate custom MCP servers using the provided evaluation datasets.
FAQ from π¦ MCPBench: A Benchmark for Evaluating MCP Servers
What MCP server types are supported?
Web Search, Database Query, and GAIA.
What metrics are measured?
Task completion accuracy, latency, and token consumption.
What are the runtime requirements?
Python β₯3
Frequently asked questions
What MCP server types are supported?
Web Search, Database Query, and GAIA.
What metrics are measured?
Task completion accuracy, latency, and token consumption.
What are the runtime requirements?
Python β₯3
Basic information
More Other MCP servers
Awesome Mlops
visengerA curated list of references for MLOps
MCP Toolbox for Databases
googleapisMCP Toolbox for Databases is an open source MCP server for databases.
Glasswarp
GlasswarpSee and control a real Windows PC you own β from any MCP client, locally or remotely. Observe (UIA + screenshots), click/type/drag/scroll, launch apps, owner Live View. BYOH: your machine, your key.
MCP Go π
mark3labsA Go implementation of the Model Context Protocol (MCP), enabling seamless integration between LLM applications and external data sources and tools.
Unity MCP β¨
justinpbarnettUnity MCP acts as a bridge between AI assistants and your Unity Editor. Give your LLM tools to manage assets, control scenes, edit scripts, and automate tasks within Unity.
Comments