A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.
Overview
What is MCP Evals?
MCP Evals is a Node.js package and GitHub Action designed for evaluating Model Context Protocol (MCP) tool implementations using LLM-based scoring. It ensures that your MCP server's tools are functioning correctly and performing optimally.
How to use MCP Evals?
To use MCP Evals, you can install it as a Node.js package or integrate it into your GitHub Actions workflow. For Node.js, run npm install mcp-evals. For GitHub Actions, add the specified YAML configuration to your workflow file.
Key features of MCP Evals?
- Evaluates MCP tool implementations using LLM-based scoring.
- Provides detailed evaluation results including accuracy, completeness, relevance, clarity, and reasoning scores.
- Automatically posts evaluation results as comments on pull requests in GitHub.
Use cases of MCP Evals?
- Ensuring the accuracy of tool implementations in MCP servers.
- Automating evaluations during pull requests to maintain code quality.
- Providing feedback on tool performance to developers.
FAQ from MCP Evals?
- Can MCP Evals be used with any MCP tool?
Yes! MCP Evals is designed to work with any tool that follows the Model Context Protocol.
- Is there a specific Node.js version required?
It is recommended to use Node.js version 20 or higher.
- How do I view the evaluation results?
The results are posted as comments on the pull request where the evaluations are run.