Llama-Server MCP Proxy

What is Llama-Server MCP Proxy?

Llama-Server MCP Proxy is a Node.js application that acts as a proxy for llama-server, intercepting chat completion requests to add Model Context Protocol (MCP) tool capabilities. It allows a local LLM to iteratively use external tools defined in an mcp-config.json file. The proxy runs on Node.js 18 or higher and works with any llama-server instance or OpenAI-compatible API endpoint.

How to use Llama-Server MCP Proxy?

Set up the proxy by creating a project with the provided package.json, installing dependencies via npm install, and editing mcp-config.json to point to your MCP tool servers. Start the proxy with npm start (default port 9090), then point your LLM GUI client (e.g., a browser) to the proxy’s address (http://localhost:9090) and interact normally. The proxy automatically detects tool calls in the LLM output and executes them.

Key features of Llama-Server MCP Proxy

• MCP Tool Integration: connects to any MCP-compatible tool server.
• Dynamic system prompt automatically lists available tools.
• Iterative tool use in a single user turn.
• Streaming support (Server-Sent Events) with execution messages.
• Robust error handling that returns errors to the LLM.
• Configurable via environment variables (port, server URL, timeout).
• Optional verbose debug logging.

Use cases of Llama-Server MCP Proxy

• Enable a local LLM to perform web searches (Brave Search, search1api).
• Allow iterative tool use for complex multi-step queries.
• Prototype MCP tool integrations with a local LLM instance.
• Add filesystem or browser automation tools (e.g., puppeteer) to a local chat interface.

FAQ from Llama-Server MCP Proxy

What does Llama-Server MCP Proxy do?

It acts as a proxy that adds MCP tool capabilities to llama-server, intercepting chat completion requests and enabling the LLM to call external tools iteratively.

Which models are supported?

Any model runnable by llama-server (i.e., any llama.cpp‑compatible model) or any OpenAI‑like API endpoint.

How does it connect to MCP servers?

The proxy reads mcp-config.json to discover tool servers, launches them as subprocesses (using their command and args), and communicates via the @modelcontextprotocol/sdk.

Is streaming supported?

Yes, the proxy maintains streaming (SSE) of LLM responses, including intermediate messages like [Executing tool...] and tool results.

What known limitations exist?

Filesystem access can be problematic depending on the model; tool names must exactly match those registered by MCP servers. The project is still experimental.