LLM Eval Harness

io.github.razilzul/llm-eval-harness

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

View repository →

Connect this server

Add this to your MCP client config — claude_desktop_config.json (Claude Desktop) or .cursor/mcp.json (Cursor).Suits: Claude Desktop · Cursor.

{
  "mcpServers": {
    "llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "env": {
        "ANTHROPIC_API_KEY": "<your ANTHROPIC_API_KEY>",
        "GEMINI_API_KEY": "<your GEMINI_API_KEY>"
      }
    }
  }
}
Environment variables
  • ANTHROPIC_API_KEYsecretOnly needed to evaluate anthropic:* models. Not needed for mock models.
  • GEMINI_API_KEYsecretOnly needed to evaluate gemini:* models. Not needed for mock models.

A starting point — check the repository for exact arguments and setup. Restart your client after editing the config.

Category
Developer Tools
Version
1.0.0
Registry
active
Published
10/18/2026

More Developer Tools servers

Browse all Developer Tools →

state-trace

razroo

Bounded working memory for coding agents — typed nodes, current-vs-stale state, MCP-mountable.

Developer ToolsAdded 6 months ago

Razz Games

razz-games

Play provably fair games with real SOL wagering from any AI agent

Developer ToolsAdded 7 months ago

Claudexor

razzant

Route coding work across Claude Code, Codex, Cursor, and OpenCode with shared context.

Developer ToolsAdded 1 week ago

Awesome Copilot MCP Server

RbBtSn0w

Model Context Protocol server for awesome-copilot agents and collections

Developer ToolsAdded 6 days ago

Hindsight MCP

Rbfinch

MCP server for AI-assisted coding with development history: git, tests, Copilot.

Developer ToolsAdded 9 months ago

Scherlok

rbmuller

Zero-config data-quality monitoring: profile a warehouse, detect anomalies, gate CI.

Developer ToolsAdded 5 days ago

Add the badge to your site

Show visitors that LLM Eval Harness is listed here — it’s also the link we check for a free listing.

LLM Eval Harness — listed on Serverdex

HTML — for your website

<a href="https://serverdex.io/server/io-github-razilzul-llm-eval-harness?utm_source=badge"><img src="https://serverdex.io/brand/badge.svg" alt="LLM Eval Harness — listed on Serverdex" width="123" height="40" loading="lazy"></a>

Recommended. A dofollow link from your own domain is the one that counts for search engines, and it's the link we check for a free listing.

Markdown — for your README

[![LLM Eval Harness — listed on Serverdex](https://serverdex.io/brand/badge.svg)](https://serverdex.io/server/io-github-razilzul-llm-eval-harness?utm_source=badge)

Nice for visibility, but GitHub marks README links nofollow, so it doesn't pass authority. Use the HTML badge on your site as well.