llm-eval-harness

An agent skill by daymade, from daymade/claude-code-skills. Tags: api, evaluation, ml, performance, testing.

What it does

Test/evaluate any LLM behind an OpenAI- or Anthropic-compatible endpoint: availability (max_tokens-aware), request fidelity (does system prompt/tools/history REACH the model, or does the gateway silently drop it), speed (TTFT+tok/s), concurrency (before a workshop), Anthropic protocol compliance, quality regression, vendor bug reports, deployment gates, resident canaries. Use BEFORE hand-rolling a curl loop (it skips the N=10 sampling, Connection:close, and env-var key handling this bakes in). Use when someone tests/benchmarks/测评/压测 a model/endpoint, onboards a provider, decides whether to switch or temporarily fail over to an alternate channel (outage/quota), writes a supported-models list, debugs "model ignores system prompt", or verifies a tok/s claim. NOT for TTS/voice-clone supplier eval (audio side has its own skill). Triggers on "benchmark this model", "测一下这个模型/渠道/API", "接入新模型先测一下", "system prompt 不生效", "这个渠道能不能用/稳不稳", "临时切换过去顶一阵子" — even without "eval", even wrapped in business narrative.

Install

With the skills CLI, which installs into Claude Code, Codex, Cursor and other agents:

npx skills add daymade/claude-code-skills --skill llm-eval-harness

Or copy the skill folder into Claude Code's skills directory by hand (~/.claude/skills for every project, or .claude/skills inside one):

git clone --depth 1 https://github.com/daymade/claude-code-skills
cp -r claude-code-skills/llm-eval-harness ~/.claude/skills/llm-eval-harness

Safety box score

Grade C, moderate risk: 37 of 100 risk points, rated 2026-10-16. The grade rates what the skill and its scripts can reach on the machine of whoever installs it, across eight categories from shell execution to secrets access. How the score works.

This skill benchmarks third-party LLM endpoints by sending prompts, tool calls, multi-turn history, and API keys over the network, then saving results and evidence files locally. The main risks are external data exposure to the chosen vendor, runtime dependency downloads via `uv run --with ...`, and persistent on-disk artifacts under `~/.llm-eval/` or `--raw-dir` paths.

Category levels, from 0 (none) to 3 (high)
Category	Level	Why
Code & shell execution	2 (moderate)	The intended workflow runs local Python code via commands like `uv run --with aiohttp python scripts/speed_probe.py ...` and `uv run --with openai ...`, so it executes scripts and may resolve dependencies at runtime.
Network access	2 (moderate)	All probes POST to user-supplied `--base-url`/`--url` endpoints and send prompts, tool schemas, history, and auth headers; `fidelity_probe.py` and `concurrency_probe.py` also explicitly manage HTTP sessions and proxies.
Filesystem reach	2 (moderate)	The probes write outputs and evidence files (`--output`, `--raw-dir`, `~/.llm-eval/runs/<model>`) and read use-case libraries from home-directory paths, so they persist data outside the repo.
Secrets & credentials	1 (low)	API keys are read from env vars and forwarded in `Authorization` or `x-api-key` headers; the scripts avoid command-line leakage but still transmit a live secret to the remote endpoint.
Privilege & persistence	0 (none)	There is no sudo, service registration, shell startup editing, or OS-level configuration change in the shipped files.
Instruction hijack surface	0 (none)	The files contain benchmark guidance about system prompts and judging, but nothing that tells the agent to ignore higher-level instructions, follow remote content, or hide behavior.
Irreversible actions	1 (low)	The harness can autonomously fan out repeated and concurrent benchmark requests, but it does not deploy, pay, post, delete, or otherwise take irreversible external actions.
Transparency	2 (moderate)	Several commands rely on `uv run --with aiohttp/openai` without pinned versions, so third-party packages may be downloaded and executed at runtime rather than being fully vendored in the bundle.

Before installing

Use a throwaway or least-privilege API key first; the probes will send real authenticated requests to the vendor you point them at.
Read the output paths you plan to use (`--raw-dir`, `--output-dir`, `~/.llm-eval/`) so you know where prompts and responses will be stored.
Expect package fetches on first run from `uv run --with aiohttp/openai`; if that is a concern, test in a scratch environment before using a production machine.

Rated 2026-10-16 by an automated review of 12 files from the skill's repository.

Source

Repository: daymade/claude-code-skills (all skills from this repository)
Path: llm-eval-harness/SKILL.md
Branch: main
Updated: 2026-09-20

benchmark — Performance regression detection. (gstack)
fine-tuning-expert — Use when fine-tuning LLMs, training custom models, or adapting foundation models for specific tasks.
test-master — Generates test files, creates mocking strategies, analyzes code coverage, designs test architectures.
ab-testing — When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.
agents-optimize — Use when measuring or improving agent quality and performance — set up evaluators, online monitoring, CI/CD quality gates, observability, or cost optimization.
amazon-bedrock — Builds generative AI applications on Amazon Bedrock.