llm-eval-harness

An agent skill by daymade, from daymade/claude-code-skills. Tags: api, evaluation, ml, performance, testing.

What it does

Test/evaluate any LLM behind an OpenAI- or Anthropic-compatible endpoint: availability (max_tokens-aware), request fidelity (does system prompt/tools/history REACH the model, or does the gateway silently drop it), speed (TTFT+tok/s), concurrency (before a workshop), Anthropic protocol compliance, quality regression, vendor bug reports, deployment gates, resident canaries. Use BEFORE hand-rolling a curl loop (it skips the N=10 sampling, Connection:close, and env-var key handling this bakes in). Use when someone tests/benchmarks/测评/压测 a model/endpoint, onboards a provider, decides whether to switch or temporarily fail over to an alternate channel (outage/quota), writes a supported-models list, debugs "model ignores system prompt", or verifies a tok/s claim. NOT for TTS/voice-clone supplier eval (audio side has its own skill). Triggers on "benchmark this model", "测一下这个模型/渠道/API", "接入新模型先测一下", "system prompt 不生效", "这个渠道能不能用/稳不稳", "临时切换过去顶一阵子" — even without "eval", even wrapped in business narrative.

Install

With the skills CLI, which installs into Claude Code, Codex, Cursor and other agents:

npx skills add daymade/claude-code-skills --skill llm-eval-harness

Or copy the skill folder into Claude Code's skills directory by hand (~/.claude/skills for every project, or .claude/skills inside one):

git clone --depth 1 https://github.com/daymade/claude-code-skills
cp -r claude-code-skills/llm-eval-harness ~/.claude/skills/llm-eval-harness

Safety box score

Not rated yet. A safety box score grades what a skill and its scripts can reach on the machine of whoever installs it, across eight categories from shell execution to secrets access. Anyone can request one from this page; it is saved for everyone. How the score works.

Source

Repository

daymade/claude-code-skills (all skills from this repository)

Path

llm-eval-harness/SKILL.md

Branch

main

Updated

2026-09-20

Related skills

• fine-tuning-expert — Use when fine-tuning LLMs, training custom models, or adapting foundation models for specific tasks.
• test-master — Generates test files, creates mocking strategies, analyzes code coverage, designs test architectures.
• ab-testing — When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.
• agents-optimize — Use when measuring or improving agent quality and performance — set up evaluators, online monitoring, CI/CD quality gates, observability, or cost optimization.
• amazon-bedrock — Builds generative AI applications on Amazon Bedrock.
• coverage-analysis — Measures and interprets what a fuzzing campaign actually reaches, using llvm-cov, lcov, or a fuzzer's own coverage output.