promptfoo-evaluation

An agent skill by daymade, from daymade/claude-code-skills. Tags: ai, evaluation, python, testing, validation.

What it does

Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".

Install

With the skills CLI, which installs into Claude Code, Codex, Cursor and other agents:

npx skills add daymade/claude-code-skills --skill promptfoo-evaluation

Or copy the skill folder into Claude Code's skills directory by hand (~/.claude/skills for every project, or .claude/skills inside one):

git clone --depth 1 https://github.com/daymade/claude-code-skills
cp -r claude-code-skills/promptfoo-evaluation ~/.claude/skills/promptfoo-evaluation

Safety box score

Grade C, moderate risk: 32 of 100 risk points, rated 2026-10-20. The grade rates what the skill and its scripts can reach on the machine of whoever installs it, across eight categories from shell execution to secrets access. How the score works.

This skill configures and runs Promptfoo evals, so it reads local prompt/test files, executes Python assertion helpers, and sends prompts to external LLM APIs or a relay. The main risks are network exposure of evaluation content, dependence on API credentials, and pulling/running the CLI from `npx @latest` rather than a pinned local install.

Category levels, from 0 (none) to 3 (high)
Category	Level	Why
Code & shell execution	2 (moderate)	It explicitly runs `npx promptfoo@latest ...` and supports executable Python assertions via `type: python` pointing at `file://scripts/metrics.py:custom_assert`, so evals can execute code.
Network access	2 (moderate)	Providers are OpenAI/Anthropic or a relay with `apiBaseUrl`, so prompts, few-shot examples, and test data are sent to remote model endpoints during evals.
Filesystem reach	1 (low)	The skill reads `file://` prompts/tests/data/scripts and writes results like `outputPath: results/eval-results.json`, but the examples keep activity mostly inside the project tree.
Secrets & credentials	2 (moderate)	It relies on `ANTHROPIC_API_KEY` and `OPENAI_API_KEY`, and even says to pass a relay token as `ANTHROPIC_API_KEY`, so existing API credentials are part of normal use.
Privilege & persistence	0 (none)	No sudo, services, global installs, shell startup edits, or OS-level settings are required.
Instruction hijack surface	0 (none)	The files mention system prompts and personas as prompt content examples, but they do not instruct the agent to ignore safety rules, fetch remote instructions, or hide actions.
Irreversible actions	0 (none)	This skill evaluates prompts and reports results; it does not deploy, publish, message, pay, book, or delete anything on its own.
Transparency	1 (low)	The CLI is invoked as `promptfoo@latest`, which pulls behavior from an external package not present in the repo and therefore is not fully reviewable here.

Before installing

Pin a specific Promptfoo version instead of using `@latest`.
Read `scripts/metrics.py` and any `file://` files you plan to reference; they are executed/loaded during eval.
Confirm which API keys are set (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`) and that you are okay sending eval prompts/data to those providers.
Try the echo-provider preview config first to verify prompt rendering before any live API calls.

Rated 2026-10-20 by an automated review of 3 files from the skill's repository.

Source

Repository: daymade/claude-code-skills (all skills from this repository)
Path: promptfoo-evaluation/SKILL.md
Branch: main
Updated: 2026-09-20

ab-testing — When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.
advisor — Advisor mode. Consult a stronger (or different) model at key checkpoints: before major decisions, when stuck on an error, and before declaring a task done.
agents-pay — Use when THIS agent needs to pay for x402-protected content at runtime: hitting a paywall mid-task, settling it via AgentCore Payments.
atheris — Sets up and runs Atheris, the coverage-guided Python fuzzer built on libFuzzer.
aws-sdk-python-usage — AWS SDK for Python (boto3/botocore) development patterns. You MUST use this skill when writing Python code that uses AWS services via boto3 or botocore.
blast-radius — Find what a change could break somewhere else before it ships, beyond the diff.