ab-test-analysis

An agent skill by phuryn, from phuryn/pm-skills. Tags: automation, spreadsheets, trend-analysis, web.

What it does

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

Install

With the skills CLI, which installs into Claude Code, Codex, Cursor and other agents:

npx skills add phuryn/pm-skills --skill ab-test-analysis

Or copy the skill folder into Claude Code's skills directory by hand (~/.claude/skills for every project, or .claude/skills inside one):

git clone --depth 1 https://github.com/phuryn/pm-skills
cp -r pm-skills/pm-data-analytics/skills/ab-test-analysis ~/.claude/skills/ab-test-analysis

Safety box score

Grade A, minimal risk: 7 of 100 risk points, rated 2026-09-23. The grade rates what the skill and its scripts can reach on the machine of whoever installs it, across eight categories from shell execution to secrets access. How the score works.

This skill is a local A/B-test analysis helper: it reads user-provided experiment data, computes statistics, and writes a markdown summary. The main risks are limited code execution for generated Python analysis and handling of whatever data files the user gives it; it does not include any native scripts or destructive actions.

Category levels, from 0 (none) to 3 (high)
Category	Level	Why
Code & shell execution	1 (low)	It may "generate Python scripts for statistical calculations when needed" and "run" them for raw data, but there are no bundled executables or shell commands.
Network access	0 (none)	There is no instruction to download data or send results to a remote host; the only external references are read-only "Further Reading" links.
Filesystem reach	1 (low)	It reads user-provided CSV/Excel/analytics exports and says to "Save as markdown," which implies writing output in the working directory only.
Secrets & credentials	0 (none)	The skill does not ask for tokens, passwords, key files, or browser sessions.
Privilege & persistence	0 (none)	No sudo, global installs, services, shell startup edits, or other privilege-changing actions are requested.
Instruction hijack surface	0 (none)	The text contains normal analysis guidance like "Think step by step" but no instruction to ignore policies, hide actions, or follow remote instructions.
Irreversible actions	0 (none)	It produces analysis and recommendations, but it does not push, post, deploy, pay, or otherwise act on its own.
Transparency	0 (none)	The skill is plain markdown with no binaries, embedded blobs, or hidden/runtime-decoded code.

Before installing

Open `SKILL.md` and confirm you are comfortable with it generating/running Python for stats.
Use a scratch dataset first if you do not want it reading sensitive CSV/Excel exports.
Check whether you want markdown output written into the current workspace or a separate folder.

Rated 2026-09-23 by an automated review of 1 file from the skill's repository.

Source

Repository: phuryn/pm-skills (all skills from this repository)
Path: pm-data-analytics/skills/ab-test-analysis/SKILL.md
Branch: main
Updated: 2026-09-19

cohort-analysis — Perform cohort analysis on user engagement data — retention curves, feature adoption trends, and segment-level insights.
sentiment-analysis — Analyze user feedback data to identify segments with sentiment scores, JTBD, and product satisfaction insights.
growth-loops — Identify growth loops (flywheels) for sustainable traction. Evaluates 5 loop types: Viral, Usage, Collaboration, User-Generated, and Referral.
user-segmentation — Segment users from feedback data based on behavior, JTBD, and needs. Identifies at least 3 distinct user segments.
aws-blocks — Guides building full-stack applications with AWS Blocks — an Infrastructure-from-Code framework.
chrome-mcp-troubleshooting — Diagnose and fix Claude in Chrome MCP extension connectivity issues.