spark-engineer

An agent skill by Jeffallan, from Jeffallan/claude-skills. Tags: analytics, cloud, data, mlops, performance.

What it does

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

Install

With the skills CLI, which installs into Claude Code, Codex, Cursor and other agents:

npx skills add Jeffallan/claude-skills --skill spark-engineer

Or copy the skill folder into Claude Code's skills directory by hand (~/.claude/skills for every project, or .claude/skills inside one):

git clone --depth 1 https://github.com/Jeffallan/claude-skills
cp -r claude-skills/skills/spark-engineer ~/.claude/skills/spark-engineer

Safety box score

Grade C, moderate risk: 30 of 100 risk points, rated 2026-10-19. The grade rates what the skill and its scripts can reach on the machine of whoever installs it, across eight categories from shell execution to secrets access. How the score works.

This is a documentation-only Spark engineering skill that helps the agent draft PySpark/Scala jobs and tuning advice. It does not ship scripts or installers, but its examples cover remote data sources/sinks like S3, Kafka, JDBC, and checkpointed streaming writes, so the main risk is generated code that moves or overwrites data outside the workspace.

Category levels, from 0 (none) to 3 (high)
Category	Level	Why
Code & shell execution	1 (low)	It contains directly executable Spark code snippets such as `SparkSession.builder...getOrCreate()` and `df.writeStream...start()`, but no native shell scripts or code-eval payloads.
Network access	2 (moderate)	The examples explicitly connect to external services and storage (`s3://bucket/...`, `kafka.bootstrap.servers`, `jdbc:postgresql://host:5432/db`), so an agent using the skill may read from or send data over the network.
Filesystem reach	2 (moderate)	The skill shows writes and checkpoints outside the project tree, including `result.write.mode("overwrite").parquet("s3://bucket/output/")` and `setCheckpointDir("hdfs://path/checkpoints/")`.
Secrets & credentials	1 (low)	It includes secret-bearing connection settings like `option("password", "<db-password>")` and Kafka security options, but it does not itself read or harvest credentials.
Privilege & persistence	0 (none)	No sudo, service registration, shell startup edits, or global installs appear in the skill files.
Instruction hijack surface	0 (none)	There is no text telling the agent to ignore rules, hide actions, fetch remote instructions, or adopt a deceptive persona.
Irreversible actions	2 (moderate)	The code templates include external writes and publishes (`writeStream.format("kafka")`, JDBC `foreachBatch`, `mode("overwrite")`), which can modify or publish data once run.
Transparency	0 (none)	All files are plain markdown with readable code; there are no binaries, hidden blobs, or downloaded installers.

Before installing

Read `SKILL.md` and search the generated code for `overwrite`, `kafka`, `jdbc`, and `s3://` before running it.
Replace any placeholder secrets like `"<db-password>"` with secret-managed configs, not hard-coded values.
Run the first generated job only against test buckets/topics and a scratch Spark cluster, not production sinks.

Rated 2026-10-19 by an automated review of 6 files from the skill's repository.

Source

Repository: Jeffallan/claude-skills (all skills from this repository)
Path: skills/spark-engineer/SKILL.md
Branch: main
Updated: 2026-09-19

aws-billing-and-cost-management — Analyze AWS costs, find savings, manage budgets, evaluate Savings Plans and Reserved Instances, right-size EC2/Lambda/RDS/EBS with Compute Optimizer.
aws-messaging-and-streaming — Guides general use of AWS messaging and streaming services.
aws-observability — Builds, configures, debugs, and optimizes AWS observability with CloudWatch (Log Insights, Metrics, Alarms, Dashboards, EMF), X-Ray, CloudTrail.
pandas-pro — Performs pandas DataFrame operations for data analysis, manipulation, and transformation.
agents-optimize — Use when measuring or improving agent quality and performance — set up evaluators, online monitoring, CI/CD quality gates, observability, or cost optimization.
analytics — When the user wants to set up, improve, or audit analytics tracking and measurement.