/suede-ai-eval · AI evaluation

An AI feature you can't fail isn't tested.

Suede AI Eval turns an AI-powered product surface into an AI-SPEC, a failure-mode rubric, concrete prompt or retrieval eval cases, acceptance gates, and a coverage audit. Use it for LLM routes, classifiers, recommenders, agents, RAG/search, generated media, or any AI workflow that needs measurable quality and safety checks.

Install the skill View skill folder

"make this AI feature measurable before it ships" /suede-ai-eval Ask in plain words. The router reads the request and loads this lane; MCP agents find it with list_suede_skills.

$ /suede-ai-eval spec the support bot failure modes mapped · 24 eval cases written acceptance gate: hallucinated refund = release fail AI-SPEC READY · GATES ARMED

Public install command

Claude Code — add the marketplace and install the orchestration subset:

/plugin marketplace add JasonColapietro/suede-creator-skills
/plugin install suede-agent-workflows@suede

Codex — this is the public route. It installs from GitHub as a standard Codex skill folder.

python3 ~/.codex/skills/.system/skill-installer/scripts/install-skill-from-github.py \
  --repo JasonColapietro/suede-creator-skills \
  --path skills/suede-ai-eval

Restart Codex after installing the skill.

What it produces

AI-SPEC: user promise, inputs, outputs, allowed sources, forbidden behavior, fallback, latency, cost, and success signal.
Failure-mode rubric: severity, likelihood, detectability, current evidence, ship gate, and required fix.
Eval cases: stable IDs, input, setup, expected pass traits, forbidden traits, grade lane, and gate.
Coverage audit: what existing tests, logs, fixtures, prompt snapshots, screenshots, or live readbacks prove today.

Where it fits

Run it before shipping a new AI behavior, after a production miss, or when a product claim says AI quality is handled but the repo only has prompt review or happy-path manual testing.

It pairs naturally with Suede Agent Teams for large launches and Suede Code for implementation review. The eval plan defines what must be true; the code review checks whether the system actually enforces it.

Eval dimensions by system type

Start the failure-mode map from the canonical dimensions for the surface's system type, then add product-specific failure modes on top. Safety and task completion apply regardless of type.

RAG / retrievalContext faithfulness, hallucination, answer relevance, retrieval precision, source citation.

Multi-agentTask decomposition, inter-agent handoff correctness, goal completion, loop detection.

ConversationalTone and style, safety, instruction following, escalation accuracy.

Extraction / structured outputSchema compliance, field accuracy, format validity.

Autonomous / tool-using agentSafety guardrails, tool-use correctness, cost and token adherence, task completion.

Content generationFactual accuracy, brand voice, tone, originality.

Code generationCorrectness, safety, test pass rate, instruction following.

Each dimension gets a measurement approach before the eval case is written: code-based checks for schema and thresholds, an LLM judge for tone and reasoning quality, calibrated against a human-reviewed sample before the score counts as evidence, or human review for edge cases and the calibration itself.

Default tooling

Detected first, not mandated. An existing eval or tracing tool in the repo wins over any default.

Tracing and observability defaults to Arize Phoenix — open-source, self-hostable, framework-agnostic.
RAG eval metrics default to RAGAS — faithfulness, answer relevance, context precision and recall out of the box.
Prompt regression in CI defaults to Promptfoo — CLI-first, no platform account required.
LangChain or LangGraph pipelines default to LangSmith instead of Phoenix.

Coverage scoring

A reference dataset needs at least 10 examples to start, 20+ before coverage counts as production-grade, weighted toward edge cases and known failure modes over happy-path samples.

Coverage and infrastructure score separately, then combine:

coverage = covered dimensions / total dimensions × 100
infra = (tooling + dataset + CI/CD + guardrails + tracing) / 5 × 100
overall = coverage × 0.6 + infra × 0.4

Core gates

Severity 5Legal, financial, rights/provenance, privacy, security, payment, irreversible user harm, or public trust collapse blocks release when uncovered.

Severity 4Core wrong outcome, broken agent action, major cost spike, or misleading published statement needs an eval case, fallback behavior, and owner before release.

RegressionEvery real observed failure needs a fixture, scripted check, or named manual readback path.

Claim truthPublic copy cannot claim eval coverage that does not exist in tests, logs, fixtures, or recorded readbacks.

Best prompts

Use $suede-ai-eval to audit this AI-powered route and produce an AI-SPEC, failure-mode rubric, eval cases, acceptance gates, and missing coverage.

Use $suede-ai-eval on this RAG/search feature. Check stale sources, conflicting docs, missing citations, forbidden claims, and privacy boundaries.

Boundaries

The skill does not claim legal, rights, licensing, medical, financial, or compliance clearance. It does not upload data, call private services, invent datasets, or treat a model's self-judgment as sufficient evidence.