RoleGPSPath to your AI role
Agents & workflows

Evaluate and harden agents

Trajectory evals, cost/latency budgets, failure taxonomies, sandboxing and permissioning.

~12 focused hours·advanced

Tools: LangSmith, DeepEval, pytest, trajectory/step-level scoring, cost/latency budgets

Market relevance — share of job ads asking for this
Prerequisite capability — not asked for directly, but needed for others.
What employers mean

You should be able to…

  1. Score an agent's full trajectory (sequence of tool calls and reasoning steps), not just the final answer
  2. Build a failure taxonomy (wrong tool chosen, hallucinated argument, infinite loop, unsafe action) and tag failures against it
  3. Set and enforce cost and latency budgets per agent run, failing or degrading gracefully when exceeded
  4. Sandbox agent tool execution (filesystem, network, shell) so a misbehaving agent can't cause real damage
  5. Add permissioning/approval gates for high-risk tool calls and test that they actually block unauthorized actions
  6. Write regression tests that catch when a prompt or model change breaks a previously-working agent trajectory
  7. Run agents against adversarial or edge-case inputs to find where the loop breaks

Needs first: Build a multi-step agent workflow, Build an LLM evaluation harness

Learn — free, link-checked

The few resources that matter

Read · intermediate · 25 min · anthropic.com

Writing effective tools for AI agents

Practical guidance on tool naming, error messages and token-efficient outputs that make agent tool use reliable in production. — Anthropic
Read · intermediate · 30 min · anthropic.com

Building Effective AI Agents

Defines the workflow-vs-agent patterns (routing, orchestrator-worker, evaluator-optimizer) employers expect you to name and choose between. — Anthropic
Read · intermediate · 30 min · docs.smith.langchain.com

LangSmith Evaluation

Explains golden datasets, LLM-as-judge evaluators and online vs offline evals, the vocabulary used in eval-harness interviews. — LangChain
Read · intermediate · 35 min · docs.confident-ai.com

DeepEval - The LLM Evaluation Framework

Pytest-style LLM eval framework with built-in agent/trajectory metrics, plugs straight into a CI regression suite. — Confident AI
Practice

Trajectory eval harness for a support-ticket triage agent

Take the loan/ticket triage agent from the build-agent-workflow project and build a harness that replays 20 recorded conversations, scores each trajectory (right tools called, right order, no unsafe action taken) against a failure taxonomy, and enforces a per-run token/cost cap. Fail the CI run if trajectory accuracy drops below a threshold or any run exceeds its budget.

Done when
  • 20+ recorded trajectories with expected tool-call sequences as golden references
  • A scoring function that flags wrong-tool, wrong-order, and unsafe-action failures separately (a failure taxonomy, not just pass/fail)
  • A hard cost/latency budget enforced per run, with runs that exceed it marked as failed
  • The harness runs in CI (or a script) and produces a pass/fail summary with per-category failure counts
Prove it

Evidence a recruiter can check

  • Public GitHub repo with the eval harness code and a sample CI run output/report
  • A written failure taxonomy doc listing the categories you score against
  • A before/after comparison showing the harness catching a regression you deliberately introduced
Interview

Questions you'll get asked

  1. How do you evaluate an agent when there's no single 'correct' final answer, only a good-enough trajectory?
  2. Describe a failure taxonomy you'd build for a customer-support agent, and how you'd tag failures automatically.
  3. How do you sandbox an agent that can run shell commands or hit the filesystem?
  4. How would you set and enforce a per-request cost budget for an agentic workflow that can call tools recursively?
  5. What's the difference between step-level and outcome-level evaluation for agents?
  6. How do you catch an agent that gets stuck in a loop before it burns through your token budget?
  7. Tell me about a time an agent misbehaved in a way your evals didn't catch. What did you change?