Evaluation, safety & observability
Trace, monitor and debug LLM apps in production
Langfuse/LangSmith/OpenTelemetry traces, token & cost dashboards, drift and failure alerts.
~8 focused hours·intermediate
Tools: Langfuse, LangSmith, OpenTelemetry, Grafana, structured logging
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Instrument an LLM app with traces/spans so every prompt, tool call, and model response is captured with timing
- Track token usage and cost per request, per user, and per feature, rolled up into a dashboard
- Set up alerts for latency spikes, error-rate increases, or cost anomalies in production
- Correlate a user-reported bad response back to its exact trace (prompt, retrieved context, tool calls) for debugging
- Detect drift: a model or prompt update silently changing output quality or latency distribution
- Self-host or configure a tracing backend (Langfuse) vs use a managed one (LangSmith) based on data-residency needs
- Export traces via OpenTelemetry so LLM observability integrates with existing infra monitoring
Needs first: Deploy an AI service to the cloud
Learn — free, link-checked
The few resources that matter
Read · beginner · 25 min · langfuse.com
Overview
Open-source LLM observability platform overview, the most-referenced self-hostable tracing tool in Indian GenAI JDs. — Langfuse
Read · intermediate · 20 min · langfuse.com
Token & Cost Tracking
Shows exactly how per-call token/cost is captured and rolled up into dashboards, the basis for any cost-optimization work. — Langfuse
Read · intermediate · 30 min · docs.smith.langchain.com
LangSmith Observability
Alternative managed tracing stack tightly integrated with LangChain/LangGraph, with cost/latency dashboards out of the box. — LangChain
Read · intermediate · 40 min · langfuse.com
LLM Observability & Application Tracing
Hands-on guide to instrumenting an LLM app with traces/spans/generations for debugging multi-step agent runs. — Langfuse
Read · intermediate · 45 min · opentelemetry.io
Python | OpenTelemetry
Vendor-neutral tracing/metrics SDK underlying most LLM observability tools, worth knowing directly for custom dashboards. — OpenTelemetry
Course · intermediate · 90 min · deeplearning.ai
Evaluating and Debugging Generative AI
Shows how to version prompts/configs and trace experiments so eval regressions are debuggable, not just scored. — DeepLearning.AI (with Weights & Biases)
Practice
Observability dashboard for a multi-tenant RAG support bot
Take an existing (or simple mock) RAG support bot serving multiple clients, instrument every request with Langfuse tracing (prompt, retrieved chunks, model call, tool calls), and build a dashboard showing token cost and p95 latency per tenant per day. Add an alert rule that fires when p95 latency or error rate crosses a threshold, and demonstrate tracing a deliberately bad response back to its root cause.
Done when
- Every request produces a trace with nested spans for retrieval, prompt construction, and model call
- A dashboard (Langfuse UI or exported to Grafana) showing token cost and p95 latency broken down by tenant
- An alert rule configured and demonstrated firing on a simulated latency spike or error burst
- A written walkthrough of tracing one bad response from user report to root cause using the trace
Prove it
Evidence a recruiter can check
- Public GitHub repo with the instrumentation code and a screenshot of the live dashboard
- A short write-up/video of the root-cause debugging walkthrough using a real trace
- Screenshot or export of the alert firing on a simulated incident
Interview
Questions you'll get asked
- How would you debug a single bad response from a production RAG chatbot down to its root cause?
- What would you put on a cost/latency dashboard for an LLM feature, and who looks at it?
- How do you detect that a vendor's silent model update degraded your app's quality?
- Explain the difference between a trace, a span, and a generation in LLM observability tooling.
- How would you set up alerting for a spike in token cost without false-paging on normal traffic growth?
- Why would you choose Langfuse (self-hosted) over LangSmith (managed), or vice versa, for an Indian fintech client?
- How does OpenTelemetry fit into an LLM observability stack that already uses Langfuse or LangSmith?