Troubleshoot and support AI systems in production

AI support lab: break a local LLM API five ways and support it back to health

Run a small model on Ollama behind a thin FastAPI wrapper in Docker Compose, with request ids and structured JSON logs on every call. Inject five realistic failures you can switch on at will: an upstream rate limit, a model timeout, a malformed request, a bad environment variable, and a container memory limit that kills the model. For each one write the ticket as a customer would file it, the log excerpt that diagnoses it, the runbook step that restores service, and a one-page blameless RCA. Finish with a small log dashboard that shows error rate by failure class.

A small open model pulled with Ollama, wrapped in your own FastAPI service; the five failures and their tickets are scripted by you

1. Stand up the lab: Ollama + FastAPI wrapper in Docker Compose, request ids, JSON logs and a /health endpoint · ~4h
2. Inject the five failures behind toggles (rate limit, timeout, malformed request, bad env var, memory limit) and reproduce each on demand · ~5h
3. For each failure: file the ticket, pull the diagnosing log excerpt, write the runbook step that fixes it · ~6h
4. Write five one-page blameless RCAs plus a severity/SLA table and an L1-to-L3 escalation note · ~4.5h
5. Build a log dashboard (Grafana Loki or a small Python viewer) showing error rate per failure class · ~3.25h

• All five failures can be triggered and cleared with a documented command, and each produces a distinct, searchable log signature
• Five tickets, each with severity, reproduction steps, the log excerpt that diagnoses it and the runbook step that resolved it
• Five one-page RCAs with timeline, root cause, contributing factors and at least one action item each
• The dashboard shows error rate by failure class and a request id can be traced from ticket to log line

Evidence a recruiter can check

• Five tickets side by side with the log excerpt that diagnoses each, showing a 429, a timeout, a 400, a bad env var and an OOM kill are told apart from the logs alone
• The runbook, with the exact commands that clear each failure and a note on the one step you got wrong the first time
• Five one-page blameless RCAs, each with a timestamped timeline and an action item
• A dashboard screenshot showing error rate per failure class during a run where you triggered all five
• The Docker Compose file with the failure toggles and memory limit visible as configuration

Built a Dockerised LLM API support lab, injected five production-style failures (rate limit, timeout, malformed request, bad config, OOM) and diagnosed each from logs alone — with tickets, a runbook, blameless RCAs and an error-rate dashboard.