Evaluation, safety & observability
Optimize inference cost and latency
Model routing, caching, batching, smaller models, quantization/vLLM where self-hosting.
~10 focused hours·advanced
Tools: LiteLLM, vLLM, prompt caching, semantic caching, model routing/quantization
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Route easy queries to a cheaper/smaller model and only escalate hard queries to a frontier model
- Use prompt caching to avoid re-paying for repeated long system prompts or context across requests
- Add a semantic cache so repeated/similar queries are served from cache instead of hitting the model again
- Batch requests where possible to improve throughput and reduce per-request overhead
- Self-host an open model with vLLM (continuous batching, quantization) when API costs exceed self-hosting costs at scale
- Set and monitor a per-feature cost budget, alerting before it's exceeded rather than after the bill arrives
- Quantize a model (int8/int4) to cut memory and latency when self-hosting, and measure the quality trade-off
- Measure and report cost-per-request and p95 latency before and after each optimization to justify it
Needs first: Trace, monitor and debug LLM apps in production
Learn — free, link-checked
The few resources that matter
Read · intermediate · 20 min · langfuse.com
Token & Cost Tracking
Shows exactly how per-call token/cost is captured and rolled up into dashboards, the basis for any cost-optimization work. — Langfuse
Read · intermediate · 20 min · platform.openai.com
Prompt Caching
OpenAI's automatic prompt-caching behavior and pricing, know the platform differences when discussing cost optimization. — OpenAI
Read · intermediate · 25 min · docs.anthropic.com
Prompt caching
Cuts repeated-context cost/latency up to 90%, a concrete, demoable optimization for agent systems with long system prompts. — Anthropic
Read · intermediate · 30 min · docs.litellm.ai
LiteLLM
Unified proxy/SDK for calling 100+ LLM providers with one interface, the standard tool for model routing and fallback in JDs. — LiteLLM (BerriAI)
Read · intermediate · 30 min · docs.litellm.ai
Routing, Fallbacks & Load Balancing
Shows exactly how to route cheap/fast models for easy queries and fall back to stronger models, the core cost/latency lever. — LiteLLM (BerriAI)
Course · intermediate · 90 min · deeplearning.ai
Quantization Fundamentals with Hugging Face
Hands-on quantization (int8/int4) of open models to cut memory/latency when self-hosting instead of calling paid APIs. — DeepLearning.AI (with Hugging Face)
Read · advanced · 45 min · docs.vllm.ai
vLLM
The standard high-throughput inference server (PagedAttention, continuous batching, quantization) for self-hosting open models cheaply. — vLLM Project
Practice
Cost/latency optimization pass on a Hindi-support RAG chatbot
Take a working RAG support chatbot and cut its per-request cost and p95 latency by adding prompt caching for the system prompt/retrieved context, a model-routing layer (small model for FAQ-style queries, larger model for complex ones via LiteLLM), and a semantic cache for repeated questions. Measure and report cost-per-request and latency before and after each change.
Done when
- Prompt caching enabled for the repeated system prompt/context, with a measured cache-hit rate
- A LiteLLM-based router sending at least two query classes to two different-cost models, with routing accuracy reported
- A semantic cache serving repeated/similar queries, with a safeguard against serving stale/wrong answers
- A before/after report showing cost-per-request and p95 latency reduced by a measurable, stated percentage
Prove it
Evidence a recruiter can check
- Public GitHub repo with the routing/caching code and a README with before/after benchmark numbers
- A cost-per-request and latency chart (even a simple table) comparing baseline vs optimized
- Langfuse or equivalent trace screenshots showing cache hits and routed model choice per request
Interview
Questions you'll get asked
- How would you cut a chatbot's per-request cost by 50% without noticeably hurting quality?
- Explain how prompt caching works and where it gives the biggest win.
- When does it make sense to self-host with vLLM instead of calling a hosted API, and what does that trade off?
- How would you design a model-routing layer that sends easy queries to a small model and hard ones to GPT-4/Claude?
- What's the risk of an aggressive semantic cache, and how do you avoid serving a stale or wrong cached answer?
- How do you quantify the latency/cost impact of quantizing a self-hosted model?
- Walk me through a real cost-optimization you shipped — before/after numbers?