LLM application development
Manage context windows and memory
Token budgeting, summarization, conversation memory, caching prompts across turns.
~8 focused hours·intermediate
Tools: Prompt caching (Anthropic/OpenAI), Token counting (Anthropic API / tiktoken), Conversation summarization, Sliding-window memory, Vector-store-backed long-term memory
Market relevance — share of job ads asking for this
What employers mean
You should be able to…
- Budget tokens so a conversation doesn't silently exceed the model's context window
- Summarize or truncate older turns while preserving the facts that matter
- Use prompt caching to cut latency/cost on repeated system prompts or long documents
- Design conversation memory that persists key facts across sessions, not just within one context window
- Decide what to keep verbatim vs. summarize vs. drop from a long chat history
- Measure and report token/cost usage per conversation for budgeting
Needs first: Design and version prompts systematically
Learn — free, link-checked
The few resources that matter
Read · beginner · 15 min · docs.anthropic.com
Token Counting
Shows how to count tokens before sending a request so you can budget cost and avoid context overflows. — Anthropic
Read · beginner · 20 min · docs.anthropic.com
Context Windows
Clear explanation of how context window limits work in practice, the concept every memory/summarization strategy builds on. — Anthropic
Watch · beginner · 60 min · youtube.com
[1hr Talk] Intro to Large Language Models
Best single hour to actually understand what tokens, context windows and cost are before you start optimizing them. — Andrej Karpathy
Read · intermediate · 25 min · docs.anthropic.com
Prompt caching
Cuts repeated-context cost/latency up to 90%, a concrete, demoable optimization for agent systems with long system prompts. — Anthropic
Course · intermediate · 90 min · deeplearning.ai
Building Systems with the ChatGPT API
Walks through chaining prompts, moderation, and multi-turn memory into one working system, not isolated snippets. — DeepLearning.AI (Andrew Ng, Isa Fulford)
Watch · intermediate · 120 min · youtube.com
Let's build the GPT Tokenizer
Builds a BPE tokenizer from scratch so token-budgeting stops being a black box you guess at. — Andrej Karpathy
Practice
Long-conversation loan-support assistant with rolling memory
Build a chat assistant for an NBFC loan-support use case that holds a 100+ turn conversation without exceeding context limits. Implement a sliding window for recent turns plus a running summary of earlier turns, use prompt caching for the frozen system prompt, and log token usage per turn so you can graph cost growth over a long session.
Done when
- A simulated 100-turn conversation completes without hitting a context-length error
- Token usage per turn is logged and a chart/table shows caching reduces repeated-prefix cost after turn 1
- A running summary is regenerated periodically and a spot-check shows it retains key facts from more than 20 turns back
- Configurable window size and summary-refresh interval, documented in README
Prove it
Evidence a recruiter can check
- Public GitHub repo with the memory/summarization logic and a chart of token cost per turn across a long session
- cache_read_input_tokens (or provider equivalent) shown non-zero in logs as evidence caching is actually working
- A test showing a fact mentioned at turn 5 is still correctly recalled at turn 90
- README explaining the summarize-vs-truncate tradeoffs chosen and why
Interview
Questions you'll get asked
- How do you keep a multi-turn chatbot from exceeding the model's context window on a long conversation?
- What is prompt caching and when does it actually save cost — walk through prefix stability.
- How would you design memory for an assistant that needs to remember a user's preferences across sessions, weeks apart?
- What's the tradeoff between summarizing old turns vs. just truncating them?
- How do you count tokens before sending a request, and why does that matter for cost and reliability?
- A user's conversation is degrading in quality after 40 turns — how do you debug whether it's a context problem?