tiktoken
Build
Count tokens and compare how prompt changes affect the input budget.
All capabilities · LLM application development
Manage context windows and memory
Token budgeting, summarization, conversation memory, caching prompts across turns.
Explore 4 tools for this projectMarket relevance
Which roles ask for this — and how often
Share of job postings in India, per role, that name this capability.
What employers mean
You should be able to…
- Budget tokens so a conversation doesn't silently exceed the model's context window
- Summarize or truncate older turns while preserving the facts that matter
- Use prompt caching to cut latency/cost on repeated system prompts or long documents
- Design conversation memory that persists key facts across sessions, not just within one context window
- Decide what to keep verbatim vs. summarize vs. drop from a long chat history
- Measure and report token/cost usage per conversation for budgeting
Needs first: Design and version prompts systematically
Learn — free, link-checked
The few resources that matter
Read · beginner · 15 min · docs.anthropic.com
Token Counting
Shows how to count tokens before sending a request so you can budget cost and avoid context overflows. — Anthropic
Read · beginner · 20 min · docs.anthropic.com
Context Windows
Clear explanation of how context window limits work in practice, the concept every memory/summarization strategy builds on. — Anthropic
Watch · beginner · 60 min · youtube.com
[1hr Talk] Intro to Large Language Models
Best single hour to actually understand what tokens, context windows and cost are before you start optimizing them. — Andrej Karpathy
Read · intermediate · 25 min · docs.anthropic.com
Prompt caching
Cuts repeated-context cost/latency up to 90%, a concrete, demoable optimization for agent systems with long system prompts. — Anthropic
Course · intermediate · 90 min · deeplearning.ai
Building Systems with the ChatGPT API
Walks through chaining prompts, moderation, and multi-turn memory into one working system, not isolated snippets. — DeepLearning.AI (Andrew Ng, Isa Fulford)
Watch · intermediate · 120 min · youtube.com
Let's build the GPT Tokenizer
Builds a BPE tokenizer from scratch so token-budgeting stops being a black box you guess at. — Andrej Karpathy
Tools for practice
Choose a tool for the job
Start with one tool for each part of your project. You don’t need to learn them all.
4 tools to explore
Claude API
Build · Test
Build model-backed features with messages, tool use and responses you can evaluate.
OpenAI API
Build · Test
Connect model calls, tool use and structured responses to your own application.
Chroma
Data · Build
Index embeddings and metadata, then retrieve relevant records for a query.
Practices & references
- Prompt caching
- Conversation summarisation
- Memory retrieval
Practice
Rolling-memory assistant for 100-turn conversations
Build a chat assistant that holds a 100+ turn conversation without ever hitting the context limit. Drive it from a transcript you script yourself - a borrower asking follow-up questions about a public bank loan FAQ page gives you the repetitive, back-referencing turns real support produces. Combine a sliding window of recent turns with a running summary of everything older, cache the frozen system prompt, and log token usage per turn so you can chart how cost grows across a long session.
Start from
A 100-turn conversation you script yourself - a simulated borrower asking follow-ups about a public bank loan FAQ page, replayed turn by turn
Milestones
- Script the 100-turn conversation and a runner that replays it · ~1h
- Add the sliding window and per-turn token logging · ~1.5h
- Summarise turns as they fall out of the window and refresh on an interval · ~1.5h
- Turn on prompt caching and chart cost per turn across the whole session · ~1.5h
Done when
- A simulated 100-turn conversation completes without hitting a context-length error
- Token usage per turn is logged and a chart/table shows caching reduces repeated-prefix cost after turn 1
- A running summary is regenerated periodically and a spot-check shows it retains key facts from more than 20 turns back
- Configurable window size and summary-refresh interval, documented in README
Prove it
Evidence a recruiter can check
- A cost-per-turn chart across the full 100-turn run, showing where summarisation and caching bend the curve
- cache_read_input_tokens (or the provider equivalent) non-zero in the logs from turn 2 onward
- A recall test: a fact stated at turn 5 answered correctly at turn 90, with both turns quoted
- The summarise-versus-truncate tradeoff written up with the window size and refresh interval you settled on
Signal it
Built a rolling-memory chat assistant that holds 100+ turn conversations inside a fixed context budget - sliding window plus periodic summarisation and prompt caching, with per-turn token cost charted across the session.
Interview
Questions you'll get asked
- How do you keep a multi-turn chatbot from exceeding the model's context window on a long conversation?
- What is prompt caching and when does it actually save cost — walk through prefix stability.
- How would you design memory for an assistant that needs to remember a user's preferences across sessions, weeks apart?
- What's the tradeoff between summarizing old turns vs. just truncating them?
- How do you count tokens before sending a request, and why does that matter for cost and reliability?
- A user's conversation is degrading in quality after 40 turns — how do you debug whether it's a context problem?