RoleGPSThe free map to your AI job
All capabilities · Retrieval & knowledge systems

Ingest and chunk documents

Parse PDFs/HTML/Docs, clean text, choose chunking strategies, preserve metadata.

~8 focused hoursbeginner
Explore 4 tools for this project
Market relevance

Which roles ask for this — and how often

Share of job postings in India, per role, that name this capability.

Prerequisite capability — not asked for by name, but needed for others. Needed for: Generate embeddings and run vector search.
What employers mean

You should be able to…

  1. Parse PDFs, HTML pages, and Word docs into clean plain text
  2. Choose a chunking strategy (fixed-size, recursive, semantic) appropriate to the document type
  3. Preserve metadata (source, page number, section) alongside each chunk for later citation
  4. Handle messy real-world documents: scanned PDFs, tables, multi-column layouts
  5. Deduplicate and clean noisy text (headers/footers, boilerplate) before chunking
  6. Build a repeatable ingestion pipeline that can be re-run as documents are added/updated

Needs first: Write production-quality Python for AI work

Learn — free, link-checked

The few resources that matter

Read · beginner · 20 min · docs.anthropic.com

PDF Support

Covers sending PDFs directly to a multimodal model as an alternative to a separate OCR/parsing step. — Anthropic
Read · beginner · 30 min · python.langchain.com

Document Loaders

Covers the standard loader interface for PDFs/HTML/Office docs so you don't hand-roll parsing for every file type. — LangChain
Read · beginner · 30 min · docs.llamaindex.ai

Loading Data (Ingestion)

LlamaIndex's ingestion pipeline docs, a second well-maintained option with strong metadata/chunking defaults. — LlamaIndex
Read · intermediate · 25 min · python.langchain.com

Text Splitters

Explains fixed-size vs. recursive vs. semantic chunking tradeoffs, the decision that most affects RAG quality downstream. — LangChain
Tools for practice

Choose a tool for the job

Start with one tool for each part of your project. You don’t need to learn them all.

Go to the practice brief

4 tools to explore

LlamaIndex

Build · Data

Connect documents to retrieval and model workflows with explicit data handling.

Visit toolGetting started

Practices & references

  • Chunking strategies
  • Metadata tagging

Selected for practice by RoleGPS. No paid placements or affiliate links.
Access can change; check the details before starting.

Think your tool is a good fit? Let us know

Practice

OCR-fallback ingestion pipeline for RBI policy PDFs

Point an ingestion pipeline at a folder of RBI master directions and circulars - public PDFs, a mix of clean digital text and older scanned ones - and produce clean, chunked, metadata-tagged text ready for embedding. Fall back to OCR whenever a PDF has no extractable text layer. Every chunk keeps its source filename and page number so an answer can be traced back to the page it came from, and re-running the pipeline over an unchanged folder skips what it already ingested.

Start from

RBI Master Directions and circulars - public PDFs, including older scanned ones with no text layer to force the OCR path

Milestones
  1. Download 10+ PDFs and get clean text out of the digital ones · ~2h
  2. Add the OCR fallback and validate it on a scanned document · ~1.5h
  3. Chunk the text with source filename and page number on every chunk · ~1.5h
  4. Add ingestion state so unchanged files are skipped, and write up the chunking choice · ~1.5h
Done when
  • Pipeline processes at least 10 PDFs including one scanned/image-only document via OCR fallback
  • Each chunk carries metadata (source filename, page number) traceable back to the original document
  • Chunking strategy (size/overlap or semantic) is documented and justified for this document type
  • Re-running the pipeline on an unchanged folder doesn't re-process already-ingested documents
Prove it

Evidence a recruiter can check

  • A chunk pulled from the scanned PDF, with its page number, shown next to the page image it came from
  • Raw extracted text beside the cleaned, chunked version of the same page
  • Run numbers: documents processed, chunks produced, wall-clock time, and how many needed the OCR path
  • The chunking strategy written up with the size and overlap you chose and what these documents did to make you choose it
Signal it

Built a document-ingestion pipeline over public RBI policy PDFs - OCR fallback for scanned files, page-level citation metadata on every chunk, and incremental re-runs that skip already-ingested documents.

Interview

Questions you'll get asked

  1. How do you decide chunk size and overlap for a RAG pipeline over long PDFs?
  2. How would you handle a scanned PDF with no extractable text layer?
  3. What metadata would you attach to each chunk, and why does it matter at query time?
  4. How do you deal with tables and multi-column layouts when extracting text from PDFs?
  5. How would you design an ingestion pipeline that re-processes only changed documents, not everything?
  6. What's the difference between fixed-size chunking and semantic chunking, and when would you pick each?
See which roles to aim at