Retrieval & knowledge systems
Ingest and chunk documents
Parse PDFs/HTML/Docs, clean text, choose chunking strategies, preserve metadata.
~8 focused hours·beginner
Tools: LangChain/LlamaIndex document loaders, pypdf/pdfplumber/Unstructured, Text splitters (recursive/semantic chunking), BeautifulSoup for HTML, Metadata tagging
Market relevance — share of job ads asking for this
Prerequisite capability — not asked for directly, but needed for others.
What employers mean
You should be able to…
- Parse PDFs, HTML pages, and Word docs into clean plain text
- Choose a chunking strategy (fixed-size, recursive, semantic) appropriate to the document type
- Preserve metadata (source, page number, section) alongside each chunk for later citation
- Handle messy real-world documents: scanned PDFs, tables, multi-column layouts
- Deduplicate and clean noisy text (headers/footers, boilerplate) before chunking
- Build a repeatable ingestion pipeline that can be re-run as documents are added/updated
Needs first: Write production-quality Python for AI work
Learn — free, link-checked
The few resources that matter
Read · beginner · 20 min · docs.anthropic.com
PDF Support
Covers sending PDFs directly to a multimodal model as an alternative to a separate OCR/parsing step. — Anthropic
Read · beginner · 30 min · python.langchain.com
Document Loaders
Covers the standard loader interface for PDFs/HTML/Office docs so you don't hand-roll parsing for every file type. — LangChain
Read · beginner · 30 min · docs.llamaindex.ai
Loading Data (Ingestion)
LlamaIndex's ingestion pipeline docs, a second well-maintained option with strong metadata/chunking defaults. — LlamaIndex
Read · intermediate · 25 min · python.langchain.com
Text Splitters
Explains fixed-size vs. recursive vs. semantic chunking tradeoffs, the decision that most affects RAG quality downstream. — LangChain
Practice
Ingestion pipeline for NBFC loan policy PDFs
Build an ingestion pipeline that takes a folder of NBFC/bank loan policy PDFs, a mix of clean and scanned, and produces clean, chunked, metadata-tagged text ready for embedding. Handle at least one scanned/image-based PDF via OCR fallback, and preserve page-level citations on every chunk.
Done when
- Pipeline processes at least 10 PDFs including one scanned/image-only document via OCR fallback
- Each chunk carries metadata (source filename, page number) traceable back to the original document
- Chunking strategy (size/overlap or semantic) is documented and justified for this document type
- Re-running the pipeline on an unchanged folder doesn't re-process already-ingested documents
Prove it
Evidence a recruiter can check
- Public GitHub repo with the pipeline code and a sample of input PDFs plus output chunks with metadata
- A before/after example showing raw extracted text vs. cleaned/chunked text
- README documenting the OCR fallback and how it was validated
- Numbers: docs processed, chunks produced, time taken
Interview
Questions you'll get asked
- How do you decide chunk size and overlap for a RAG pipeline over long PDFs?
- How would you handle a scanned PDF with no extractable text layer?
- What metadata would you attach to each chunk, and why does it matter at query time?
- How do you deal with tables and multi-column layouts when extracting text from PDFs?
- How would you design an ingestion pipeline that re-processes only changed documents, not everything?
- What's the difference between fixed-size chunking and semantic chunking, and when would you pick each?