Case Study · Data / AI Infrastructure
Document / RAG Ingestion Pipeline
Extract, chunk, embed, and index documents into a per-tenant vector store — cheap to re-run, safe under partial failure, isolated across tenants.
AWS TextractBedrock TitanOpenSearchStep FunctionsDynamoDB
01
The problem & requirements
Functional
- Ingest documents from multiple sources: upload, S3 drop, web crawl, CMS webhook
- Extract text (OCR for scanned PDFs), chunk, embed, index for retrieval
- Support incremental re-ingestion — an edited doc shouldn't require reprocessing the whole corpus
- Multi-tenant isolation — no tenant's documents leak into another's retrieval results
Non-functional
- Freshness: newly ingested docs searchable within minutes
- Idempotency: re-ingestion must not duplicate vectors or double-bill embedding calls
- Cost control: avoid re-embedding unchanged content
- Durability: raw docs retained in S3 for re-embedding if the model changes
02
Scale & constraints
Assume 10,000 documents, ~10 pages each, ~2,000 tokens/page → ~20,000 tokens/doc.
400,000
chunks for the initial bulk load (512-token chunks with overlap, ~40 chunks/doc)
<15 min
pure embedding time at ~4,200 batched calls, 5 calls/sec — OCR is the real bottleneck
20,000/day
steady-state chunks (500 updates/day) — trivial, but needs a content-hash check first
03
API design
extract(doc) → {text, metadata, page_count}
Textract for scanned PDFs, native parsers for HTML/DOCX.
chunk(text, strategy) → [{chunk_text, position, token_count}]
embed(chunks[]) → [{chunk_id, vector}]
Batched calls to Bedrock Titan / Cohere embeddings.
upsert(tenant_id, chunk_id, vector, metadata)
Writes vector namespace + metadata store, with rollback on partial failure.
query(tenant_id, query_text, k) → [{chunk, score, source}]
Retrieval-side consumer interface.
04
Data model
| Entity | Key fields |
|---|---|
| Document | doc_id, tenant_id, source_uri, content_hash, status, version, last_ingested |
| Chunk | chunk_id, doc_id, tenant_id, chunk_text, token_count, content_hash, position |
| Vector index entry | chunk_id, embedding_vector, embedding_model_version |
| Ingestion job | job_id, doc_id, stage, retries, error, started_at |
content_hash at both doc and chunk granularity is what makes re-ingestion cheap — skip unchanged docs entirely, or re-embed only the chunks that actually changed.
05
Architecture
Sources trigger a Step Functions orchestrator running four stages — extract, chunk, embed, upsert — writing into a per-tenant isolated storage layer.
⇧
Document sources
S3 · crawl · CMS
→
◈
Orchestrator
Step Functions
↓
⇄
Extract
OCR / Textract
✂
Chunk
Semantic split
⬡
Embed
Bedrock Titan
⇩
Upsert
Write vectors
↓
Storage layer · Per-tenant isolated
▲
Vector store
OpenSearch, per-tenant NS
▦
Metadata DB
DynamoDB doc/chunk records
Content-hash dedup cacheDLQ + retriesCloudWatch / X-RayTenant IAM scoping
Fig. 3 — Four-stage pipeline writing into a per-tenant vector store and metadata database.
06
Key decisions & trade-offs
Bottleneck — OCR is the slow stage, not embedding. Parallelize across documents with a Step Functions Map state; skip OCR entirely for text-native PDFs via a fast pre-check.
Cost — re-embedding unchanged content. Hash at doc and chunk granularity; only re-embed chunks whose hash changed.
Trade-off — chunking strategy. Fixed-size is simple and fast but can split mid-thought; semantic/sentence-boundary chunking retrieves better at more compute cost.
Consistency — partial-failure visibility. A doc's status only flips to indexed after every chunk write succeeds; failures route to a DLQ and the doc stays invisible to retrieval until resolved.
Multi-tenant isolation should be structural — separate vector namespaces plus IAM scoping, not just an application-layer filter.
System
Document / RAG Ingestion Pipeline
Primary services
Textract · Bedrock · OpenSearch
Status
Architecture complete
Type
Multi-tenant pipeline