How Teams Actually Build RAG (and Where It Quietly Falls Apart)

Ask a dozen teams why they built a retrieval-augmented generation (RAG) system and most will say the same thing: they had a body of documents the model never saw during training, and they wanted the model to answer questions against those documents instead of guessing. RAG has become the default pattern for grounding large language model responses in data the model did not memorize, and it is pragmatic in a way that fine-tuning usually is not.

The other thing is: people get it wrong in predictable places. Here is how RAG actually goes together in production, and where it quietly falls apart.

The pipeline is smaller than the marketing

RAG is conventionally four steps, and only the first is done up front:

1. Convert your documents into embeddings: numeric representations of text chunks that capture their meaning, then index those embeddings in a vector database.
2. A human submits a query in natural language.
3. An orchestrator runs a similarity search over the vector index to find the chunks closest to the query, and adds them to the prompt as context.
4. The model generates an answer using that retrieved context.

That's it. There is no fine-tuning, no training, and if you use the right building blocks, no second database either.

pgvector: the "we don't need another vendor" move

The most interesting shift in the last couple of years is how many teams skip a dedicated vector store entirely. pgvector is a PostgreSQL extension that adds vector similarity search to the database you probably already run. Instead of standing up Pinecone, Weaviate, or Qdrant, keeping a sync layer to copy data into it, and monitoring a second service, you enable an extension and add a vector column.

The structural reasons are strong. Your document metadata, permissions, and tenant IDs already live in Postgres, so splitting retrieval across two systems means duplicating that metadata or doing a two-hop query. With pgvector, updates to a document and its embeddings are atomic in one transaction, and row-level security policies apply to vector searches automatically. For many RAG workloads, the dedicated vector database is the part you can safely not buy.

The embedding dimension depends on the model you pick, and OpenAI's text-embedding-3-small produces 1536-dimensional vectors. That number gets baked into your column type, which is a feature: pin the dimension at the column so an accidental insert from the wrong model fails loudly instead of silently corrupting your index.

For the index itself, pgvector ships two approximate-nearest-neighbor options, IVFFlat and HNSW. For retrieval, HNSW is usually the better default. It uses more memory but gives faster, more accurate search and, unlike IVFFlat, it does not need to be trained on existing data, which matters when your knowledge base grows incrementally.

The thing teams underestimate: chunking

Here is the sentence that should be printed on every RAG whiteboard: chunking strategy affects retrieval quality more directly than the embedding model or the index parameters. A chunk that is too small loses the context that disambiguates meaning. A chunk that is too large dilutes the similarity signal and wastes prompt tokens.

The common mistake is to split on fixed character counts, which can cut sentences in half and destroy meaning. The reliable pattern is to split on sentence boundaries with a little overlap, or to use paragraph boundaries for natural prose. Retrieval quality improves immediately when you stop chopping at arbitrary offsets.

How you chunk also determines how you search. Purely semantic (vector) search is excellent at synonyms and intent, but it can miss exact matches that full-text search catches, like a clause number or a product SKU. That is why production RAG increasingly does hybrid retrieval: run vector search and Postgres full-text search (tsvector, the BM25-style signal) together, then fuse the two rankings. For identifier-heavy questions, hybrid meaningfully beats either signal alone.

Retrieval quality is the bottleneck, not the model

Teams keep swapping the frontier model to fix complaints that are actually about retrieval. The right framing: most production AI products are bottlenecked by what gets retrieved, not by what generates. Get chunking, metadata filtering, and hybrid search right and you can swap models without rewriting anything. Get retrieval wrong and no prompt engineering recovers the trust deficit when an answer cites the wrong source.

The good news is that a grounded RAG system reduces hallucination in a real, structural way, because the model answers from provided passages rather than its memorized guesses. It does not eliminate hallucination, which is why the second-half habits matter: verify retrieved context before you spend tokens on a model call, tell the model to say "I don't know" when the context does not answer, and show the user which source each claim came from.

The honest caveats

A few things nobody puts in the demo:

• Similarity thresholds matter. If you have no threshold, you always return results, even when the knowledge base has nothing relevant. A similarity cutoff keeps you from surfacing garbage confidently.
• Embedding model changes are a migration. If the vector you stored was 1536 dimensions and you switch to a 3072-dimension model, you re-embed. Record which model produced each embedding so you can detect staleness.
• pgvector has a ceiling. Around five to ten million vectors, query performance degrades unless you partition. Most applications never get close, but it is worth knowing the limit exists before you architect around it.
• RAG is a retrieval problem disguised as an AI problem. The model is the easy part. The retrieval is where the real system lives.

Key takeaways

• RAG starts with embedding documents into a vector index, then retrieving relevant chunks at query time and feeding them to the model as context.
• pgvector lets PostgreSQL be both the document store and the vector index, which removes the sync layer and second database for most teams.
• Chunking on sentence or paragraph boundaries with overlap beats fixed character counts, and hybrid (vector + full-text) search catches identifiers that pure vector search misses.
• Retrieval quality, not model size, is the usual bottleneck, and similarity thresholds plus source attribution are what keep a RAG system honest.