Building Adaptive RAG on Google Cloud

Where one-shot RAG breaks

The first version did what most tutorials do. Embed the question, pull the nearest documents, paste them into the prompt, ask the model to answer. On a small corpus with well-separated topics, that works.

Ours was neither. A frontier model reading whole documents, with no decomposition, landed at 44,322 input tokens per query, a source hit-rate of 0.892, key-fact recall of 0.848, and 7.6 s median latency against a 10.8 s p95. Cost was not instrumented yet, so that build has no price.

The token count is the tell. Putting whole documents in the prompt is not retrieval, it is handing the search back to the model and paying for it. More context does not raise the ceiling either. A four-part question needs evidence from four places, and one similarity search returns one neighbourhood, matched to the whole question rather than any part of it.

Two things shaped everything below. Multi-part questions, where one retrieval pass cannot reach every part. And vocabulary mismatch, where the user knows the exact identifier and the embedding barely weighs it.

The corpus, and what makes it hard

The system indexes a public documentation corpus: 11,129 documents that split into 100,515 chunks. It is all HTML, crawled with aiohttp, parsed with BeautifulSoup over lxml, and walked recursively into plain text shaped like markdown. Headings become ##, list items - , table rows become pipe-delimited lines, with navigation, headers, footers and scripts stripped first. No PDF anywhere in the pipeline, which removes an entire category of parsing problems.

What makes this corpus hard is not its size. It is that most of it is templated. Thousands of record pages share the same section headings in the same order, run to similar lengths, and differ in a handful of fields: an identifier, a term, a credit count, a name.

That is a problem for dense retrieval in particular. To a bi-encoder, two pages that share ninety percent of their text and differ only in the identifier at the top sit at almost the same point in space. The one field that decides whether an answer is right is the field the embedding pays least attention to. Everything in the retrieval design below is built around that.

Chunking: 1,024 characters, 130 overlap

We split with LangChain's RecursiveCharacterTextSplitter at chunk_size=1024, chunk_overlap=130, with the separator ladder ["\n\n", "\n", ". ", " ", ""]. One configuration for the whole corpus, with no per-document-type tuning.

One detail trips people up: length_function=len counts characters, not tokens. No tokenizer is involved in chunking at all. For English prose 1,024 characters lands near 256 tokens, and the 130-character overlap is 12.7%, inside the range the published benchmarks recommend, only counted a different way. Copy a token count from a paper into a character-based splitter and you get chunks roughly four times larger than you intended, with no error to tell you.

Two limits set the size. Going larger runs into the embedding model: gemini-embedding-001 accepts at most 2,048 input tokens per text, so a chunk that runs much past a thousand tokens gets silently cut at the top of the pipeline. More overlap costs you twice. The index grows, and the same passage returns several times in one result set, crowding out the variety the retrieval stage was there to provide. Past roughly 20–25% that second cost outweighs the extra recall.

Each chunk carries metadata beyond its text: source path, category, canonical URL, page title, a chunk_index ordinal, and identifiers and numeric fields pulled out by regex at ingest. That metadata is not decoration. The lexical retriever reads from it, and so does the reranker.

Embeddings, and the truncation trap

Embeddings come from gemini-embedding-001 on Vertex AI, pinned to us-central1 because the embedding endpoint is regional while the chat models we use are global. The model emits 3,072 dimensions natively. We store 1,536.

That truncation is safe because the model is trained with Matryoshka Representation Learning, which trains the vector so that any prefix of it still works as an embedding. Google's own figures put MTEB at 67.99 for 768 dimensions against 68.16 at 2,048, a quarter of the storage for a difference that rounds away. We took 1,536 rather than 768 because pgvector's HNSW index caps at 2,000 dimensions, and while that index still existed, 1,536 was the largest useful value under it.

Here is the part that costs people weeks. When you request fewer than 3,072 dimensions, the API returns vectors that are not normalised. Google's documentation states it plainly, and it is easy to skim past, because nothing fails. Cosine similarity over unnormalised vectors still returns numbers, still ranks documents, still looks like it works. It is just measuring the wrong thing, because length leaks into a comparison that is supposed to be about direction. Our wrapper divides every batch by its row-wise L2 norm before anything downstream sees it, guards the zero-norm case, and does this on both the document and the query path.

The document path and the query path differ on purpose. task_type is RETRIEVAL_DOCUMENT at ingest and RETRIEVAL_QUERY at search. The same sentence embeds differently depending on which side of the search it sits on, which is the point: a question and the passage answering it do not look alike, and naming the role lets the model place them near each other anyway.

There is one hard limit worth knowing before you design around it. gemini-embedding-001 accepts exactly one instance per request, and the LangChain Vertex client will happily hand it a whole list, which returns a 400. Batching has to happen at the thread level instead: 24 workers each embedding a sequential slice, inserting 500 rows at a time, sustaining roughly 20 chunks per second, or about an hour and a half for 100,000 chunks.

Retries matter more than you would expect at that concurrency. The waits are 2, 8, 20 and 40 seconds, each with up to 25% jitter, and only RESOURCE_EXHAUSTED and 429 retry. A 400 or 403 raises immediately instead of sleeping through four attempts. The jitter is not cosmetic: twenty-four workers that hit a per-minute quota wall together will, without it, wake together and hit it again. The client library's own retry gives up well before a per-minute quota refills, which is why we wrote our own.

Postgres as the store, then dropping the vector index

Vectors live in Cloud SQL for PostgreSQL with the pgvector extension. Choosing Postgres over a dedicated vector database was about how much there is to run, not about benchmarks: one store holds the vectors, the chunk text and the JSONB metadata together, one transaction covers a write to all three, and one backup covers the lot. A separate vector service means a second thing to keep in sync and a second thing to restore.

The embedding table holds vector(1536), the chunk text, and a JSONB metadata column with a GIN index over it. Row IDs are deterministic UUIDv5 values derived from the collection name, source path and chunk index, so a re-run upserts in place instead of duplicating.

Then we deleted the HNSW index.

It had been the obvious thing to build: USING hnsw (embedding vector_cosine_ops), the standard approximate-nearest-neighbour setup every pgvector guide tells you to add. It was also 1,992 MB, 47% of the entire database. What replaced it is not another index. It is a different first stage.

query
  |
  +-- BM25 sweep over the whole corpus  -->  200 candidate ids
  |                                             |
  |                          embeddings fetched BY PRIMARY KEY
  |                          (one batched SELECT ... id = ANY(:ids))
  |                                             |
  |                                   MMR in numpy, lambda 0.87  -->  23 docs
  |
  +-- BM25 top-k                                                 -->  23 docs
                                                  |
                                   Reciprocal Rank Fusion (k=60, alpha=0.48)
                                                  |
                                              25 candidates

The only index left on the read path is the primary key. Every measured metric moved the right way.

The reason it works is the corpus, not a flaw in HNSW. An ANN index answers "what is near this vector," and on templated near-duplicates near is the wrong question. The nearest neighbours of a record page are the several hundred other record pages sharing its boilerplate. A wide lexical sweep answers "what actually contains these terms," which is a far better candidate set to run vector math over. And once that set is two hundred rows, fetching their embeddings by primary key costs one batched round trip. There is nothing left for an approximate index to accelerate.

BM25 in process, and why not Postgres full-text search

The lexical arm is rank_bm25's BM25Okapi, running in process, returning 23 documents. There is no tsvector, no ts_rank, no to_tsquery anywhere in the system.

That is deliberate, and worth being clear about, because "Postgres full-text search" and "BM25" get used interchangeably and are not the same thing. ts_rank and ts_rank_cd score term frequency within a document, weighted by position and by the labels you assign. Neither sees corpus-wide document frequency. There is no IDF term. So a rare identifier that appears in four documents out of a hundred thousand and a common word that appears in eighty thousand count for about the same, which is exactly backwards for this workload. Postgres full-text search is a good matcher and a weak ranker. Treating it as both is a mistake you only notice once you start tuning.

One detail did more for retrieval quality than any parameter sweep: the BM25 index is fitted on augmented text. Before indexing, each chunk gets its page title, category, filename keywords and URL slug prepended. The stored document, the one that eventually reaches the model, is left byte-identical. Only the index sees the augmentation.

That makes the lexical arm aware of what a page is, not only what it contains, with a side effect we did not expect: it made a HyDE-style query rewriter redundant. Generating a hypothetical answer document to bridge the vocabulary gap buys nothing once the index already sees what each page is. That experiment was measured and dropped.

The tradeoff of in-process BM25 is that the index lives in RAM and gets rebuilt at startup. At 100,000 chunks that is fine. At ten million it would not be, and a purpose-built search service would start to make sense again.

Fusing the two arms: RRF at k=60

Two ranked lists arrive, one from the dense arm and one from BM25, and they have to become one. The problem is that a cosine distance and a BM25 score are not on the same scale and never will be.

RRF(d)  =  alpha * 1/(k + rank_dense(d))
        +  (1 - alpha) * 1/(k + rank_bm25(d))

        k = 60          alpha = 0.48

MMR(d)  =  lambda * sim(d, q)  -  (1 - lambda) * max sim(d, d_selected)

        lambda = 0.87

Normalising the scores (min-max, z-score, anything like them) makes fusion depend on the score distribution of whichever query you happened to run. A query whose top dense result scores 0.91 and one where it scores 0.62 blend completely differently, for reasons that have nothing to do with which documents are correct. RRF sidesteps this by discarding the scores and keeping only placement. Rank one is rank one regardless of what produced it.

The constant k = 60 comes from Cormack, Clarke and Büttcher's 2009 SIGIR paper, tuned on TREC and widely reused since. It controls how sharply the top of each list dominates: raise it and the curve flattens so deeper results get a say, lower it and rank one swamps everything. We swept 40 and 80 against it; 60 held.

The weighting is the interesting number. At alpha = 0.48, BM25 carries the heavier weight at 0.52, the opposite of most RAG defaults, which lean dense. This follows directly from the corpus. Questions arrive containing exact identifiers, and identifiers are precisely what a bi-encoder blurs away. The lexical arm is not a fallback here; it is the arm that knows which of five hundred nearly identical pages the user meant. We swept alpha against 0.40, 0.56 and 0.60; 0.48 won.

Maximal Marginal Relevance runs inside the dense arm at lambda = 0.87, selecting for relevance minus redundancy so that three near-identical chunks cannot eat the context window between them. The value is much higher than the 0.5 most implementations default to, and again the corpus explains it. When most pages genuinely resemble each other, a strong diversity term starts evicting documents that are relevant and similar, which is a large fraction of the correct answers. At 0.87 the redundancy penalty acts as a tie-breaker rather than a driver.

Reranking, and a latency tail worth hedging

Twenty-five fused candidates go to the Vertex AI Ranking API, model semantic-ranker-default-004, and ten come back. It is a cross-encoder: it reads query and document together in one pass instead of comparing two independently produced vectors, which lets it judge relationships a bi-encoder cannot see. The cost is that nothing can be precomputed, so it only makes sense on a short list. The API takes 200 records and 200k tokens per request with a 1,024-token ceiling per record; we send page identity in the title field so the ranker sees what each record is, not only what it says.

Then production traces showed something the documentation does not mention.

A 136 ms median against a p95 of nearly 28 seconds is not a load problem, and we ruled out the usual suspects: it does not track request volume, machine contention, cold connections, or the number of records in the call. It is a server-side tail, and it is not ours to fix.

So we hedge. If a ranking call has not answered within 400 ms, an identical duplicate request goes out and whichever response lands first wins. The median sits well under that threshold, so most calls never trigger it.

The agent loop: two nodes, three hops

The adaptive layer is LangGraph, and it has exactly two nodes. Most agentic RAG diagrams have six or seven: grade documents, rewrite query, check for hallucination, decide to retry. We tried that shape and collapsed it, because each of those nodes is a separate LLM call making a separate decision the same model could have made once.

       START
         |
         v
   +------------+   action = retrieve    +----------+
   | controller | ---------------------> | retrieve |
   +------------+ <--------------------- +----------+
         |
         | action = answer | refuse,  or hop >= 3
         v
        END

   hop cap: 3        recursion_limit: 14

The controller node is a single LLM call returning one JSON object: an action of answer, refuse or retrieve, a reason, a list of queries, and a list of what is still missing. The same node, the same schema, does two different jobs depending on when it runs. On hop zero, its decision is the route: answer directly, decline, or go and look. On hop one and after, its decision is the coverage gate: is there enough evidence now, and if not, what specifically is absent.

The retrieve node also behaves differently by hop. On the first pass it runs the full rewrite, decomposition and fan-out. On later passes it takes the controller's named queries, retrieves against them, and merges the hits into the documents already accumulated in state.

The controller never sees full documents. It sees a digest built from page identity plus structured Label | value lines, capped at 400 characters per source across at most twelve sources. A router that reads everything is just a slower generator.

Every guard that could have been a prompt instruction is a code branch instead, and that is what made the loop reliable. A concreteness check rejects vague targets. A controller reporting it is missing "more information" gets its retrieve downgraded to answer, because "more information" is not a query and looping on it is how agents burn twelve hops and return nothing. A follow-up guard forces answer to retrieve when a conversation has history but nothing was retrieved this turn. A retrieve naming no queries is downgraded too. Prompt instructions are advisory; a conditional edge is not.

The hop cap is 3, enforced on the edge. LangGraph is invoked with a deliberately generous recursion_limit of 14, so that our cap fires first and returns partial results with a log line, rather than the framework raising an exception into a user's face. In production, 8% of turns hop at all. The loop is there for the questions that need it, and stays out of the way of the ones that do not.

Map-reduce over sub-questions

The last major change came out of a problem that was hard to see and worse than it looked.

Pooling everything retrieved for a multi-part question into one context and asking a single call to answer all of it dropped roughly 4 of 48 facts per run, a different four each time. Dropping a different fact every run is worse than dropping the same one every time, because you cannot test your way out of it: the case that passes today fails next week for no reason you can reproduce. Worse, in one run 7 of 12 answers invented a value outright rather than reporting it missing. An LLM judge scored that path's faithfulness at 0.983 on one run and 0.862 on the next.

The fix was to stop asking one call to hold four questions in its head at once.

question
   |
   +-- rewrite + decompose  -->  up to 4 sub-questions (hard cap: 8 groups)
                                    |
        +---------------------------+---------------------------+
        |                           |                           |
   sub-question 1              sub-question 2              sub-question N
        |                           |                           |
   dense 23 + BM25 23          dense 23 + BM25 23          dense 23 + BM25 23
   RRF -> 25                   RRF -> 25                   RRF -> 25
   own rerank call -> 10       own rerank call -> 10       own rerank call -> 10
   own generation call         own generation call         own generation call
        |                           |                           |
        +---------------------------+---------------------------+
                                    |
                        ONE reduce call (8,192 output tokens)
                                    |
                                 answer

                            8 workers in parallel

The grounding here comes from the structure, not from the prompt. The reduce call never sees a retrieved document, only the notes each map call wrote, with the URLs behind them. It therefore cannot cite a source that no map call actually used. Citation precision has measured 1.000 across every archived evaluation run, and this is why.

Latency going down is the surprising part, and it has two causes. Eight parallel map calls replace one long serial generation over a much larger context. And because the first pass now surfaces evidence for every part of the question, the controller stops needing to hop, falling from 0.75 to 0.25 on four-part questions, with each avoided hop being a full retrieval round trip.

The cost is real and we took it knowingly: +31% on general traffic, scaling at roughly +$0.0037 per additional part, near-linearly. It buys nothing on simple questions. Only 6 of 24 turns in the general set decompose at all, and a single-sub-question turn runs one map call and skips the reduce step entirely, making it identical to the old path. We are paying a third more on every question to stop a minority of questions from silently fabricating values. That is a trade we would make again.

The cheap model won

Generation and the controller both run on the flash-lite tier at temperature 0.05 with thinking set to minimal. Only the rewriter and the evaluation judge run on the larger flash tier. This was not a budget compromise; it is what the measurements said to do, three separate times.

The explanation is that once retrieval is doing its job, generation stops being a reasoning task. Ten reranked chunks and a question that names what it wants: the answer is either in that context or it is not. A model with more reasoning headroom will spend it, and reasoning over retrieved text is exactly how you get a fluent, confident, plausible value that appears in none of the documents. The fabrication rate we measured on the pooled path was not a small-model problem. Constraining the generator is a feature.

That turns the usual advice about RAG cost around. You do not make a RAG system cheaper by shrinking the generator and hoping. You make retrieval good enough that shrinking the generator costs you nothing.

Performance and cost

Cost and answer quality both track one thing: how much searching the turn took. Every part of a question gets its own search, its own ranking pass and its own answer, and when the first round comes back missing something the system searches again before it answers. So the rows below count searches, not the questions someone typed. A four-part question answered in one round is four. A two or three-part question that needed a second round is five or six.

URL hit rate is the share of the pages a case requires that the system actually retrieved. Key-fact recall is the share of the facts a case requires that the answer states. Both are scored against sets written from the corpus, so a miss is a miss and not a disagreement about wording. Citation precision, the share of cited links that came from evidence actually retrieved, is 1.00 across every run behind this table, and out-of-scope and unanswerable questions were declined rather than answered in all of them.

Token counts cover the whole turn, not just the final call: routing, query rewriting, one answering call per part, and the call that composes them. Cost rises almost linearly with parts, about $0.0037 each, because every part buys its own ranking call and its own answering call. Time does not. Three times the work costs 1.7 times the wait, because the ranking and answering calls for every part are in flight at once, and that stage takes one to two seconds no matter how many parts there are. What grows instead is the routing overhead on turns that had to search a second time, which is what the five-to-seven row is made of.

A greeting or an out-of-scope question skips search entirely and answers in about two seconds for six hundredths of a cent. An exact repeat of an earlier question returns from cache immediately at no cost.

Times are medians measured off the production endpoint and a local harness that agree with each other for one to three parts: 5 to 7 seconds to the first token of a single answer, 9 to 13 for two or three parts. Turns delayed by an embedding-quota backoff during back-to-back benchmark runs are excluded from the median, because the benchmarking caused them and the pipeline did not. The tail is real and sits in the model calls rather than in retrieval: the slowest single question sampled the same afternoon took twenty seconds.

All of it runs on one small CPU-only virtual machine. The heavy stages, meaning embeddings, ranking and generation, are managed services on Google Cloud's Gemini Enterprise Agent Platform, which is what Vertex AI is now called. Authentication is workload identity end to end, so there is no API key in the codebase or the environment.

How we know any of this

You cannot see any of this without measuring it, and most of it survives a careful read of the outputs. The harness runs five golden sets (40 clean questions, 25 deliberately messy and multilingual ones, 24 targeted cases, 28 multi-part cases, and 14 entity-lookup cases), scoring source hit-rate, key-fact recall against alternative-phrasing groups, citation precision, a forbidden-content avoidance check, and optional LLM-judged faithfulness.

Everything runs in pairs. Same configuration, twice. A single run of a nondeterministic system tells you very little, and the map-reduce decision came out of a pair rather than a run. The deciding evidence was that two runs of the pooled path disagreed with each other about which facts to drop. A single run would have shown a respectable 0.983 and we would have shipped nothing.

The harness also found a bug in itself. The fact-recall matcher was under-counting; rescoring 49 archived runs moved recall up by 0.025 to 0.070, a mean of +0.036. The number that matters is the second one: across 816 pairwise configuration comparisons, zero rankings inverted. Every decision made on the wrong numbers was still the right decision. That is a claim you can only make after going back and checking, and checking is the part most teams skip.

Failed experiments are kept with their numbers too, which is what stops the same idea being tried twice. A fan-in retention step fired 0 times out of 12 and was removed. A reranker score threshold of 0.2, which looked like precision for free, cut multi-part recall from 0.819 to 0.611.

Engineering decisions

1. Measure your index before you assume it earns its keep. The HNSW index was 47% of the database and dropping it improved hit-rate, recall, cost and latency at once. On a corpus of templated near-duplicates, a wide lexical sweep is a better first-stage filter than approximate nearest neighbours.
2. Truncated Matryoshka embeddings come back unnormalised. Request fewer than 3,072 dimensions from gemini-embedding-001 and you must normalise them yourself. Nothing errors if you forget. Similarity search just quietly measures the wrong thing.
3. Postgres full-text search is not BM25. ts_rank has no corpus-wide IDF, so rare identifiers and common words count for about the same. If you need real lexical ranking, use a real ranking function.
4. Index page identity, serve page content. Prepending title, category and URL slug to the text the BM25 index sees, while leaving the stored document untouched, did more for retrieval than any parameter sweep, and made a HyDE-style rewriter redundant.
5. Fuse on rank, not on score. Cosine distance and BM25 scores are not comparable, and normalising them makes fusion depend on each query's score distribution. RRF at k=60 discards the scores and keeps the ordering.
6. Let the corpus set the weights. BM25 outweighing dense retrieval at 0.52 to 0.48, and MMR at λ=0.87 instead of 0.5, both look wrong against the defaults and are both right here, for the same reason: near-duplicate pages defeat the assumptions the defaults were built on.
7. Hedge undocumented latency tails. The Vertex ranking call has a 136 ms median and a 28 s p95. Firing a duplicate request after 400 ms and taking the first response halved the worst case for a second call on a small minority of requests.
8. Put agent guards in code, not in the prompt. Hop caps, concreteness checks and follow-up rules all belong on conditional edges. A prompt instruction is advisory; an edge is not. Two nodes proved sufficient. The extra nodes in most agentic RAG diagrams are decisions one model can make in one call.
9. Fan out over sub-questions so facts stop dropping at random. One call answering four questions dropped a different four facts every run and sometimes invented values. Giving each sub-question its own retrieval and generation call raised recall, cut latency by more than half, and made a fabricated citation impossible.
10. Better retrieval is what makes a cheap generator viable. The lite tier cut cost 69% at parity, beat the larger model as a router on messy phrasing, and turning thinking off improved recall while cutting p95 latency. Constraining the generator is a feature once the context is right.
11. Run evaluations in pairs. The single most important finding in this project, that pooled generation silently drops a different subset of facts each run, is invisible in a single run, which scored 0.983.