RAG over the vLLM codebase

The problem

A model that has not seen a codebase will guess. RAG fixes this by finding the right files first and giving them to the model. The hard part is search quality: finding the right chunk for a question, quickly.

What I built

• Chunking by structure. Python files are split with the AST at function and class boundaries, so a chunk is a complete unit. Markdown is split by headings.
• Hybrid retrieval: BM25 and TF-IDF indexes with a code-aware tokenizer, merged with Reciprocal Rank Fusion.
• Multi-query expansion. A question becomes up to three variants: the original, the identifiers only, and one with synonyms. This helps when the user and the code use different words.
• Answer generation with Qwen3-0.6B, with the retrieved chunks in the prompt and the model's reasoning text stripped from the output.
• An evaluation framework with Recall@k and other ranking metrics, and a 100-query benchmark. In my runs Recall@5 is about 0.76 on documentation and 0.66 on code, above the project thresholds of 0.55 and 0.45.
• Latency work: BM25S for fast scoring and hash-based caching for repeated queries.

Decisions and trade-offs

Multi-query is slower but only slightly better

Multi-query alone costs 3 to 4.5 times more latency, and the full default (hybrid plus multi-query) about 17 times the cheapest setting. On documentation it adds nothing. On code it helps the hybrid, but plain BM25 with multi-query scores higher, so my default is not the best setting on my own numbers.

Separate stages

Chunking, indexing, retrieval, generation and evaluation are separate modules, so each can be tested and swapped alone.

Limits and next steps

The benchmark has 100 questions, so each question is one percentage point. Results should be read as a range, not an exact figure.