RAG Explained: Everything You Need to Know in 11 Minutes

[00:00:00]This video is sponsored by HubSpot. Rag is one of those AI concepts that sounds simple until you actually try to build it. At a basic level, you retrieve information, pass it to an LLM, and generate an answer. But in production, that pipeline can fail in dozens of subtle ways. It can retrieve the wrong chunks, miss exact information, or give the model so much context that it ignores the answer completely. Hi friends, I'm Maddie. [music] I'm a senior software engineer who previously worked at Google and intern other big tech companies like Amazon, IBM, and Microsoft. And in this video, I'll break down how rag actually works, why basic rag pipelines fail, and the techniques senior engineers use to make them more accurate and productionready. From hybrid search and reranking to query transformation and agentic rack. So, first let's talk about what problem we're actually solving by implementing rag. Large language models like GPT or claude are trained on massive amounts of internet data up to some cutoff date.

[00:00:53]They're quite brilliant generalists.

[00:00:56]However, they have no idea what's in your company's internal wiki. They don't know what happened in last week's earning calls. If you ask them something outside their training data, they won't just say, "I don't know." They'll often just hallucinate. There are two main ways to fix this. One is fine-tuning.

[00:01:12]You retrain the model on your own data.

[00:01:15]That's expensive, slow, and every time your data changes, you have to do it all over again. The other is RAG or retrieval augmented generation. And the core idea behind rag is that instead of baking knowledge into the model's weights, you fetch the relevant information at runtime and hand it to the model as context. Speaking of directing AI precisely, if you want to level up how you actually interact with models, I want to tell you about this free resource called Advanced Tact Prompt Engineering from basic to expert in 7 days. And this is truly worth downloading. It was made by HubSpot, who's sponsoring today's video. Here's why it's relevant. Building a rag system isn't just about the pipeline architecture. It's also about how you prompt a language model at the generation step. A poorly structured prompt can tank your output quality even when your retrieval is perfect. So, this guide is a 7-day playbook that takes you from basic queries to engineering systems. Two things I found particularly useful is that day two covers the roses framework which stands for role objective scenario expected solution steps which is exactly the kind of structured approach that gets consistent outputs from models and day three gets into chain of thought prompting and token optimization which matters if you're building anything at scale where you're making a lot of model calls. My favorite part is the few shot system design section. It's a practical way to calibrate model behavior for specific tasks without fine-tuning anything. So, if you're building with AI and your prompts are still kind of vibe based, this is the resource to grab. It's completely free and the link is in the description. Thanks to HubSpot for sponsoring this one. Okay, let's get back into how Rag works. We'll walk through what actually happens in a rag pipeline. Step one is indexing. Before any user ever sends a query, you pre-process your documents. You split them into chunks and you convert each chunk into a vector embedding using an embedding model like OpenAI's text embedding three small or one of the open- source alternatives from hugging face. These embeddings are highdimensional numeric vectors that capture the semantic meaning of text.

[00:03:16]Chunks with similar meanings end up geometrically closed in that vector space. So for example, the agreement expired in Q3 and the contract last fall would land near each other even with zero word overlap. Then you store all these embeddings in a vector database.

[00:03:33]For example, pine cone, weev8, chromabb, cdrant are all common options. Step two is retrieval. The user sends a query.

[00:03:41]You embed that query using the same embedding model used during indexing.

[00:03:45]Then you do a nearest neighbor search across the vector database to find the chunks whose embeddings are most similar to the query embedding. This is semantic search. It finds documents that mean something related to what the user asked, not just documents that share exact keywords. Step three is generation. You take those retrieved chunks, format them into the prompt alongside the user's original question, and send the whole thing to the language model. The model now has actual grounded specific information to reason from your data, not its training data, and it generates a response based on that.

[00:04:18]Because the source material is right there in the prompt, you can always ask the model to site which chunks it used, which gives you traceability you'd never get from a model generating from memory alone. That's the basic pipeline. And for a lot of use cases like internal FAQ bots, documentation search, support agents over a well structured knowledge base, this works really well right out of the box. However, most real use cases aren't clean FAQ bots, and naive rag has some very predictable failure modes. The first is chunking strategy. If your chunks are too small, you lose context.

[00:04:52]Too large and you dilute the embedding signal. The vector ends up representing five different topics at once and the nearest neighbor search can't reliably match it to any specific query. There's also the question of where you split.

[00:05:04]Naive chunking just cuts at fixed character counts, which means that you'll regularly slice mid-sentence or mid clause. structure aware chunking. So respecting things like paragraph breaks, section headers, even sentence boundaries produces meaningfully better retrieval. The second failure mode is over reliance on pure semantic search.

[00:05:24]Vector search is powerful but it has a blind spot. Exact tokens, product codes, error codes, model numbers, people's names, internal acronyms. These are things where keyword matching crushes semantic similarity. So for example, if an engineer asks why does error 4092 keep firing on the checkout service and that exact string doesn't survive the embedding process cleanly, you might retrieve something thematically related but completely wrong. This is one of the most common ways that rag systems will fail silently. The pipeline runs, the model generates a response and it sounds plausible but it was answering based on the wrong chunks all along. The third failure mode is where research is called lost in the middle. When you retrieve 10 chunks and pass all of them to the model, the model's attention isn't uniformly distributed across that context. Models consistently pay more attention to content at the beginning and end of their context window and underweight information buried in the middle. So even if retrieval worked perfectly and the most relevant chunk was in your top 10, if it lands at position 7, you're banking on the model to fish it out of an intention dead zone. So what do engineers actually do about these issues? There are three upgrades that give you the most return for the added complexity. The first is hybrid retrieval. Instead of relying solely on vector search, you run BM25 in parallel. BM25 is a classic keyword-based ranking algorithm. It excels at exactly the kind of exact token matching that vector search is weak at. So, for example, like I previously mentioned, proper nouns, error strings, these are great for this.

[00:06:55]You run both searches simultaneously and combine the ranked result lists using a technique called reciprocal rank fusion, which merges rankings without needing to normalize scores across two very different scoring systems. Vector search catches semantic meaning and BM25 catches literal matches and combining them consistently outperforms either alone. Recent benchmarks show hybrid retrieval improving recall by about 17% over vectoronly search and on domains with precise technical terminology the gap is even wider. The second upgrade is reranking. Initial retrieval is optimized for speed. You're doing approximate nearest neighbor search across potentially millions of vectors and that means some relevant chunks don't make it into your top results and some irrelevant ones do. After retrieval gives you a candidate pool of say the top 20 or top 50 chunks, you pass those through a cross encoder ranking model.

[00:07:48]Unlike the by encoder used for retrieval, a cross encoder looks at the query and each candidate chunk together with full attention. It's slower but produces much more accurate relevant scores. A standard production pipeline looks like this. You retrieve the top 50 with hybrid search, rerank to top five, and then pass those five to the model.

[00:08:07]This also directly addresses the lost in the middle problem because now you're only passing the most relevant chunks ranked correctly. The third is query transformation. The user's raw question is often a bad retrieval query. People write questions conversationally with pronouns, implied context, and ambiguous references, none of which retrieval systems handle well. Query transformation means having the model itself rewrite or expand the query before retrieval. You might break down a complex question into two or three more focused sub questions and run separate retrievalss for each. Or you can use a technique called hide or hypothetical document embeddings where you ask the model to generate a hypothetical answer to the question first, then use the hypothetical answer as the retrieval query. The intuition here is that a plausible answer to a question looks more like the documents you want to retrieve than the question itself does.

[00:08:59]Hide works well for open-ended questions, but adds latency and an extra LM call. So, it's a trade-off that you tune per use case. There's one more evolution worth knowing about. A dentic rag. The linear pipeline retrieve once, generate once, works fine for simple direct [music] questions. But a lot of real queries don't fit that shape. For example, if someone asks, "How does our refund policy interact with the promo code terms for enterprise accounts that signed up before 2023?" That answer might live across three different documents, require cross referencing to policies, and involve inference that models can't do with a single retrieval pass. In a standard rag system, you get one shot only. If that shot doesn't pull the right combination of context, the model either hallucinates or says it doesn't have enough information. A gentic rag wraps retrieval in a reasoning loop. Instead of retrieve then generate, the agent follows a plan, retreat, evaluate, repeat cycle. It plans which retrieval queries to run, executes them, evaluates whether the results are sufficient to answer the question, and if not, reformulates its queries and retrieves again before generating a final answer. This handles multihop reasoning, contradictory information across documents, and queries that need to synthesize facts from multiple sources. Now, let's talk about why rag might matter for your career. It is the single most deployed AI pattern in enterprise right now.

[00:10:22]Almost every single internal AI assistant, every enterprise chatbot, every document Q&A system, rag is the mechanism under the hood. And the demand for engineers who understand this pattern at depth is not going away. But simply knowing that rag exists is not a differentiator anymore. Most people interviewing for AI engineering roles in 2026 can describe the basic pipeline.

[00:10:44]What separates candidates is layer below that. So understanding the failure modes, knowing which upgrade applies to which problem and being able to build evaluations and a proper evale harness.

[00:10:54]So in conclusion, rag is useful to ground language models in your real specific data by fetching the relevant context at query time instead of relying on what the model memorized during training. The basic pipeline chunk, embed, retrieve, generate gets you started and hybrid retrieval and reanking will get you to production quality. Query transformation handles cases where the raw user question is a bad retrieval signal and agentic rag handles multihop multi-document reasoning that the linear pipeline breaks on. And that's all I have for you in this video. If this was helpful, please hit that like button, hype this video, and subscribe for more tech and AI content. Thanks for watching and I'll see you in the next one.