Why RAG Solutions Fail with Complex Documents & Vector Databases

[00:00:00]Humans are complex.

[00:00:01]Because we are complex, our language is complicated.

[00:00:04]It's filled with nuance and subtle differences that only our complex brains can decipher.

[00:00:10]This complexity can be difficult for AI to understand and capture.

[00:00:14]RAG solutions based on complex documents can lead to confusing or even incorrect answers.

[00:00:19]In this video, we'll discuss how to build practical solutions when you have a complex set of documents.

[00:00:25]Let's say you ask a friend.

[00:00:27]Who won the 2020 NCAA football national championship?

[00:00:32]And they reply, Louisiana State University or the LSU Tigers, of course.

[00:00:37]Another friend, however, might say that it was the Alabama Crimson Tide.

[00:00:42]Can they both be right or must one be wrong?

[00:00:46]The correct answer depends on the intent of the original question.

[00:00:50]So, when you talk about a football season, it typically lasts from August... To December.

[00:00:59]But when you talk about championships, that game is usually played the following January.

[00:01:07]This can cause confusion.

[00:01:09]This is why in common language, people will refer to LSU as both the 2019 and the 2020 champions.

[00:01:18]In the real world, sets of documents are compiled by multiple people over time.

[00:01:23]Laws or policies written in the 1990s may contradict with laws or policies written in 2020s.

[00:01:29]People are messy and so is their written history.

[00:01:32]Documents often contradict themselves just like people do.

[00:01:36]It is very likely that inside a repository there are documents, for example, that refer to LSU as both the 2019 and the 2020 champion.

[00:01:46]So what's a RAG solution supposed to do when the document set provides more than one correct answer to a question?

[00:01:53]Let's look at some practical solutions.

[00:01:55]Let's start with a very simple architecture.

[00:01:58]So RAG architecture, very simplistic form, is going to start with an end user.

[00:02:06]This end user is going to ask a question.

[00:02:13]And that question is going to be sent to.

[00:02:18]A vector database.

[00:02:24]Inside the vector database, that question is going to yield search results.

[00:02:28]And then those search results, and the original question.

[00:02:40]Are then passed to an LLM.

[00:02:46]That uses the information from the vector database and the original question to answer the question, which is sent back to the original user.

[00:02:58]So how can you augment this if you're dealing with a complex document set?

[00:03:04]So number one, you need to make sure you don't make any unforced errors.

[00:03:09]And by unforced error, I mean specifically that you don't have documents in this vector database that should not be there.

[00:03:16]So let's take, for example, you have a policy that was implemented in 2019, and another policy that was meant to replace the original policy in 2024.

[00:03:28]You don't want both policies in that vector database, because if you have both of them there, you may confuse your RAG solution.

[00:03:34]So number one, you need a good document management solution to ensure you don't have any of these unforced errors.

[00:03:44]And everything coming out of the search results really should be there.

[00:03:49]The other thing you can do is you can use something called a clarification loop.

[00:03:54]In a clarification loop.

[00:04:04]Is kind of what it exactly sounds like.

[00:04:07]It's a mechanism built into the AI solution so that the right question is asked.

[00:04:14]And you know, this could be something very simple.

[00:04:16]Like if a user is using a RAG solution and they say, why are banana?

[00:04:21]That's a ridiculous question.

[00:04:23]So you want your RAG to identify that as a ridiculous and prompt the user to say, hey, you know what, I don't quite understand this.

[00:04:32]Can you maybe rephrase this?

[00:04:35]But you can also use clarification loops in your process to make sure the question is specific enough.

[00:04:42]So for example, if somebody says, who won the championship in 2010?

[00:04:49]There's obviously a lot of championships out there.

[00:04:52]Is it sports?

[00:04:53]Is it the spelling bee?

[00:04:55]I mean, what is it?

[00:04:55]So that would identify that question as not being specific enough, and it could prompt the user to give more details.

[00:05:02]So at the end result, the answer is specifically what they're looking for.

[00:05:08]But probably the most important thing, and this may seem obvious, but you'd be amazed at how often I see this error, is that the AI... Has to be less than or equal to your data.

[00:05:28]So what do I mean by that?

[00:05:30]So if you ask a question, and according to the documentation in the database, if a human were to read everything in there, they would understand that the answer to that question could be X, Y or Z.

[00:05:45]There's more than one answer to the question.

[00:05:49]So you need to make sure that your AI solution is robust enough to understand this complexity.

[00:05:55]So for example, if I ask the question, it didn't come back with just x or it didn't back with just y, it would come back, well, it could be x, y, or z.

[00:06:04]And where you see this a lot is in policies and laws.

[00:06:10]It's not uncommon for a law written in 2012 to contradict with one written in 1912.

[00:06:19]It's not uncommon if you've got some kind of agency that has both federal and state regulations for those regulations to contradict with each with one another.

[00:06:31]You know, again, it depends on what data you have in the database.

[00:06:35]For example, if you have all you have the database is legal opinions, you don't wanna present those opinions as facts.

[00:06:41]You want to present them in the context.

[00:06:43]Of that they should be interpreted with.

[00:06:48]And what all this means, I think the point here is that you really need to understand the data, the underlying data that you're building your AI solution on before you start to develop that solution.

[00:06:59]Because if you develop a solution, you know, thinking that there's one answer to a lot of questions or to a question, and it turns out there's more than one. If you build a solution, that's... Based on data that's opinion and not fact, and you have that solution yielding information like it is a fact, you're gonna run into problems and people will say it's hallucinating.

[00:07:22]But in reality, it's really not a hallucination, it just is not designed correctly.

[00:07:27]So tell me, what kind of complex document sets are you currently working with?

[00:07:32]Let me know in the comments.

[00:07:33]Thanks so much.