# RAG chatbot explained: how retrieval-augmented generation works

URL: https://instantknowledgebot.com/learn/rag-chatbot-explained  
Last updated: 2026-10-14

> A RAG chatbot uses retrieval-augmented generation: for each question it first retrieves the most relevant passages from a set of documents, then gives them to a large language model, which writes an answer grounded in that text. RAG lets a chatbot answer from your own, current content without retraining the AI model.

A **RAG chatbot** is a chatbot built on retrieval-augmented generation. Instead of relying only on what an AI model learned during training, it **retrieves** relevant passages from your documents for each question, then **generates** an answer from them. That's how a chatbot can answer "What's your cancellation policy?" with your exact policy, even though the AI model has never seen your business before.

## Why do chatbots need RAG at all?

Large language models (LLMs) are trained on huge amounts of public text up to a cut-off date. They're excellent at language, but:

- **They don't know your business:** your prices, policies, opening hours or products.
- **Their knowledge is frozen** at training time.
- **When they don't know, they may guess**, producing fluent but wrong answers (often called hallucinations).

RAG solves the first two problems directly and greatly reduces the third: the model is handed the facts it needs, at the moment it needs them.

## How does retrieval-augmented generation work, step by step?

**1. Prepare the knowledge base.** Documents are uploaded, their text is extracted, and long documents are split into overlapping chunks (passages) so each can be found on its own.

**2. Receive a question.** A visitor asks, for example, "Can I bring my dog?"

**3. Retrieve relevant passages.** The system ranks the chunks by relevance to the question and picks the best ones that fit a size budget. Common ranking methods:

- **Keyword ranking** such as BM25, which scores passages by shared words, weighted by how distinctive each word is.
- **Semantic (vector) search**, which compares embeddings, numeric representations of meaning, so "pets" can match "dogs".
- **Hybrids** of both.

**4. Augment the prompt.** The selected passages are inserted into the model's prompt with instructions such as "Answer only from the knowledge base below; if the answer isn't there, say so."

**5. Generate the answer.** The model writes a reply using those passages, in the visitor's language.

Some systems also include recent conversation turns in the retrieval step, so follow-ups like "what about cats?" still find the right passage.

## What happens when the knowledge base is small?

If all your documents fit comfortably within the model's context window (the amount of text it can read at once), retrieval isn't strictly needed: the model can read everything for every question. That avoids the risk of retrieval missing a relevant passage.

InstantKnowledgeBot uses this hybrid approach: **small knowledge bases are given to the model in full; larger ones are searched**, with BM25 keyword ranking over overlapping chunks, plus the visitor's recent messages to resolve follow-ups. The most relevant passages, in document order, are sent with the question.

## RAG vs fine-tuning: what's the difference?

| | RAG | Fine-tuning |
|---|---|---|
| **What changes** | The prompt: relevant documents are added per question | The model's weights, via additional training |
| **Updating facts** | Instant: replace the document | Requires retraining |
| **Traceability** | Answers come from passages you can inspect | Hard to trace where an answer came from |
| **Best for** | Answering from business documents that change | Teaching a style, format or specialised behaviour |
| **Cost and effort** | Low: upload documents | Higher: data preparation, training runs, evaluation |

For a business FAQ, support or document chatbot, RAG is almost always the right foundation. Fine-tuning and RAG can be combined, but most small businesses never need fine-tuning.

## Where can a RAG chatbot go wrong?

RAG reduces errors but doesn't eliminate them. Common failure points:

- **Missing information:** the answer isn't in the documents. The right behaviour is "I don't know" plus a fallback contact.
- **Contradictory documents:** two versions of a price list. The model may pick the wrong one.
- **Retrieval misses:** the relevant passage uses very different words from the question, or is buried in a huge document.
- **Unreadable sources:** scanned PDFs without a text layer, or facts that only exist in images.
- **Ambiguous questions:** "How much is it?" without saying what "it" is. A good bot asks a clarifying question.

Most of these are fixed by improving the documents, not the AI. See [preparing documents for an AI chatbot](/learn/preparing-documents-for-ai-chatbot) and [AI chatbot hallucinations](/learn/ai-chatbot-hallucinations).

## How do you evaluate a RAG chatbot?

You don't need a data science team. A simple, repeatable test set works:

1. Write 20 to 30 real questions, with the correct answer and the document it comes from.
2. Include tricky ones: different wording, questions needing two documents, and questions the documents don't answer.
3. Ask each question and mark the answer right, partly right or wrong.
4. Fix the documents behind wrong answers, then re-run the set after big changes.

In production, watch unanswered questions (knowledge gaps) and negative feedback. InstantKnowledgeBot's bot overview lists both.

## What are chunks, embeddings and context windows?

- **Chunk:** a passage of a document, often a few paragraphs long. Chunks usually overlap slightly so a sentence split across a boundary isn't lost.
- **Embedding:** a list of numbers representing the meaning of a piece of text, used for semantic search.
- **Context window:** the maximum amount of text a model can consider at once, including the instructions, retrieved passages, conversation and answer.
- **Grounding:** instructing the model to base its answer on the provided passages, and to say when they don't contain the answer.

## Do you need to build RAG yourself?

Developers can build RAG pipelines with open-source libraries, an embeddings model, a vector store and an LLM API. That's worthwhile if you need custom data sources or deep integration. For a website FAQ, support or lead capture bot, a hosted builder gives you the same pattern in minutes, plus the chat widget, analytics and hosting.

With InstantKnowledgeBot you upload documents, test and embed: [sign up free](/signup) or see [how to train a chatbot on PDFs](/guides/train-chatbot-on-pdf). For the bigger picture, read [what is a knowledge base chatbot](/learn/what-is-a-knowledge-base-chatbot), and compare plans on the [pricing page](/pricing).

## Frequently asked questions

**What does RAG stand for?**

Retrieval-augmented generation. The term comes from a 2020 research paper by Lewis and colleagues at Facebook AI Research, and it now describes any system that retrieves relevant text and gives it to a language model to generate an answer.

**Is RAG better than fine-tuning for a business chatbot?**

For answering questions from changing business documents, usually yes. RAG uses your current documents at question time, so updates are instant and answers can be traced to the source. Fine-tuning changes the model's behaviour but is a poor way to keep facts current.

**Do RAG chatbots need vector databases?**

Not necessarily. Many use embeddings and vector search, but keyword-based ranking such as BM25, or simply giving a small knowledge base to the model in full, are also valid approaches.

**Can a RAG chatbot still hallucinate?**

It can, though much less when it's instructed to answer only from retrieved text. Wrong answers usually come from missing or contradictory documents, or from retrieval missing the right passage.

**Does InstantKnowledgeBot use RAG?**

Yes. Small knowledge bases are given to the model in full; larger ones are searched with keyword-based ranking (BM25) and the most relevant passages, plus recent conversation context, are used for each question.