Do Larger Context Windows Remove the Need for RAG?

Objective and Scope

In this article, I evaluate two different approaches to grounding generated content using OpenAI's ChatGPT-4o model. I will use company documents to provide context, aiming to transform ChatGPT into an expert on the internal processes of GenUI. The primary goal is to ensure that the generated answers from the model are more contextually relevant and factually accurate when prompted about GenUI specifics.

Methodology

Question Selection

A set of 40 questions relevant to GenUI's internal processes was selected from the new employee handbook and onboarding deck.

Method 1: Inline Context

This approach involves appending all relevant context directly into the model's prompt. The steps are:

• Parse PDFs files into text and append all content directly into the prompt.
• Generate answers from this single extensive prompt.
• Record the generated answers to each of the 40 questions.

Method 2: RAG (Vector Store Retrieval)

This method utilizes OpenAI's vector stores to upload and create embeddings of custom files. This method embodies the principles of RAG, integrating retrieval mechanisms to augment the generation process. The steps are:

• Upload the handbook and onboarding deck to OpenAI's vector stores and create embeddings.
• Query the vector stores for each question to retrieve relevant information.
• Generate answers using the retrieved embeddings.
• Record the generated answers to each of the 40 questions.

Data Analysis

To compare the effectiveness of the inline context and vector DB approaches, BLEU and ROUGE scores are used as evaluation metrics. These metrics quantify the accuracy and relevance of responses by comparing generated answers to reference answers.

• The BLEU score is a measure of the precision of retrieved answers, indicating how many words match with the reference answers.
• ROUGE scores measure recall, indicating how well the retrieved answers cover the reference answers.

By employing these metrics, the study aims to quantify the accuracy and relevance of responses generated using both methods.

Results

For an in-depth look at how the results were obtained, you can review the Jupyter notebooks hosted in our research repository.

Performance comparison

Table 1. Performance comparison

As seen in Table 1, the inline context method consistently outperformed the vector DB method across all metrics, suggesting better retrieval of accurate and relevant answers. However, the difference is slight, suggesting both methods are close in overall performance and could be viable options depending on specific use cases and requirements. Interestingly, while both methods produce the correct answers, the Vector DB method tends to provide more verbose answers overall which might affect the BLEU and ROUGE scores. Take question 15 “What are the paid holiday benefits provided by GenUI? As an example, both results for method 1 and method 2 focused on the holidays rather than the benefits, with the vector DB method producing a longer answer. A verbose answer is not necessarily a wrong answer, after all the level of verbosity is dictated by your business needs but is worth mentioning, as this affects the scores.

Figure 1. BLEU score comparison

Figure 2. ROUGE score comparison

Cost comparison

Table 2. Cost comparison

Cost calculation based on Open AI pricing.

The cost comparison reveals that the Inline Context method incurs a higher cost due to the extensive prompt size required to include all relevant context. Interestingly, the Vector DB method uses a lot more input tokens than one might expect, with the inline method using on average 20% more input tokens. This suggests that at this level of input tokens (~30K per prompt), using OpenAI's vector stores to retrieve relevant context is still adding to the prompt about 80% of the total context. Which does not sound like a compelling benefit if we extrapolate this trend with larger input token counts.

Conclusion

For use cases where the context information is small (within 30K tokens), it seems that if the priority is accurate responses, it is better to append the relevant context directly in a single prompt albeit this approach comes with increased API usage costs.

However, the cost-efficiency of the Vector DB method is still significant, especially for applications where cost is a critical factor. Considering that in terms of accuracy as shown in Table 1, you are guaranteed to get results almost as good as the inline context method with a 20% reduction in API costs.

Future Exploration

This experiment does not show a promising benefit between OpenAI's vector stores against using inline context directly. This opens up the possibility to explore much wider input token thresholds.

If we were to increase the context size to 50K, 100K or even 150K tokens, would OpenAI's vector stores still retrieve and append 80% of the total context into our prompts or will it plateau elucidating the benefits of a vector store in the first place?

On the next step of this research we shall increase the amount of tokens to find out how much better it is to insist on a RAG solution by OpenAI's vector stores or if sending all context on a single prompt remains a valid approach to get ChatGPT-4o to generate company specific answers.