In this article, I will share the retrieval strategy I use to form the final context formation at EazyRAG. Before diving into the article, there is one concept you need to understand.

What is RAG?

LLMs such as GPT lack factual consistency and lack up-to-date knowledge. To fix this, RAG is a technique that retrieves related documents to the user's question, combines them with LLM-base prompt, and sends them to LLMs like GPT to produce more factually accurate generation.

I've written an extensive article on building an RAG pipeline without using Langchain. It will give you a good idea of what RAG is.

Here is what the RAG (Before) pipeline looks like.

Retrieval/Context formation

Now, here is how we differ from the general RAG pipeline: we don't just retrieve chunks shove them under the base prompt, and call it a day. This approach may work for questions like 'What is the height of the Eiffel Tower?' but may not be effective all the time. Just imagine, intuitively, you have an article called 'How to make spaghetti,' and you've split the long article into pieces and indexed it. When a user queries it, you retrieve all the relevant chunks. Herein lies the problem: some general cooking instructions, such as 'boil the water and season it,' won't have a relatable relevance score to the user's search query, which, in turn, results in poor context formation and increases hallucinations.

Not only that but just shoving unstructured messy chunks will also decrease the quality of the answer.

Here is the comparison between Langchain (code taken from the Quickstart example) and EazyRAG.

Langchain

from langchain.document_loaders import WebBaseLoader
from langchain.indexes import VectorstoreIndexCreator

loader = WebBaseLoader("https://bun.sh/docs/installation")
index = VectorstoreIndexCreator().from_loaders([loader])
index.query("How do I install this on a Mac?")

Output by Langchain

" You can install Bun on a Mac using the Homebrew package manager. Run the command 'brew tap oven-sh/bun' followed by 'brew install bun'."

And here is the answer by EazyRAG

Not to brag, but I think EazyRAG's answer is better with multiple options. But, what went wrong with Langchain is that when I dug into the final prompt, all the chunks retrieved were in a confusing format without proper spaces. Anyway, this is not a rant about Langchain; I have huge respect for that project.

So how does EazyRAG do it?

When I was coding this layer, I knew that this API had to work with all types of datasets and with any type of search intent. I couldn't manually customize the context formation for every type of dataset, so why not let GPT itself handle context formation?

Here's how it works:

When a user queries it, we retrieve the relevant documents related to the document title and send them to GPT to classify whether we should consider feeding the entire documents for this user query. If so, which documents should we consider? Then, we feed all the documents mentioned by the GPT classifier. If the GPT classifier returns nothing, we simply shove all the chunks into base prompts and send them to LLM for answer generation. And it is working great!

In fact, this is an early version of this idea. I have a lot of ideas surrounding this concept of "GPT itself handling context formation" that I need to try.

So, what do you think of this idea? I'd be happy to hear your feedback and improve EazyRAG.

That's what we are trying to solve with EazyRAG. We have the entire RAG pipeline implemented, and you can use it with a simple API call. What's even better is that we also support grounded answer generation from the browser. Here is a demo where I indexed the entire Bun Docs on EazyRAG, which is making RAG requests from the browser.

Here is a sample code.

Index your content from the server side.

curl --location 'https://api.eazyrag.com/v1/add' \
--header 'Authorization: sk-<Your server key>' \
--header 'Content-Type: application/json' \
--data '{
 "contents": [
   {
       "content":"You entire document or chunk of content",
       "title":"Document title (Optional)"
   }
 ],
 "collection": "sample"
}'

Once you have indexed your content, it's just a dead-simple way to generate grounded answers with a simple API call from your browser.

  const fetchResponse = async (question: string) => {
    try {
      const response = await axios.get("https://api.eazyrag.com/v1/answer", {
        params: {
          query: question,
          collection: "sample",
        },
        headers: {
          Authorization:
            "ck-<Your client key>",
        },
      });

      // Grounded answer generation
      console.log(response.data);

    } catch (error) {
      console.error("Error fetching data:", error);
    }
  };

We also support streams, and you can use EventSource to implement an API with streams on the browser side.

Our longer-term vision (if we survive) is to build a dead-simple API on top of LLM, where even someone who is just getting started with coding can create powerful LLM agents without needing to understand the underlying concepts.

If you have any questions or feedback, don't hesitate to click on the chat icon.

A short intro about RAG.

Quoting from the Facebook (aka Meta) OG RAG paper.

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems.

Yeah, that's the exact problem RAG solves. In simple words, LLMs such as GPT lack factual consistency and lack up-to-date knowledge. To fix this, RAG is a technique that retrieves related documents to the user's question, combines them with LLM-base prompt, and sends them to LLMs like GPT to produce more factually accurate generation.

Here is the image I made to make visualization easier.

Let’s build

I will split the RAG pipeline into 5 parts:

• Data preparation

• Chunking

• Vector store

• Retrieval/Prompt preparation

• Answer Generation

Shameless plug: If you don't want to go through the process above or customize chunking strategies using trendy libraries, please consider visiting eazyrag.com. We handle everything. Here is a demo I built in which I indexed the entire Bun Docs/Guides with just two simple network calls for indexing and answer generation.

Back to the article.

Data preparation

As we are planning to create a HackerNews (HN) support page, I have downloaded the relevant HTML pages, converted them to raw text, and saved them as JSON files.

data/
├── legal.json
├── newsfaq.json
├── newsguidelines.json
└── security.json

JSON file looks something like this.

{
   "content":"HTML converted into string...",
   "url":"https://news.ycombinator.com/newsfaq.html"
}

There is a reason I am adding a relevant URL to the file, when the LLM generates an answer, we can link back to the original URL used as context for citation.

Chunking

This is one of the important parts of the RAG pipeline. There are a whole lot of chunking strategies that deserve their own post, which I might write. But in short, here is how I decide this: when the answer is generated and cited with the citation number, I should be able to go back to the original sentence that has the answer on the HackerNews page by using text fragments.

So, we are going to split the content by sentence and save it as JSON with the URL key in every text fragment.

I removed the imports, don’t worry. I linked the final code at the bottom. Now let's only focus on the main parts of the code.

nlp = spacy.load("en_core_web_sm")

def process_file(file_path):
    with open(file_path) as f:
        data = json.load(f)
        content = data['content']
        url = data['url']
        doc = nlp(content)
        
        return [{'text': sent.text, 'url': url} for sent in doc.sents]

chunks = [chunk for file in os.listdir('data') for chunk in process_file(os.path.join('data', file))]

chunks = [{'id': i, **chunk} for i, chunk in enumerate(chunks)]

with open('chunks.json', 'w') as f:
    json.dump(chunks, f)

In the code above, I simply read all the files in the 'data' directory, iterated through the files, split them by sentence, assigned IDs to them (which we will use later), and saved them as 'chunks.json'.

Vector store

Now, we are going to convert all the chunks into embeddings. I am going to use the sbert all-mpnet-base-v2 model, but there are a lot of options available. Here is the leaderboard, and there is even a new state-of-the-art model (I’m still reading this paper).

sentences = [chunk['text'] for chunk in chunks]

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')

embeddings = model.encode(sentences, show_progress_bar=True)

faiss_index = faiss.IndexFlatIP(model.get_sentence_embedding_dimension())
faiss_index.add(embeddings)

In the code above, I converted all the texts into embeddings using the pre-trained embedding model and built the index using Faiss. If you have a large amount of data and are planning to use a bigger dimensional model, I'm pretty sure memory will be a problem.

If you are looking to quantize the faiss index size, you can manually quantize it, or you can use this nice tool which will produce the index at the specified size. It's a really cool tool and a huge time saver.

Retrieval/Prompt preparation

We are going to retrieve the relevant chunks to form the final prompt.

base_prompt = """You are an AI assistant. Your task is to understand the user question, and provide an answer using the provided contexts. Every answer you generate should have citations in this pattern  "Answer [position].", for example: "Earth is round [1][2].," if it's relevant. 

Your answers are correct, high-quality, and written by an domain expert. If the provided context does not contain the answer, simply state, "The provided context does not have the answer."

User question: {}

Contexts:
{}
"""

k = 50
question = 'Why HackerNews is so popular?' 

query_embedding = model.encode([question])
distances, indices = faiss_index.search(query_embedding, k)

context = '\n'.join([f'{i}. {sentences[index]}' for i, index in enumerate(indices[0])])
prompt = f'{base_prompt.format(question, context)}'

In the code above, I retrieved all the relevant chunks, attached them to the base prompt, along with the user's question and context retrieved from the faiss_index, and formed the final prompt that we are going to send to GPT-3.5.

Answer Generation

Just send the prompt to OpenAI.

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo-16k-0613",
    temperature=0,
    messages=[
        {"role": "system", "content": prompt},
    ]
)

print(response.choices[0].message.content)

The response I got for “Why HackerNews is so popular?”

Hacker News is popular for several reasons:

1. Quality Content: Hacker News focuses on providing content that would be interesting to "good hackers" [8]. This means that the platform curates and promotes high-quality articles, discussions, and resources related to technology, startups, and hacking [8].

2. Community Engagement: Users are encouraged to vote and comment on content that they find interesting, rather than for self-promotion [12]. This fosters a sense of community engagement and encourages meaningful discussions.

3. Y Combinator Association: Hacker News is associated with Y Combinator, a well-known startup accelerator [11]. This association brings credibility and attracts entrepreneurs, investors, and tech enthusiasts to the platform.

4. Privacy and Security: Hacker News takes privacy and security seriously [2]. They have implemented measures to protect personal information and have a privacy policy in place [41]. This commitment to privacy and security helps build trust among users.

5. Diverse Topics: Hacker News covers a wide range of topics beyond just hacking and startups [4]. This diversity attracts users with different interests and backgrounds, making the platform more appealing and engaging.

6. Active Moderation: Hacker News has active moderation to prevent abuse and maintain the quality of discussions [22]. This helps create a positive and respectful environment for users. Overall, the combination of quality content, community engagement, Y Combinator association, privacy and security measures, diverse topics, and active moderation contribute to the popularity of Hacker News. [8][12][41][4][22][11].

Now you can parse the citations and find the relevant chunks. I am skipping this since that’s not the goal of this article.

Conclusion

I hope you now have a good understanding of the RAG pipeline. Here is the final code.

Now, the above process may look simple, but a lot of effort needs to be done for perfect retrieval and precise context generation. EazyRAG, the tool we are building, tackles this. We have incorporated a lot of different strategies to form a good context, which increases the quality of the answer.

Visit: https://eazyrag.com

Subscribe now

This is just a small thought experiment for a different ranking algorithm for HN. There's nothing wrong with the current one, but it's just a small thought experiment.

Just to refresh, according to Paul Graham's comment and FAQ, the HN algorithm is...

P = Points

T = Age in hours

G = 1.8

Minus one removes the user's own upvote. By default, HN upvotes its own user submission.

I’ve seen one with penalties but let’s stick to what’s official. Also, I just did a quick Grep search and found this article and wonderful repo about “Reverse Engineering the Hacker News Ranking Algorithm“

Intuitively, the ranking algorithm is simple: the more upvotes a link receives in a short amount of time, the higher it will be ranked at the top. As time passes, the post will gradually move down the rankings.

Here is my approach

I can't find the official page on why Paul Graham created HN, but to me, HN is the place to find the most interesting links and have a healthy discussion about various topics. In fact, discussions are my favorite part, except for the useless comments.

Given that discussions are my favorite part, why don't we apply PageRank for every user based on the upvotes they receive for the comments they leave on any post, and replace the P (points) value in the current version of the HN algorithm?

PageRank is an algorithm created by the founders of Google. It is used to determine the popularity of web pages on the internet.

By the way, the name PageRank did not suit this use case. I am going to call it HackerRank (HR). Here is a visualization if you are trying to picture it.

Since it's likely that one user may upvote multiple comments from the same user, we check whether a user has already upvoted a comment from that specific user before considering their upvote. In other words, we treat user profiles as nodes and upvotes for comments as edges.

Considering this, a HackerRank user profile will look like this:

N = Total number of user profiles on HN

D = Damping factor

HR = HackerRank score for the user who upvoted

TU = Total upvotes given

By default, the HR for every profile will be 1/N.

But I won't consider HackerRank as it is. Sometimes, HN comments are inappropriate, and they will be flagged. We should consider the flagging because, remember, HN should be a place to have healthy discussions.

Let's consider that 1 flag equals a deduction of 20% from the HR score. However, we will only take into account the flags received in the current month, as people can change from being unpleasant to becoming better human beings.

So, HR with “flag” consideration will look like this.

TF = Total flags received in the current month

FP = Flag penalty which is 20%

If HR is negative, then it will be 0.

We now have HR for every user profile, so the final ranking algorithm will look like this by hooking HR into the current version of the HN algorithm.

HR = HR value represents the HackerRank score of the profile that upvoted this post

T = Age in hours

G = 1.8

This is a very simple approach, but here are some other ideas that are worth exploring.

• HackerRank score, which is also determined by the upvotes a user receives for their submissions.

• Reading time for the article.

• Track how well a website is performing on HN and put it on the front page if the website has a high reputation for performing well.

Will I ever publicly write about how HN ranks posts if I am Dang (HN moderator)? No, because Pagerank can be manipulated by people despite its reputation. In fact, Pagerank is being exploited for years. Moreover, there are financial incentives for companies to get on the first page in HN.

I would use HackerRank for ranking posts but publicly say that we are using original PaulGraham’s algorithm for ranking posts, and hide the upvotes for comments since it is powering HackerRank, also take some additional steps to avoid reverse engineering and rank manipulation.

But, I am curious. How would you have done it? If you were designing the HN algorithm, please leave your thoughts in the comments. I am curious to know.

Discussion on HN → https://news.ycombinator.com/item?id=35510413

Plug: Hey, we are building a new kind of search engine. Our goal is to deliver authoritative and non-SEO-spammed results. Please check it out and let me know your feedback.