So I did what any reasonable person would do.

I sat down.

Relaxed.

Waited.

Five minutes passed.

Ten minutes passed.

Twenty-three minutes passed.

Turns out… she wasn’t ready.

She was stalling.

And that’s exactly how browsers treat your API requests.

Your backend might be ready.

Your frontend might be ready.

But Chrome? Chrome might just be sitting there like:

“Yeah… hold on. Not yet.”

#The Tab Everyone Pretends Doesn’t Exist

Let’s be honest most debugging sessions look like this:

• Console → checking for red errors ✅
• Network → checking request status and response times ✅
• Elements → maybe poking around ❓
• Timing → wait… what’s that? ❌

Even when developers do open Network, they usually stop at:

• Headers
• Payload
• Response

Meanwhile, Timing sits right there, quietly exposing the real reason your app feels slow.

#That Tiny “Timing” Tab You’ve Been Ignoring

This view lives inside Chrome DevTools → Network, after you click a request, right next to:

• Headers
• Payload
• Preview
• Response
• Initiator
• Timing ← this one

Most developers never make it past “Response.” That’s a mistake.

Because the Timing tab doesn’t tell you what the server returned it tells you how long the browser made your request wait before it even got a chance.

Now look closely at the screenshot.

#What’s Actually Happening Here

From the Timing breakdown:

• Queueing: ~5.05s
• Stalled: ~7.65s
• Waiting for server response: 10.22s
• Content download: ~1.78ms

Total time before the frontend gets usable data:

👉 22.92 seconds

Here’s the punchline:

The server only needed 10.22 seconds. The page waited 22.92 seconds before showing the main data.

That means more than half the time was spent doing absolutely nothing just waiting for the browser to allow the request to proceed.

#Queueing vs Stalled (In Human Terms)

• Queueing The browser wanted to send the request… but couldn’t. All available connection slots for that domain were already taken.
• Stalled A slot finally opened up, but the browser still had to wait on internal scheduling, connection reuse, or other in-flight requests.

None of this involves your backend. None of this shows up in server logs. None of this throws an error.

From the server’s point of view, this request was fast.

From the user’s point of view, your app felt broken.

#Why This Screenshot Is So Dangerous (And So Useful)

If you only looked at:

• the Response tab → “server is slow-ish”
• backend logs → “everything looks normal”

You’d miss the real issue completely.

The Timing tab exposes the uncomfortable truth:

Your app wasn’t slow because the server took too long. It was slow because the browser made the request wait in line.

#The Browser Is Throttling You, And It’s Not Asking Permission

Browsers enforce a maximum number of concurrent connections per origin.

For most modern browsers, that limit is roughly:

~6 simultaneous requests per domain

Once that limit is hit:

• New requests are queued
• They don’t reach the server
• They don’t show up in backend logs
• They just… wait

This happens before JavaScript errors, before your API, and before any monitoring tools notice.

#Yes, You’ve Written This Code Don’t Lie

Imagine a dashboard page that loads the following on initial render:

• User profile
• Organization settings
• Permissions
• Feature toggles
• Notifications count
• Sidebar navigation config
• Main dashboard data (critical)

All fired at once.

All sent to the same API domain.

All triggered by various useEffect hooks because “they’re independent.”

The browser looks at this and says:

“Cool. I’ll take the first six.

The rest can wait.”

Guess which request often gets queued?

Yep the one the page actually depends on.

#Why Everything Looks Fine Until You Look Here

From a developer’s point of view:

• APIs respond fast when tested alone
• Backend logs show low latency
• No errors in Console
• No failed requests in Network

But in the Timing tab you see something like:

• Queueing: 6–8 seconds
• Request sent: microseconds
• Waiting (server): 20ms
• Download: 1ms

That’s when it clicks.

#Why the Browser Doesn’t Care Which Request Is “Important”

From the browser’s perspective, these are all equal:

fetch('/profile')
fetch('/settings')
fetch('/permissions')
fetch('/features')
fetch('/notifications')
fetch('/navigation')
fetch('/dashboard-data')

Same origin.

Same priority.

Same timing.

The browser doesn’t understand your business logic.

It only understands connection slots.

Requests are allowed through in the order they’re initiated, not by importance.

#The Real Bug Is “Let’s Just Fetch Everything”

This isn’t about “too many requests.”

It’s about:

• Too many requests
• At the same time
• Competing for limited browser connections

This pattern is extremely common in:

• React apps
• Hook-based data fetching
• “Load everything on mount” designs

🔥 Hot take

If your page needs 6+ API calls at the exact same time just to become usable, the problem isn’t performance — it’s architecture, and the way how you are writing code.

#Fixing It by Designing for Zero Queued Requests

The real fix is not “optimize the slow request.”

The real fix is to design your system so that the browser never has a reason to queue a critical request at all.

#How to Debug This in Practice

Next time someone says “the app feels slow”, do this:

1. Open DevTools → Network
2. Click the slow request
3. Open the Timing tab (next to Headers / Payload / Response)
4. Look at:

• Queueing time
• Waiting time

If Queueing > 3–4ms, you’ve found the culprit.

#Common Mistakes to Avoid

• ❌ Treating all requests as equal
• ❌ Firing all requests on component mount
• ❌ Assuming no errors = good performance
• ❌ Ignoring the Timing tab entirely

Performance bugs don’t always throw stack traces.

Some just quietly waste seconds.

#Key Takeaways (The Stuff Worth Remembering)

• The Timing tab lives inside Network and it’s critical
• Queueing means the browser hasn’t sent the request yet
• Browsers limit concurrent connections per domain
• Too many parallel requests introduce hidden latency
• Better sequencing beats aggressive preloading
• The real fix is to design your data fetching so critical requests never have to wait in the queue

#Wrapping It Up (Design So Nothing Queues)

Your app wasn’t broken.

Your backend wasn’t slow.

Your code wasn’t “wrong” in the traditional sense.

The problem was architectural: you let the browser treat every request as equally important and equally urgent.

Once you:

• Define a clear critical path
• Fire that path with minimal competition
• Batch and delay non-essential calls
• Centralize orchestration instead of sprinkling fetch everywhere

…you stop “optimizing slow requests” and start designing a system where important requests never get queued in the first place.

So next time performance feels off, don’t just ask “how do I make this request faster?”

Ask:

“Why did the browser have to queue this at all and how can I design things so it never needs to?”

That’s where the real performance win lives, long before the request ever shows up in your logs.

Mid sentence.

And weirdly… I loved it.

Not because I enjoy being cut off, but because for the first time, an AI assistant felt human enough to jump into the conversation.

That’s the magic of real-time voice AI.

#The Story Behind the Silence

• Turn-Based Voice AI feels like a classroom: you speak, then the AI waits….. silently….. until you’re done. Only then does it think, respond, and speak. Predictable… but awkward.
• Real-Time Voice AI, however, listens and responds as you speak. It interrupts to clarify, builds anticipation, and makes the interaction feel alive. It’s not just hearing you it’s conversing with you.

#What Makes Real-Time Feel Real?

But under the hood? It’s orchestration chaos managing barge in detection, aligning streams, handling interruptions, and keeping latency under 1 second.

#When to Pick Which?

1. Turn-Based (classic STT → LLM → TTS pipeline):

✅ Easier to build and debug.

❌ Feels robotic with 0.7 to 3s delays.

3. Real-Time (Speech-to-Speech):

✅ Natural, fluid, human like.

❌ Architecturally complex, less modular.

#In Practice

Modern systems still rely on STT → NLP → TTS, but optimized with:

• Streaming ASR (<300 ms)
• Low-latency inference (<500 ms)
• Chunked TTS (<200 ms to first audio)

Done right, the whole pipeline feels instant.

#TL;DR

Turn-based AI listens.

Real-time AI converses.

And that tiny shift from waiting to weaving makes the difference between talking to a machine and talking with one.

#Chapter 1: The Dream

It all started on a rainy afternoon. I was talking to my computer (as one does when he is a remote worker), and realized:

Wouldn’t it be cool if my computer could actually listen, understand, and answer me with real knowledge from my own files?

Not just “Hey Siri, what’s the weather?” but “Hey AI, what’s in my project docs?” or “Remind me what the HR policy says about bringing cats to work?”

And so, the quest began:

I would build a Voice AI RAG agent!

(That’s Retrieval-Augmented Generation, but let’s just call it “RAG” because it sounds like a pirate.)

#Chapter 2: The Ingredients

Before you can summon your own digital sidekick, you’ll need a few magical artifacts:

• Python 3.11+ (the spellbook)
• Cartesia (for making your AI talk like a human, not a fax machine)
• AssemblyAI (so your AI can understand your voice, even if you mumble)
• Anthropic Claude (the brain—OpenAI is cool, but Claude is the new wizard in town)
• LiveKit (for real-time voice rooms, so your AI can join you in a virtual “room”)
• A pile of your own documents (so your AI knows your world)
• API keys (the secret runes—don’t lose them!)

#Chapter 3: The Spell (a.k.a. The Code)

Here’s the full incantation. Don’t worry, I’ll explain every part after you read it.

(Copy, paste, and prepare to be amazed!)

import logging
import os
from dotenv import load_dotenv
from livekit.agents import JobContext, JobProcess, WorkerOptions, cli
from livekit.agents.job import AutoSubscribe
from livekit.agents.llm import (
    ChatContext,
)
from livekit.agents.pipeline import VoicePipelineAgent
from livekit.plugins import cartesia, silero, llama_index, assemblyai

load_dotenv()

logger = logging.getLogger("voice-assistant")
from llama_index.llms.anthropic import Anthropic
from llama_index.core import (
    SimpleDirectoryReader,
    StorageContext,
    VectorStoreIndex,
    load_index_from_storage,
    Settings
)
from llama_index.core.chat_engine.types import ChatMode
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

load_dotenv()

# Set up the embedding model and LLM
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
llm = Anthropic(model="claude-3-haiku-20240307", max_tokens=512)
Settings.llm = llm
Settings.embed_model = embed_model

# check if storage already exists
PERSIST_DIR = "./chat-engine-storage"
if not os.path.exists(PERSIST_DIR):
    # load the documents and create the index
    documents = SimpleDirectoryReader("docs").load_data()
    index = VectorStoreIndex.from_documents(documents)
    # store it for later
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    # load the existing index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()

async def entrypoint(ctx: JobContext):

    chat_context = ChatContext().append(
        role="system",
        text=(
            "You are a funny, witty assistant."
            "Respond with short and concise answers. Avoid using unpronouncable punctuation or emojis."
        ),
    )
    
    chat_engine = index.as_chat_engine(chat_mode=ChatMode.CONTEXT)

    logger.info(f"Connecting to room {ctx.room.name}")
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    participant = await ctx.wait_for_participant()
    logger.info(f"Starting voice assistant for participant {participant.identity}")

    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        stt=assemblyai.STT(),
        llm=llama_index.LLM(chat_engine=chat_engine),
        tts=cartesia.TTS(
            model="sonic-2",
            voice="bf0a246a-8642-498a-9950-80c35e9276b5",
        ),
        chat_ctx=chat_context,
    )

    agent.start(ctx.room, participant)

    await agent.say(
        "Hey there! How can I help you today?",
        allow_interruptions=True,
    )

if __name__ == "__main__":
    print("Starting voice agent with Anthropic...")
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        ),
    ) 

#Chapter 4: The Magic Explained

Let’s break down this spellbook, line by line:

1. Imports and Setup

We import all the libraries:

• livekit for voice rooms
• cartesia for text-to-speech
• assemblyai for speech-to-text
• llama_index for RAG (so your AI can actually know things from your docs)
• Anthropic for the LLM (the brain)

We also load environment variables with dotenv—because hardcoding API keys is a rookie mistake.

2. Embeddings and LLM

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
llm = Anthropic(model="claude-3-haiku-20240307", max_tokens=512)
Settings.llm = llm
Settings.embed_model = embed_model

• The embedding model turns your docs into “AI food” (vectors).
• The LLM (Claude) is the brain that answers questions using those vectors.

3. Document Indexing

PERSIST_DIR = "./chat-engine-storage"
if not os.path.exists(PERSIST_DIR):
    documents = SimpleDirectoryReader("docs").load_data()
    index = VectorStoreIndex.from_documents(documents)
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

• If you haven’t indexed your docs before, it reads everything in docs/ and builds a knowledge base.
• If you have, it loads the existing index (so it doesn’t have to re-read your 500-page PDF every time).

4. Voice Activity Detection (VAD)

def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()

• This makes sure your AI only listens when you’re actually talking, not when you’re yelling at your cat.

5. The Entrypoint: Where the Magic Happens

async def entrypoint(ctx: JobContext):
    chat_context = ChatContext().append(
        role="system",
        text=(
            "You are a funny, witty assistant."
            "Respond with short and concise answers. Avoid using unpronouncable punctuation or emojis."
        ),
    )
    chat_engine = index.as_chat_engine(chat_mode=ChatMode.CONTEXT)
    ...

• Sets the “personality” of your AI (witty, concise, no weird punctuation).
• Prepares the chat engine with your indexed docs.

    logger.info(f"Connecting to room {ctx.room.name}")
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
    participant = await ctx.wait_for_participant()
    logger.info(f"Starting voice assistant for participant {participant.identity}")

• Connects to a LiveKit room (more on this soon).
• Waits for a participant (that’s you!) to join.

    agent = VoicePipelineAgent(
        vad=ctx.proc.userdata["vad"],
        stt=assemblyai.STT(),
        llm=llama_index.LLM(chat_engine=chat_engine),
        tts=cartesia.TTS(
            model="sonic-2",
            voice="bf0a246a-8642-498a-9950-80c35e9276b5",
        ),
        chat_ctx=chat_context,
    )
    agent.start(ctx.room, participant)
    await agent.say(
        "Hey there! How can I help you today?",
        allow_interruptions=True,
    )

• Sets up the full voice pipeline: listens, understands, thinks, and talks back.
• Greets you with a friendly message.

6. The Main Event

if __name__ == "__main__":
    print("Starting voice agent with Anthropic...")
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        ),
    ) 

#Chapter 5: Summoning Your AI (a.k.a. Running the Code)

1. Install your dependencies (see requirements.txt).
2. Put your API keys in a .env file:

    ANTHROPIC_API_KEY=your_anthropic_key
    ASSEMBLYAI_API_KEY=your_assemblyai_key
    CARTESIA_API_KEY=your_cartesia_api_key
    LIVEKIT_URL=your_livekit_url
    LIVEKIT_API_KEY=your_livekit_api_key
    LIVEKIT_API_SECRET=your_livekit_api_secret

1. Add your documents to the docs/ folder.
2. Run

    python voice_agent_anthropic.py start

#Chapter 6: Entering the LiveKit Room

• What’s a LiveKit room?

Think of it as a virtual meeting room where your AI is always waiting for you.

• How do you join?

Use the LiveKit Playground or your own LiveKit client, enter your room name, and your AI will greet you like an old friend (who actually remembers your last conversation).

#Chapter 7: The Result

Now, you can:

• Talk to your AI: Ask questions, get answers from your own docs.
• Get witty, concise responses: No more boring bots!
• Impress your friends: “Yeah, my Voice AI actually knows what’s in my files.”

if you have any doubts you can contact me 🙂

#Happy coding! 🚀

Introduction

Progressive Web Apps (PWAs) have gained massive popularity in modern web development. PWAs are web applications designed to deliver app-like experiences directly through the browser. They can be installed on a device’s home screen, work offline, and utilize push notifications, making them a bridge between native apps and web apps.

One of the key features of a PWA is the ability to allow users to install it directly from the browser. While browsers often display a default "Add to Home Screen" banner, creating a custom install button ensures better control over the user experience.

In this blog, we’ll walk through how we created a custom download button for our PWA using:

1. The beforeinstallprompt event.
2. State management (using Zustand).
3. A seamless experience that doesn’t require a page refresh.

#What is the beforeinstallprompt Event?

The beforeinstallprompt event is fired by the browser when a web app meets the PWA installation criteria (e.g., HTTPS, manifest, service worker). This event gives developers control over how and when to show the installation prompt, allowing for custom UI elements like an "Install App" button.

#Key Points about beforeinstallprompt:

• Prevents default behavior: By calling e.preventDefault(), you stop the browser from automatically showing the default install banner.
• Stores the event: The event must be stored temporarily to trigger the prompt later when the user clicks the custom install button.
• Single-use: The beforeinstallprompt event can be used only once after firing. If dismissed, it won't fire again unless the user navigates to another page.

#The Challenge

While the beforeinstallprompt event is straightforward to use, it has limitations:

1. The event is lost when navigating between pages, requiring a reload to trigger again.
2. Storing and reusing the event requires careful handling since it's an event object and not serializable.

Our goal was to overcome these challenges by implementing state management to store and reuse the event, allowing for a seamless install flow without refreshing the page.

#Solution: Managing the beforeinstallprompt Event with Zustand

Zustand is a lightweight state management library perfect for global state management in React applications. We used Zustand to store the beforeinstallprompt event, making it accessible across components without needing to refresh the page.

Here’s how we did it:

#1. Capturing the beforeinstallprompt Event

On the login page, we added an event listener to capture the beforeinstallprompt event and stored it in the Zustand store for later use.

import { useEffect } from 'react';
import useInstallPromptStore from './stores/useInstallPromptStore';

const LoginPage = () => {
  const setDeferredPrompt = useInstallPromptStore((state) => state.setDeferredPrompt);

  useEffect(() => {
    const handleBeforeInstallPrompt = (e) => {
      e.preventDefault(); // Prevent the default banner
      setDeferredPrompt(e); // Store the event in Zustand
      window.sessionStorage.setItem('deferredPromptFired', true); // Optional: Store a flag
    };

    window.addEventListener('beforeinstallprompt', handleBeforeInstallPrompt);

    return () => {
      window.removeEventListener('beforeinstallprompt', handleBeforeInstallPrompt);
    };
  }, [setDeferredPrompt]);

  return <div>Login Page</div>;
};

#2. Storing the Event in Zustand

We created a Zustand store to manage the beforeinstallprompt event and a flag to track whether it has been fired.

import { create } from 'zustand';

const useInstallPromptStore = create((set) => ({
  deferredPrompt: null, // Store the event
  isPromptFired: false, // Flag to indicate event firing

  setDeferredPrompt: (event) => set({ deferredPrompt: event, isPromptFired: true }),  
  clearDeferredPrompt: () => set({ deferredPrompt: null, isPromptFired: false }),
}));

export default useInstallPromptStore;

#3. Displaying the Custom Install Button

On the next page (e.g., the main dashboard or home page), we retrieved the stored beforeinstallprompt event from the Zustand store and used it to display a custom install button.

import { useEffect, useState } from 'react';
import useInstallPromptStore from './stores/useInstallPromptStore';

const HomePage = () => {
  const deferredPrompt = useInstallPromptStore((state) => state.deferredPrompt);
  const isPromptFired = useInstallPromptStore((state) => state.isPromptFired);
  const clearDeferredPrompt = useInstallPromptStore((state) => state.clearDeferredPrompt);

  const [showInstallButton, setShowInstallButton] = useState(false);

  useEffect(() => {
    const promptFired = window.sessionStorage.getItem('deferredPromptFired');
    if (isPromptFired || promptFired) {
      setShowInstallButton(true);
    }
  }, [isPromptFired]);

  const handleInstallClick = () => {
    if (deferredPrompt) {
      deferredPrompt.prompt(); // Trigger the install prompt
      deferredPrompt.userChoice.then((choiceResult) => {
        if (choiceResult.outcome === 'accepted') {
          console.log('User accepted the install prompt');
        } else {
          console.log('User dismissed the install prompt');
        }
        clearDeferredPrompt(); // Clear the stored event after use
        window.sessionStorage.removeItem('deferredPromptFired');
      });
    }
  };

  return (
    <div>
      {showInstallButton && <button onClick={handleInstallClick}>Install App</button>}
    </div>
  );
};

export default HomePage;

#Why This Works

1. Centralized State with Zustand:

Using Zustand, we stored the beforeinstallprompt event in a central location, making it accessible across components without needing to reload the page.

3. Session Flag for Fallback:

By storing a flag (deferredPromptFired) in sessionStorage, we ensured the install button is displayed even if the user navigates back and forth between pages.

5. Custom UI for Better Experience:

The custom install button gave us full control over the user experience, ensuring the prompt appears when users are most likely to engage with it.

#Conclusion

Creating a custom install button for your PWA using the beforeinstallprompt event and Zustand is a robust solution for managing the PWA installation flow. By leveraging state management, you can provide a seamless user experience, eliminate the need for page reloads, and encourage more users to install your app.

PWAs are powerful tools for bridging the gap between web and native applications. Adding a touch of customization like this can make a significant difference in user adoption.

Happy coding! 🚀

How did we do it? With a dash of Mongoose magic and a sprinkle of geospatial indexing. We’re dealing with ads, and not the kind that you can skip after 5 seconds. No, these are the persistent kind, and they can’t be all over the same place. We need to make sure each ad is spaced out like introverts at a party. The challenge? Preventing any ad from being placed within 3 meters of another.

#The Solution

Enter Mongoose, our trusty sidekick, armed with geospatial superpowers. By using Mongoose's support for GeoJSON objects and 2dsphere indexes, we can efficiently handle geospatial data and queries. Here's how I did it:

#Schema Design

First, the schema. You know, the thing that defines the structure of our data. Here’s what it looks like:

import mongoose from "mongoose";

const Schema = mongoose.Schema;

const adPostSchema = new Schema({
  // other irrelevant stuff
  Latitude: Number,
  Longitude: Number,
  location: {
    type: { type: String, enum: ["Point"], required: true },
    coordinates: { type: [Number], required: true },
  },
});

adPostSchema.index({ location: "2dsphere" });

const AdPost = mongoose.model("AdPost", adPostSchema);

export default AdPost;

Let me break it down for you:

• location.type: This is always "Point". It’s not rocket science, just geometry.
• location.coordinates: It’s a pair of numbers—longitude and latitude. Yes, the Earth is round.
• 2dsphere index: This is where the magic happens. It tells MongoDB, "Hey, treat this like a globe." Because, guess what? The Earth isn’t flat, no matter what your conspiracy-theorist uncle says.

#Geospatial Query Implementation

To prevent ad overcrowding, we crafted a query to check for existing ads within a 3-meter radius before adding a new one. Think of it as a digital bouncer, ensuring no ad gets too close for comfort:

const postAd = async (req, res) => {
  const {
    // other irrelevent stuff
    Latitude,
    Longitude,
  } = req.body;

  try {
    // Define the location in GeoJSON format
    const location = {
      type: "Point",
      coordinates: [Longitude, Latitude],
    };

    // Check for existing ads within 3 meters (0.003 km)
    const existingAd = await AdPost.findOne({
      location: {
        $near: {
          $geometry: location,
          $maxDistance: 3, // distance in meters
        },
      },
    });

    if (existingAd) {
      return res.status(400).json({ message: "An ad already exists within 3 meters of this location." });
    }

    // Proceed to save the ad if no conflicts are found

  } catch (error) {
    res.status(500).send("Error adding post: ");
    console.log(error);
  }
};

Here's what happens:

• We define the location using GeoJSON
• We use the $near operator to find any ads within a 3-meter radius. If we find one, we block the new ad. Simple as that.

#Conclusion

And that’s it. We managed to keep ads from piling up like trash on a New York street. Thanks to the 2dsphere index, we can efficiently and accurately manage ad placements without the headache. For those who want to dive deeper into the glorious details, check out Mongoose's GeoJSON documentation. Enjoy.

Let's build a Retrieval-Augmented Generation (RAG) application using Groq (Llama-3) and Llama-Index without any OpenAI API keys and with minimal compute. If this project can run on my laptop, it can run on anyone’s laptop because I only have 8 GB of RAM and an SSD—no graphics card, no fancy stuff. But before we dive into the code, let me share a little tale about my experience with Llama-Index.

Llama-Index is a gem. However, its documentation is... how do I put this nicely? Let's just say it could use a lot more hand-holding and a lot less cryptic messaging. I was frustrated because I was unable to build a RAG chat app with my data, my custom prompt, and chat history memory in Llama-Index. The documentation didn’t even hint that it was possible! I almost gave up, but then I decided to dive into the source code of Llama-Index to see how it was implemented. And bless the engineers who wrote that code—it's beautifully well-structured and commented. Finally, I understood how to implement exactly what I wanted and even found some great ways to customize it to fit my needs.

So, if you're ready, let's get our hands dirty and build some cool stuff!

#Setting Up the Environment

First, we need to set up our environment. We use Flask to create a web application and several modules from Llama-Index for document processing and embedding.

pip install Flask requests llama-index==0.10.18 llama-index-llms-groq==0.1.3 llama-index-embeddings-huggingface==0.2.0

Installing a few necessary packages

from flask import Flask, request, render_template
import re
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
StorageContext,
ServiceContext,
load_index_from_storage,
)
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.groq import Groq
from llama_index.core.base.llms.types import ChatMessage, MessageRole
import warnings
warnings.filterwarnings('ignore')

Ignoring warnings is very important because:

#Initializing Flask App and API Key

Next, we initialize the Flask app and set up the API key for Groq.

GROQ_API_KEY = "<insert your API key here>"
app = Flask(__name__)

#Defining the RAG Chatbot Function

The core of our application is the chat_with_rag function. This function takes a user message and chat history, processes the documents, and generates a response using the RAG approach. In your root directory make a txt or a pdf file named “paul_graham_essays.txt” and put in some of his essays

def chat_with_rag(message, history):
		reader = SimpleDirectoryReader(input_files=["./paul_graham_essays.txt"]) 
		documents = reader.load_data()
		text_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=200)
		nodes = text_splitter.get_nodes_from_documents(documents, show_progress=True)
		
		embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
		
		llm = Groq(model="llama3-70b-8192", api_key=GROQ_API_KEY)
		
		prompt = "enter your system prompt here"
		
		service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm, 
		system_prompt=prompt)
		
		vector_index = VectorStoreIndex.from_documents(documents, show_progress=True, 
		service_context=service_context, node_parser=nodes)
		
		vector_index.storage_context.persist(persist_dir="./storage_mini")
		
		storage_context = StorageContext.from_defaults(persist_dir="./storage_mini")
		
		index = load_index_from_storage(storage_context, service_context=service_context)
		
		query_engine = index.as_chat_engine(service_context=service_context,
		chat_mode='condense_plus_context')
		
		resp = query_engine.chat(message, chat_history=history)
		
		return resp.response

Step-by-step Explanation:

• Document Reading: We use SimpleDirectoryReader to read documents from a file.
• Text Splitting: The SentenceSplitter splits the documents into chunks for efficient processing.
• Embedding Model: We use HuggingFaceEmbedding with the sentence-transformers/all-MiniLM-L6-v2 light weight model.
• Language Model: The Groq model is initialized with the llama3-70b-8192 model. (Why Gooq? because it uses LPUs which are insanely fast you should read more about)
• Service Context: Combines the embedding model, language model, and a system prompt. (I found this from source code, its not mentioned in the docs)

• Vector Index: Creates a vector index from the documents and persists it to disk.
• Query Engine: Loads the index and sets up the query engine to handle chat interactions.

with query_engine you can use 2 methods

and another one that I have used is

Something else that you can play with is this line in chat_with_rag function

query_engine = index.as_chat_engine(service_context=service_context, chat_mode='condense_plus_context')

I used chat_mode='condense_plus_context' but you can customize things according to your need

#Setting Up Flask Routes

We define our chat routes in our Flask application

@app.route('/chat', methods=['POST'])
def chat():
		data = request.json
		history_json = data.get('messages', [])
		system_prompt = {
		    'role': 'system',
		    'content': '<enter your system prompt here optional>'
		}
		history_json.insert(0, system_prompt)
		
		user_message = ''
		history = []
		
		if history_json:
		    user_message = history_json[-1]['content']
		    history = [convert_to_chat_message(msg['role'], msg['content']) for msg in history_json]
		
		bot_response = chat_with_rag(user_message, history=history)
		
		return {'response': bot_response}

Explanation:

• Chat Route: Handles POST requests with user messages, processes the chat history, and returns the bot's response. data.get.messages is an array of json objects that should come from your client or from your other server if you have a microservice arch. Each json should be in the form

{role:"user or assistant", content:"some sample text"}

#Helper Function

The convert_to_chat_message function converts chat history from JSON format to ChatMessage objects. This step is very important if you want to maintain a chat_history and basically chat with your RAG and guess what…… this step is not mentioned in the docs 😞

def convert_to_chat_message(role: str, content: str) -> ChatMessage:
    """Converts a role and content into a ChatMessage object."""
    # Convert role string to MessageRole enum
    role_enum = MessageRole(role)
    # Create and return a ChatMessage object
    return ChatMessage(role=role_enum, content=content)

#Running the Application

Finally, we run the Flask app.

if __name__ == '__main__':
    app.run(debug=True)

#Conclusion

By following these steps, you can build a RAG application using Groq and Llama-Index without relying on OpenAI API keys. This setup ensures minimal compute requirements while maintaining the efficiency and effectiveness of the RAG approach. Just remember, while Llama-Index might have a bit of a learning curve due to its documentation, it's definitely worth the effort. Happy coding!