It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.

It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.

(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)

The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.

RAG is routing and decision making.

I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.

I agree with you.

Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.

It maintains itself.

Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.

I found my niche with RAG selling and I build them myself.

I take pride in them.

So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.

And there are regulated businesses where certain information is required to be always factual correct - pricing for example.

Vector search becomes a liability for this use case.

So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.

I love this kind of stuff.

And there is personal information etc.

Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.

And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.

It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.

I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.

RAG gives me dbCORE vibes so to say.

Good luck and fun with your RAG systems.

https://www.anthropic.com/engineering/contextual-retrieval

This is from two years ago, but I think it's still SotA?

Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)

It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.

Similar to SEO on marketing pages, we started rewriting product docs around the idea that it will be consumed by a RAG. Mostly by putting a lot of focus on well structured headlines, thinking more carefully about technical terminology vs common human-language questions, occasionally using variations of keywords in the text, etc. This applies to pure LLM consumption too, not just hybrid search.

Once you start tracking what users are asking you learn to adapt the documentation around it. And LLMs can also suggest improvements by comparing questions vs search results vs LLM responses.

No - rag is doing search before you call the llm to give it context from some corpus like your helpdesk articles.

We've all learnt that it's not really communication, and so can be dispensed with.

"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model

That is where I stopped

"Recipe 4: On-The-Fly Embedding (The Fresh Data Play)

The insight If your data changes frequently, why pay to re-embed everything?"

This reads like every Claude generated presentation I've seen.

This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”

RAG is simply fetching external data (retrieval) and adding it to LLM context (augmenting) prior to generating a final response.

Any time LLMs do a grep or a web search to answer the query, it’s RAG. Many people use vector db for their own RAG implementation bc of the semantic search benefits.

If you're looking for feedback, I'd suggest adding a short demo at the end. It would be nice to see you send it a prompt that says, "hey, remember this" and then tell it to recall that memory. Or show what the memories look like on their way to the model. Are the memories added to the context on every turn or only once per conversation?

step 1: sparse index retrieval (FTS/BM25) - say with k = 10 step 2: re-rank the 10 records using embeddings

the difference in this approach is during step 2, you convert text to embeddings on the fly - when you're running the retrieval pipeline, meaning you don't need to have all of your corpus pre-embedded in a vector db

https://cursor.com/blog/semsearch