Is RAG Dead? Why Long Context Windows Haven't Killed Retrieval Augmented Generation
I built a RAG chat on my portfolio, then wondered whether million-token context windows made the whole pipeline pointless. Here's how I think about the tradeoff now.
I built RAG, then wondered if I was doing it the hard way
When models started shipping with million-token context windows, my first reaction was mild embarrassment. I was chunking my own blog posts into 500-character pieces, embedding them, ranking them with cosine similarity. For a corpus that would fit comfortably in a single prompt.
So is RAG dead? That's the question I had while building the chat on my portfolio. I'm not here to settle Twitter arguments. I wanted to know what I'd ship this week versus what I'd ship at work.
The part I kept forgetting: models are frozen at their training cutoff. They don't know my blog posts or my resume unless I put them in the prompt. That's the context injection problem. Roughly two ways to solve it.
Approach 1: RAG
RAG is the engineering approach. Chunk documents, embed the chunks, store vectors somewhere, retrieve the relevant pieces at query time, inject them into the prompt with the user's question.
It works if retrieval works. That second part gets glossed over in a lot of demos.
This is what I built on my portfolio, even though my whole knowledge base would fit in one context window with room to spare. I wanted source cards and a way to see which chunk screwed up. That mattered more to me than the shortest path to answers.
Approach 2: Long context
Long context is brute force. Skip the database, skip the embedding model, dump your documents into the context window and let the model figure it out.
For a long time that wasn't viable. Early models had 4K token windows. You couldn't fit a novel, let alone a corporate knowledge base. Today's models support 1M+ tokens. That's roughly 700,000 words. The Lord of the Rings trilogy plus The Hobbit.
Part of me wanted to skip all of it and paste my resume plus five blog posts into the system prompt. I didn't. The temptation was real though.
Where long context wins
Infrastructure is the boring reason. A production RAG stack has chunking strategy, an embedding model, a vector database, maybe a reranker, sync logic to keep everything current. Long context removes most of that. Get the data, send it to the model.
The failure mode that got me: silent failure. Semantic search is probabilistic. Sometimes it returns the wrong chunks. The answer existed in your data, the model never saw it, and nothing obvious tells you that happened. With long context there's no retrieval step to go wrong.
When I first wired up top-5 retrieval on my portfolio chat, I saw a milder version of this. The answer pulled the right chunk from my About section and an unrelated chunk from a blog post. Technically grounded, still wrong. Long context wouldn't fix bad prompting, but it would have removed one failure mode entirely.
RAG can't answer questions about what's missing either. Product requirements in one doc, release notes in another. Ask which security requirements got cut from the final release. RAG will surface snippets about security and requirements. It can't retrieve the gap between them. You need both documents in full, side by side. You can't chunk your way to that answer.
Small corpus, need to reason across the whole thing? Long context is often simpler.
Where RAG still wins
The cost argument is the one I hear most. Putting a 500-page manual in a context window costs roughly 250K tokens on every query. RAG pays that once at indexing time. At real query volume, the difference adds up.
At my scale, token cost isn't the argument. I'm not paying for a 500-page manual on every query. What mattered to me was focus. Five paragraphs instead of everything I've ever written.
What surprised me was attention. If data is in the window, you'd think the model would use it. Past a few hundred thousand tokens, that breaks down. Ask about one paragraph buried in a 2,000-page document and the model often misses it, sometimes hallucinating from nearby text instead. RAG narrows the window to relevant chunks.
Then there's raw scale. A million tokens sounds enormous until you remember enterprise data lakes are measured in terabytes. No context window is touching that. Querying across an organization's full knowledge base still needs a retrieval layer.
Big corpus, data that changes every day, lots of queries: RAG is still the only thing that fits.
What I'd actually do
On my portfolio (~6k words, static, source cards in the UI): RAG, which is what I shipped. Citations and debugging were worth the extra code.
Same corpus, no UI, just answers: I'd probably long-context the whole thing and call it a day.
Company internal docs at scale: RAG, for cost, freshness, and size.
If I were starting the portfolio chat from zero today, I might paste the whole corpus into the prompt first, ship in an afternoon, and add retrieval only when the corpus outgrew the window or I needed better observability. RAG on a six-post blog is over-engineering. I'm fine with that because I wanted to understand the pipeline, not because every side project needs a vector store.
Picking one (or both)
| Use case | Approach |
|---|---|
| Bounded dataset, global reasoning (legal analysis, book summarization) | Long context |
| Large enterprise knowledge base | RAG |
| Dynamic, frequently changing data | RAG |
| Small team, minimal infrastructure overhead | Long context |
I'll probably end up using both anyway. RAG to find the right chunks, long context to reason over them.
Hand-rolled RAG on my portfolio today. pgvector next, probably. Long context didn't kill RAG for me. It just made the question sharper: am I trying to fit data in the window, or find the right paragraph first?
Side project? Ship the demo. Ignore the hype.
Inspired by Martin Keen's IBM Technology video "Is RAG Still Needed? Choosing the Best Approach for LLMs," after building my portfolio chat.