RAG is Not Magic: Lessons from Production

We integrated OpenAI's GPT-4 into a legal firm's document search. Here is what failed immediately.

1. Vector Search isn't Semantic Understanding Searching for "breach of contract" in a vector DB might return a document that says "There was NO breach of contract" because the vectors are similar. **Fix:** You need a Re-ranking step. - Step 1: Vector similarity (Get top 50 matches). - Step 2: Cross-Encoder model (Re-rank specific to query). - Step 3: Pass top 5 to LLM.

2. The Context Window Limit Even with 128k context, stuffing 50 pages of legal text degrades reasoning. "Lost in the Middle" phenomenon is real. **Strategy:** Chunking is an art key. - Don't just split by character count. - Split by logical section headers.

3. Cost Control User: "Summarize this 500 page PDF." API Cost: $4.00 per click. **Solution:** Caching is mandatory. Hash the input prompt and document ID. If they ask again, return the cached summary.

Code Snippet: Robust Chunking ```python def semantic_chunking(text): # Don't break sentences. Use spaCy or NLTK. doc = nlp(text) chunks = [] current_chunk = "" for sent in doc.sents: if len(current_chunk) + len(sent) < 500: current_chunk += sent.text else: chunks.append(current_chunk) current_chunk = sent.text return chunks ```