Context Window Management
Select, rank, and structure retrieved content to fit within LLM token limits while preserving the information quality needed for accurate generation.
Every LLM enforces a hard token limit on the combined length of system instructions, retrieved context, conversation history, and the generated response. RAG systems retrieve far more content than fits in that window. Context window management is the discipline of deciding which retrieved chunks make it into the prompt, in what order, and in what form.
OpenBase implements a multi-stage context manager that scores retrieved chunks by semantic relevance, re-ranks them with a cross-encoder, applies business rules for recency and authority, then assembles a structured context that fits the target model's budget. When retrieved content exceeds the available window, the manager compresses long documents through extractive summarization and preserves metadata so the agent can still cite sources accurately.
This capability runs on every query. The context manager monitors token usage in real time, reserves budget for system prompts and conversation history, and adjusts the retrieved content allocation dynamically. The result is a prompt that uses the available context window efficiently without truncating critical information or wasting tokens on redundant chunks.
Problem & solution
Retrieval returns more than the model can read
A semantic search across your knowledge base returns twenty relevant chunks. Your LLM has an 8,000-token context window. System instructions take 800 tokens. Conversation history takes 1,200. The twenty chunks total 12,000 tokens. You must decide which ten chunks to keep, in what order, and whether to compress any of them — and you must make that decision in milliseconds, on every query, without breaking source attribution.
Staged ranking and dynamic assembly
OpenBase scores every retrieved chunk by semantic similarity to the query, then re-ranks the top candidates with a cross-encoder that evaluates query-chunk relevance more precisely. Business rules layer on top: recent documents score higher for time-sensitive queries, authoritative sources rank above informal notes, and user permissions filter out inaccessible content. The context manager then allocates the remaining token budget, fitting as many high-scoring chunks as possible while reserving space for instructions and history. Long chunks are summarized extractively; metadata and source references are preserved so Donna can cite accurately even when the full text was compressed.
What you see after 90 days
- Prompts that use 90%+ of available context window without truncation
- Sub-100ms context assembly latency per query
- Source citations that remain accurate after summarization
- Token cost reduction of 30-40% through intelligent compression
- Consistent generation quality across queries of varying complexity
Who benefits most
- AI leads tuning RAG performance for production workloads
- Platform teams managing LLM cost at scale
- Compliance officers requiring audit trails on retrieved content
- Domain experts who need answers grounded in specific, citable sources
Frequently asked questions
How does OpenBase decide which retrieved chunks to drop when the context window is full?
The context manager ranks chunks by a composite score: semantic similarity from the vector search, cross-encoder relevance, recency weight, source authority, and user access level. Lowest-scoring chunks are dropped first. If a high-scoring chunk is too long, it is summarized extractively rather than dropped entirely.
Does summarization break source attribution?
No. OpenBase preserves the original chunk metadata and document reference through the summarization step. When Donna cites a source, the citation links to the full document even if the context manager compressed the chunk before adding it to the prompt.
Can I configure how much of the context window is reserved for conversation history?
Yes. The context manager exposes a token budget policy per tenant. You can set fixed reserves for system instructions and conversation history, or use dynamic allocation where history budget shrinks when retrieved content is especially relevant.
What happens when a query retrieves nothing relevant and the context window is mostly empty?
The context manager still assembles a valid prompt with system instructions and conversation history. Donna generates a response indicating no relevant documents were found, rather than hallucinating content to fill the window.
How does context window management interact with multi-turn conversations?
The context manager tracks conversation history across turns and includes prior exchanges in the token budget. As the conversation grows, older turns are summarized or dropped to make room for new retrieved content, but the most recent exchange is always preserved in full.
Can I see which chunks were included or dropped for a given query?
Yes. OpenBase logs the context assembly decision for every query: which chunks were retrieved, their relevance scores, which made it into the final prompt, and which were compressed or dropped. These logs are available in the tenant audit trail.
Does OpenBase support models with very large context windows like GPT-4 Turbo or Claude 100k?
Yes. The context manager adapts to the target model's token limit. For large-window models, it includes more retrieved chunks and compresses less aggressively. The same ranking and prioritization logic applies regardless of window size.
What is the performance overhead of re-ranking with a cross-encoder?
Cross-encoder re-ranking adds 40-80ms per query for typical retrieval sets of 10-20 chunks. OpenBase runs the cross-encoder only on the top-k results from the initial vector search, not on the entire corpus, keeping latency acceptable for interactive use.
In this cluster
Hub: enterprise-rag-architecture