3 hints available

MediumAI EngineeringTheoryRAG Architecture Token Efficiency Prompt Engineering

RAG System Token Cost Optimisation

A RAG system is running in production with the following configuration:

10,000 user requests per day
Each request retrieves top-10 chunks, each chunk ~300 tokens
System prompt: 800 tokens
Average user question: 50 tokens
Using GPT-4o ($2.50 per million input tokens)
Average response length: 200 tokens ($10 per million output tokens)

Calculate the daily cost of this system broken down by: system prompt tokens, retrieved context tokens, user question tokens, output tokens.
The team wants to reduce costs by 60% without degrading answer quality. Propose three specific optimisations, estimate the token savings from each, and explain any quality tradeoffs.
The system prompt is identical on every request. What specific API feature would you use to reduce the cost of the system prompt tokens, and by approximately how much would it reduce that component of the cost?

Sign in to attempt this problem

Free account gives you full access to community problems with the complete solution reveal: golden answer, senior walkthrough, and score breakdown, after submission.

Start free →Already have an account? Sign in