Guide · AI in existing apps
RAG for an existing SaaS: architecture and cost per query
By MervenorPublished Last reviewed 4 min read
Retrieval-augmented generation (RAG) answers questions from your own documents and data by finding the right passages first and only then asking a model to write. In an existing product the hard parts are not the model: they are keeping the index in step with your data, enforcing each user's permissions and proving the answers are right. Start with a set of real questions, then build the smallest pipeline that answers them well.
When RAG is the right tool
RAG suits questions whose answers already exist in text: help-centre articles, contracts, tickets, specifications, internal wikis. It is the wrong tool for calculations or live figures from your database; for those, let the model call your API instead. Fine-tuning changes how a model writes, not what it knows about your latest data, so it rarely replaces retrieval.
The architecture
Sources
the systems the text lives in, with their own permissions.
Ingestion
extract the text, clean it, split it into chunks and add context to each chunk.
Index
store an embedding (a numeric fingerprint of meaning) for each chunk, plus a keyword index.
Retrieval
search both indexes, filtered by what this user may see.
Reranking
reorder the best candidates with a model trained for relevance.
Generation
answer from the retrieved passages only, with citations back to the source.
Logging and evaluation
record questions, retrieved passages and answers, and score them.
Where the vectors live
If your product already runs PostgreSQL, the pgvector extension keeps embeddings next to the data they describe, inside the database, backups and permissions you already manage. A dedicated vector database earns its place at larger scale, with heavy filtering or very low latency targets. Start with what you already operate, and move only when measurements say so.
Retrieval quality
Most bad answers are retrieval failures: the right passage was never found. Three techniques help. Add context to each chunk before indexing it, so a passage from page 40 still says which document and section it came from. Combine embedding search with keyword search (BM25), which catches exact names and codes. Then rerank the candidates. Anthropic's tests of this approach reported 35% fewer failed retrievals with contextual embeddings, 49% fewer when contextual keyword search was added, and 67% fewer with reranking on top.
Permissions
Filter by tenant and user at retrieval time, inside the query, so a passage the user may not see never reaches the model. Never rely on the prompt to keep data apart. The OWASP Top 10 for LLM applications (2025) lists vector and embedding weaknesses and sensitive information disclosure among its main risks for exactly this reason.
Keeping the index fresh
Update the index when the source changes (an event or a queue), not on a nightly guess. Handle deletions as carefully as additions, so a removed document stops being quoted. Plan a full rebuild for the day you change the embedding model, because old and new embeddings cannot be compared.
Evaluation
Before building, collect a set of real questions with the passages that should answer them. Measure two things separately: did retrieval find the right passages, and was the answer correct and supported by them? Run the set after every change to chunking, models or prompts, and review a sample of real answers by hand each month.
What drives the cost per query
- Context size: the number of retrieved passages sent to the model, usually the largest lever.
- The model: a larger model for hard questions, a smaller one for simple ones.
- Reranking: an extra model call per query, often worth it.
- Embeddings: paid once per chunk at ingestion and once per query; re-embedding everything after a model change.
- Infrastructure: the index, its replicas and its backups.
- Caching: repeated questions answered from a cache cost a fraction of new ones.
- Volume: how many users ask how often.
Questions people ask
Can we build this on the database we already run?
Often, yes. With PostgreSQL and pgvector, retrieval lives next to your data; measure before adding another system.
Do we need to fine-tune a model?
Rarely. Retrieval keeps answers current and citable; fine-tuning changes style, not knowledge of last week's documents.
How do we know it is good enough to ship?
Agree a pass mark on the evaluation set before building, and ship only when the system meets it.
Ingenara, AI feature integration, starts with an evaluation set of your real questions and ships only when it meets the agreed threshold, at a fixed scope and price.