RAG, fine-tuning or prompting: how to add an LLM to your product

Start with prompting, add retrieval-augmented generation (RAG) when the model needs facts from your own data, and fine-tune only when you need a consistent style, format or narrow skill that prompting cannot deliver. In most business products the winning setup is a good prompt, structured outputs, retrieval over your documents and an evaluation set to prove it works. Fine-tuning is the last step, not the first, and agents suit a small set of problems.

Key takeaways

• Prompting changes what you ask the model, RAG changes what the model can see, and fine-tuning changes the model itself; they solve different problems and are often combined.
• If the answer depends on documents that change, RAG is almost always the right first move, because you update an index instead of retraining a model.
• Fine-tuning teaches behaviour (tone, format, classification boundaries), not facts, and it is a poor way to make a model “know” your knowledge base.
• A RAG system is mostly a search problem: chunking, hybrid search and reranking decide quality more than the choice of language model.
• Agents add cost, latency and failure modes, so use a fixed workflow whenever the steps are known in advance.
• Privacy is an architecture decision: where documents are stored, which provider processes them, in which region, and who may see which passage.
• Whatever you choose, measure it with an evaluation set and a cost per answer before users see it.

What are RAG, fine-tuning and prompting?

They are three ways to make a large language model (LLM) useful for your task, and they work at different layers.

Prompting means giving the model instructions, examples and context in the request itself. Nothing is trained. You can change behaviour in minutes and roll it back with a deploy.

RAG (retrieval-augmented generation) means the system first searches your data for relevant passages, puts them into the prompt and asks the model to answer from them. The model is unchanged. Your knowledge lives in a search index that you can update at any time.

Fine-tuning means training an existing model further on your examples, so its weights change. It needs a training set, an evaluation set, and a process to repeat it when the base model is replaced.

How do you choose between them?

Choose by what is missing, knowledge or behaviour, and start with the cheapest option that could work.

Criterion	Prompting (with tools and structured output)	RAG	Fine-tuning
Data freshness	Only what you paste into the prompt	As fast as you re-index	Frozen at training time
Cost to build	Lowest	Medium: ingestion, search, evaluation	Highest: data prep, training, evaluation
Cost per answer	Depends on prompt length	Longer inputs (retrieved passages)	Often shorter prompts; sometimes a smaller model
Latency	One model call	Retrieval plus one model call	One call; small tuned models can be fast
Accuracy on your facts	Poor unless facts are in the prompt	Good, and verifiable with citations	Unreliable for facts, good for style and format
Effort to maintain	Prompt versioning	Pipeline, index, access rules	Retraining when data or base model changes
Privacy surface	Whatever you send per request	Documents stored in your index; snippets sent	Training data leaves your systems unless self-hosted

If facts change weekly and users need to verify answers, RAG wins. If you need a precise house style at high volume, consider fine-tuning. If you are still discovering what the feature should do, stay with prompting. For why so many AI projects stall after the demo, see AI in your business: from an impressive demo to something people use.

How does a RAG system work, step by step?

A RAG system has an offline phase that prepares documents for search and an online phase that retrieves passages and writes a cited answer.

1. Ingestion. Get content out of PDFs, wikis, help articles, tickets and databases into clean text with metadata: document ID, URL, page, version, date, and who may read it. This is the unglamorous part that decides everything. If a table turns into a stream of numbers, no model will recover its meaning.

2. Chunking. Split documents into passages small enough to retrieve precisely and large enough to make sense alone. A common start is a few hundred tokens, split on headings and paragraphs with a little overlap. Better is structure-aware splitting: keep a section with its heading and a table with its caption, and prepend the document title and section path to each chunk. Test sizes on your evaluation set; there is no universal best value.

3. Embeddings. An embedding is a list of numbers representing the meaning of a text, produced by an embedding model. Similar meanings land close together, so “How do I cancel?” can match a passage titled “Ending your subscription”. Embed every chunk once and each query at question time. Multilingual content needs a multilingual model, and changing the model later means re-embedding everything, so record the model version.

4. Vector store. If you already run PostgreSQL, the pgvector extension keeps vectors beside the rest of your data, with the same backups, transactions and access control, and lets you filter by tenant or permission in the same SQL query. That is often the simplest correct answer for a long time. Move to a dedicated vector database when measured scale demands it, not before.

5. Hybrid search. Combine vector search (good at meaning and paraphrase) with keyword search such as PostgreSQL full-text search or BM25 (good at product codes, error numbers, names and clause numbers). Run both and merge the ranked lists, for example with reciprocal rank fusion. Adding keyword search to a vector-only system fixes many “it missed the obvious document” complaints.

6. Reranking. Take the top candidates, say 30, and re-score them with a model that reads the query and each passage together (a cross-encoder or an LLM-based reranker). Keep the best handful. Retrieval is fast and wide; reranking is accurate and narrow. It adds latency and cost, so keep it only if it improves your evaluation scores.

7. Generation with citations. Instruct the model to answer only from the passages, reference the passage IDs it used, and say “I don’t know” when the passages lack the answer. Your application turns those IDs into links. Citations are the most effective trust feature in a RAG product: they turn “believe me” into “check for yourself”. Also verify in code that every cited ID was actually retrieved.

8. Permissions. Filter chunks by what the current user may read before anything reaches the model. A prompt saying “do not reveal confidential documents” is not a security control.

When does fine-tuning actually help?

Fine-tuning helps when you need consistent behaviour that prompting cannot deliver reliably, or a smaller, cheaper model for one narrow job.

Good reasons:

• A strict style or format at high volume, where long prompts are costly or flaky.
• A narrow classification or extraction task with plenty of labelled examples, where a small tuned model can match a large general one at lower cost and latency.

Poor reasons:

• “We want it to know our documents.” Fine-tuning is unreliable at memorising facts, cannot cite sources and goes stale when documents change. Use RAG.
• “Answers are sometimes wrong.” Without an evaluation set you do not know why. Often the cause is retrieval or a vague prompt.
• “We have no examples yet.” You need hundreds to thousands of reviewed input-output pairs, and collecting them is most of the work.

Before committing, build the strongest baseline you can: a good prompt with examples, structured outputs and retrieval. If it already meets your target, stop. If you do fine-tune, keep the tuned model and the baseline in the same evaluation harness, so you can prove the gain and notice when a new base model makes yours obsolete. See how to evaluate an LLM feature before it reaches production.

How far can prompting, tools and structured outputs go?

Further than most teams expect; a well-designed prompt with structured outputs and a few tools covers a large share of real business features.

• System prompt and examples. State the role, rules and refusal behaviour, and add three to ten examples including hard cases.
• Structured outputs. Ask for JSON matching a schema and validate it in code. This turns a text generator into a component: classify the ticket, extract invoice fields, route the request. On validation failure, retry or escalate.
• Tool use (function calling). Give the model a short list of functions, such as get_order_status(order_id). Your code runs them, with permission checks, and returns the result. The model never touches your database directly.
• Model routing. Use a small, fast model for classification and a larger one only where it matters.

Do you need an agent?

Usually not: use an agent only when the sequence of steps cannot be known in advance, and a fixed workflow when it can.

An agent is an LLM in a loop: it picks a tool, looks at the result, picks the next step and repeats until it decides it is done. That suits open-ended tasks, such as investigating a problem across several systems.

The costs are real. Each iteration is another model call, so latency and spend multiply. Errors compound, the same input can follow different paths, and the more tools an agent holds, the more damage a wrong or manipulated decision can do.

If you can write the steps down (classify, retrieve, draft, validate, hand to a person), build a workflow where each step is testable and priced alone. Where one step needs flexibility, give only that step some autonomy, with caps on iterations, spend and permitted actions. Keep human approval for anything with external consequences, such as sending an email, issuing a refund or changing a record.

What about privacy, GDPR and data residency?

Privacy is decided by architecture: what data goes to which provider, where it is processed, how long it is kept, and who can see which passage.

This is not legal advice, and your counsel should confirm the details, but if you handle personal data of people in the EU or UK, the engineering checklist is:

1. Map the data flow. Which fields reach the model? Can names and identifiers be masked first?
2. Read the provider terms for your plan: training on prompts, retention, and whether a data processing agreement is available.
3. Check region options. Some providers and cloud platforms process in specific regions, including the EU. This differs by model and changes over time, so verify it for the exact model you plan to use.
4. Keep the index in your own cloud account, in the region you choose. Only retrieved snippets travel to the model.
5. Enforce per-user access at retrieval time and log which passages were shown to whom.
6. Plan for deletion. Know which chunks, embeddings, logs and caches hold a person’s data.
7. Consider a self-hosted open-weights model if contracts or regulation rule out third-party processing. It adds operations work.

Fine-tuning complicates deletion requests, because training data may be reflected in the weights. Hosting and data setup go hand in hand with our Cloud & DevOps work.

What does one answer cost? A worked example

Cost per answer is input tokens plus output tokens multiplied by the model’s prices, plus a small retrieval cost; here is a hypothetical calculation.

Imagine a support assistant. Every number below is invented for illustration; use your provider’s current price list. Assume a hypothetical premium model at €3 per million input tokens and €15 per million output tokens, and a cheaper one at a tenth of those prices.

Item	Tokens
System prompt and rules	600
Five retrieved passages, about 400 each	2,000
Conversation history and the question	400
Total input	3,000
Answer with citations (output)	300

On the premium model, input costs 3,000 × €3 / 1,000,000 = €0.009 and output costs 300 × €15 / 1,000,000 = €0.0045, about €0.0135 per answer. Adding embedding and reranking calls, which are typically much smaller, call it roughly €0.015. At 20,000 answers a month that is about €300; on the cheaper model, about a tenth of that.

An agent loop making eight calls with growing context could cost many times more, which is why cost per answer should be measured, not assumed. The biggest levers are fewer, better passages and a smaller model for the easy majority of questions.

What are the common failure modes?

Most failures come from retrieval and missing evaluation, not from the model.

• Retrieval misses. The right passage exists but is not retrieved. Fix with hybrid search, better chunking, query rewriting and reranking.
• Confident answers without support. The model fills gaps with plausible text. Fix with “answer only from the passages”, mandatory citations and a tested “I don’t know” path.
• Permission leaks. A user sees a passage they should not. Filter at retrieval and test with accounts of different roles.
• Prompt injection. A document says “ignore previous instructions” and the model obeys. Treat retrieved text as untrusted and limit tool permissions.
• Silent regressions. A prompt tweak or provider model update changes behaviour unnoticed. Run the evaluation set on every change.
• Runaway cost. Long prompts, large models everywhere, agent loops. Track cost per answer and set alerts.

How Hidook approaches this

Hidook builds retrieval systems, copilots and agents as part of its AI and LLM integration work. We start with the simplest thing that could work, usually prompting with structured outputs, then add retrieval when the feature needs your facts. No model reaches production without an evaluation set and a cost per answer, and we measure both from week one.

In practice that means a short discovery to find the first question worth automating, a working version on a live URL every Wednesday, and decisions about chunking, search and models recorded so your team can understand them later. You own the repository and the cloud accounts from day one. If you want to start smaller, see our MVP development work.

Frequently asked questions

Is RAG better than fine-tuning?

They solve different problems. RAG gives the model your current facts and lets you show sources, while fine-tuning changes behaviour such as tone or format. For question answering over documents, RAG is the usual first choice.

Can I fine-tune a model on my documents so it knows them?

You can, but it is a weak way to add knowledge. Fine-tuned models tend to blend facts, cannot point to a source and go out of date when documents change. Retrieval keeps knowledge in an index you can update and audit.

Do I need a vector database for RAG?

Not necessarily. If you already use PostgreSQL, the pgvector extension is often enough and keeps vectors next to your other data, with the same permissions and backups. Move to a dedicated vector database only when measured scale or features require it.

How much does a RAG answer cost?

It depends on the model and the tokens you send. As a hypothetical, a 3,000-token prompt and a 300-token answer on a premium model can cost around a cent or two. Calculate yours from current provider prices and measure it in production.

When should I use an agent instead of a workflow?

Only when the steps genuinely cannot be known in advance. If you can write the steps down, a fixed workflow is cheaper, faster, easier to test and safer. Where you do use an agent, cap its iterations, spend and permitted actions.

Is it safe to send company data to an LLM provider?

It can be, if you choose the plan, region and settings deliberately. Check retention and training terms, confirm a data processing agreement, minimise personal data in prompts and enforce access rules before retrieval. Have your legal advisor confirm the specifics.

If you are weighing these options for a real product, write to no.bs@hidook.tech or book a call at https://hidook.tech/contact/ and a senior engineer will reply within one working day.