The Question Every AI Team Asks
You've got a use case. Your LLM needs domain-specific knowledge. Do you fine-tune a model on your data, or do you build a RAG pipeline to retrieve relevant context at inference time?
This is the single most consequential architecture decision in applied AI today, and most teams get it wrong — not because they pick the wrong approach, but because they don't have a clear framework for deciding.
Let's fix that.
Understanding the Fundamentals
Before we can compare, let's be precise about what each approach actually does.
What Fine-Tuning Does
Fine-tuning modifies the model's weights by training on your specific data. The knowledge becomes "baked into" the model itself.
# Conceptual fine-tuning pipeline
from openai import OpenAI
client = OpenAI()
# Step 1: Prepare your training data
# Each example teaches the model a behavior or knowledge pattern
training_data = [
{
"messages": [
{"role": "system", "content": "You are a customer support agent for Acme Corp."},
{"role": "user", "content": "How do I reset my password?"},
{"role": "assistant", "content": "To reset your Acme Corp password: 1) Go to acme.com/reset, 2) Enter your registered email, 3) Click the reset link sent to your inbox. The link expires in 24 hours. If you don't receive the email, check your spam folder or contact support@acme.com."}
]
},
# ... hundreds or thousands more examples
]
# Step 2: Upload and fine-tune
# This creates a new model variant with your knowledge embedded
fine_tune_job = client.fine_tuning.jobs.create(
training_file="file-abc123",
model="gpt-4o-mini-2024-07-18",
hyperparameters={
"n_epochs": 3,
"batch_size": 4,
"learning_rate_multiplier": 1.8
}
)What changes: The model's internal representations shift to better handle your specific patterns, terminology, tone, and domain knowledge.
What RAG Does
RAG retrieves relevant external documents at query time and feeds them to the model as context. The model's weights remain unchanged.
# Conceptual RAG pipeline
def answer_question(query: str) -> str:
# Step 1: Find relevant documents from your knowledge base
relevant_docs = vector_store.search(
query_embedding=embed(query),
top_k=5
)
# Step 2: Feed documents as context to the base model
context = "\n\n".join([doc.content for doc in relevant_docs])
response = client.chat.completions.create(
model="gpt-4o", # Base model, no fine-tuning needed
messages=[
{"role": "system", "content": f"Answer based on this context:\n{context}"},
{"role": "user", "content": query}
]
)
return response.choices[0].message.contentWhat changes: Nothing about the model itself. You're augmenting its input with retrieved information.
The Decision Framework
Here's the framework we use. Score your use case on each dimension, then follow the recommendation.
Dimension 1: Knowledge Type
| Your knowledge is... | Best approach |
|---|---|
| Factual and frequently updated (product docs, policies, prices) | RAG |
| Behavioral and stylistic (tone, format, reasoning patterns) | Fine-tuning |
| Structured domain expertise (medical, legal, financial terminology) | Fine-tuning + RAG |
| Massive and diverse (entire company knowledge base) | RAG |
Why this matters: Fine-tuning is great for teaching a model *how* to respond. RAG is great for telling it *what* to respond with.
# Example: When you need STYLE (fine-tuning territory)
# Your legal team needs the model to always respond in proper legal format
# Fine-tuning teaches the format pattern:
training_example = {
"messages": [
{"role": "user", "content": "Draft a non-compete clause for a software engineer in California."},
{"role": "assistant", "content": """RESTRICTIVE COVENANT — NON-COMPETITION
1. SCOPE AND DURATION
1.1 Subject to the limitations set forth in Cal. Bus. & Prof. Code § 16600,
Employee acknowledges that...
2. GEOGRAPHIC LIMITATION
2.1 This covenant shall apply within...
[Note: California generally prohibits non-compete agreements per
Bus. & Prof. Code § 16600. This clause may be unenforceable.]"""}
]
}
# Example: When you need FACTS (RAG territory)
# Your support team needs answers based on today's product documentation
# RAG retrieves current docs:
def handle_support_query(query):
# Always gets the LATEST documentation
docs = retrieve_from_knowledge_base(query)
return generate_answer(query, context=docs)Dimension 2: Data Freshness Requirements
| Scenario | Best approach |
|---|---|
| Knowledge changes daily or weekly | RAG |
| Knowledge is relatively stable (changes quarterly or less) | Fine-tuning or RAG |
| Knowledge is static and well-defined | Fine-tuning |
| Mix of stable patterns + dynamic facts | Fine-tuning + RAG |
The key insight: Fine-tuning is a batch process. Every time your knowledge changes, you'd need to retrain. RAG pipelines can be updated in minutes by re-indexing documents.
Dimension 3: Accuracy Requirements
# RAG gives you CITATIONS and TRACEABILITY
def rag_answer_with_sources(query: str) -> dict:
docs = retrieve(query)
answer = generate(query, docs)
return {
"answer": answer,
"sources": [
{"title": d.title, "url": d.url, "chunk": d.content[:200]}
for d in docs
],
# You can VERIFY the answer against the source material
"verifiable": True
}
# Fine-tuning gives you CONSISTENCY but no citations
def finetuned_answer(query: str) -> dict:
answer = finetuned_model.generate(query)
return {
"answer": answer,
"sources": [], # The knowledge is in the weights — no traceability
"verifiable": False # You can't easily verify WHERE it got this info
}| Requirement | Best approach |
|---|---|
| Answers must be traceable to source documents | RAG |
| Need to cite specific pages/sections | RAG |
| Consistency of format matters more than factual precision | Fine-tuning |
| Both factual accuracy AND format consistency needed | Fine-tuning + RAG |
Dimension 4: Cost Analysis
Let's do the real math:
# Cost comparison for a typical use case:
# - 10,000 queries per day
# - ~50 domain documents (500 pages total)
# - Needs domain-specific answers
# === RAG Approach ===
rag_costs = {
# Embedding cost (one-time + incremental)
"initial_embedding": 0.13 * 2, # ~2M tokens at $0.13/1M = $0.26
"monthly_re_embedding": 0.13 * 0.5, # ~500K tokens for updates = $0.065/month
# Vector DB (e.g., Pinecone starter)
"vector_db_monthly": 70, # $70/month
# Per-query costs (embedding + retrieval + generation)
"per_query_embedding": 0.00013, # Embed the query
"per_query_generation": 0.005, # GPT-4o with ~2K context tokens
# Monthly total for 10K queries/day
"monthly_total": 70 + (0.00013 + 0.005) * 10000 * 30 # = $70 + $1,539 = ~$1,609
}
# === Fine-Tuning Approach ===
finetune_costs = {
# Training cost (one-time, retrain monthly)
"training_per_run": 25, # ~1M training tokens at $25/1M
"monthly_training": 25, # Monthly retraining
# Per-query costs (no retrieval, just generation)
"per_query_generation": 0.003, # Fine-tuned 4o-mini, shorter prompts
# Monthly total for 10K queries/day
"monthly_total": 25 + 0.003 * 10000 * 30 # = $25 + $900 = ~$925
}
# === Hybrid Approach ===
hybrid_costs = {
"training_per_run": 25,
"vector_db_monthly": 70,
"per_query_total": 0.004, # Fine-tuned model + selective retrieval
"monthly_total": 25 + 70 + 0.004 * 10000 * 30 # = $95 + $1,200 = ~$1,295
}Key takeaway: Fine-tuning has lower per-query costs because you skip the retrieval step and can use shorter prompts. RAG has higher ongoing costs but much lower upfront investment and faster iteration.
Dimension 5: Latency Requirements
# Typical latency breakdown
# RAG pipeline:
rag_latency = {
"query_embedding": "50ms", # Embed the query
"vector_search": "20-100ms", # Search vector DB
"reranking": "100-300ms", # Optional cross-encoder
"llm_generation": "500-2000ms", # Generate with context
"total": "670-2450ms"
}
# Fine-tuned model (no retrieval):
finetune_latency = {
"llm_generation": "300-1500ms", # Direct generation, shorter prompts
"total": "300-1500ms"
}If you need sub-second responses, fine-tuning wins. If 1-3 seconds is acceptable (most knowledge-base and support use cases), RAG is fine.
The Decision Matrix
Score your project 1-5 on each dimension:
┌─────────────────────────────┬───────────┬──────────────┬─────────┐
│ Dimension │ RAG Score │ FT Score │ Your │
│ │ (lean RAG │ (lean FT │ Score │
│ │ if high) │ if high) │ │
├─────────────────────────────┼───────────┼──────────────┼─────────┤
│ Data changes frequently │ 5 │ 1 │ ? │
│ Need source citations │ 5 │ 1 │ ? │
│ Style/tone consistency │ 1 │ 5 │ ? │
│ Latency critical (<500ms) │ 1 │ 5 │ ? │
│ Knowledge base is massive │ 5 │ 2 │ ? │
│ Budget is limited │ 3 │ 4 │ ? │
│ Team has ML expertise │ 3 │ 2 │ ? │
│ Need to ship in < 2 weeks │ 5 │ 2 │ ? │
└─────────────────────────────┴───────────┴──────────────┴─────────┘
Total RAG score > Total FT score → Use RAG
Total FT score > Total RAG score → Use Fine-Tuning
Scores are close → Use Hybrid approachThe Hybrid Approach: Best of Both Worlds
In practice, the most effective production systems combine both. Here's the pattern:
class HybridAISystem:
"""Fine-tuned model + RAG retrieval for maximum quality."""
def __init__(self, finetuned_model: str, vector_store, embedder):
self.client = OpenAI()
self.model = finetuned_model # e.g., "ft:gpt-4o-mini:acme:support:abc123"
self.vector_store = vector_store
self.embedder = embedder
def answer(self, query: str) -> dict:
# Step 1: Classify if retrieval is needed
needs_retrieval = self._needs_retrieval(query)
# Step 2: Optionally retrieve context
context = ""
sources = []
if needs_retrieval:
query_embedding = self.embedder.embed_query(query)
results = self.vector_store.search(query_embedding, top_k=3)
context = "\n\n".join([r[0] for r in results])
sources = [r[2] for r in results]
# Step 3: Generate with fine-tuned model (knows your style/format)
# + retrieved context (has the latest facts)
messages = [
{"role": "system", "content": self._build_system_prompt(context)}
]
messages.append({"role": "user", "content": query})
response = self.client.chat.completions.create(
model=self.model,
messages=messages,
temperature=0.2
)
return {
"answer": response.choices[0].message.content,
"used_retrieval": needs_retrieval,
"sources": sources
}
def _needs_retrieval(self, query: str) -> bool:
"""Simple classifier: does this query need external knowledge?"""
# Queries about current data, specific facts -> retrieve
# Queries about process, format, general advice -> fine-tuned knowledge
response = self.client.chat.completions.create(
model="gpt-4o-mini",
messages=[{
"role": "user",
"content": f"""Does this query require looking up specific, factual information
from a knowledge base? Or can it be answered from general domain expertise?
Query: {query}
Return JSON: {{"needs_retrieval": true/false, "reason": "brief explanation"}}"""
}],
response_format={"type": "json_object"},
temperature=0
)
import json
data = json.loads(response.choices[0].message.content)
return data.get("needs_retrieval", True)
def _build_system_prompt(self, context: str) -> str:
base = "You are an Acme Corp support specialist. Always be helpful, precise, and professional."
if context:
base += f"\n\nRelevant documentation:\n{context}"
return baseWhy hybrid works: The fine-tuned model already knows your domain's vocabulary, tone, and format patterns. RAG provides it with fresh, specific facts. You get consistency AND accuracy.
Real-World Case Studies
Case Study 1: Customer Support Bot
Context: E-commerce company, 50K support tickets/month, 200-page help center, policies change monthly.
Decision: RAG (with fine-tuning for tone)
# Why RAG won:
reasoning = {
"data_freshness": "Policies, prices, and procedures change monthly",
"citation_need": "Agents need to link customers to specific help articles",
"volume": "200 pages is well within RAG's sweet spot",
"time_to_ship": "RAG pipeline built in 1 week vs 4+ weeks for fine-tuning data prep"
}
# But they also fine-tuned for tone:
tone_training = {
"purpose": "Teach the model their specific support voice",
"examples": 500, # Curated examples of ideal support responses
"result": "Model naturally writes in their brand voice without prompting"
}Case Study 2: Legal Document Drafting
Context: Law firm, need to generate contract clauses in specific legal style, references to 10,000+ case precedents.
Decision: Hybrid (fine-tuning for legal format + RAG for precedent retrieval)
reasoning = {
"why_finetune": "Legal writing has very specific format and citation style requirements",
"why_rag": "Need to reference specific case law that changes as new cases are decided",
"training_data": "2,000 examples of properly formatted legal clauses",
"rag_corpus": "10,000+ case summaries, updated weekly",
"result": "Drafts in perfect legal format with relevant, current precedent citations"
}Case Study 3: Code Review Assistant
Context: Tech company, internal coding standards, 500+ internal libraries.
Decision: RAG only
reasoning = {
"why_rag_only": "Internal standards and library APIs change with every release",
"no_finetune_because": "GPT-4o already writes excellent code reviews — just needs context",
"rag_corpus": "Internal style guides, library docs, PR review history",
"update_frequency": "Re-index on every merge to main branch",
"result": "Reviews reference current internal standards, not outdated patterns"
}Implementation Checklist
Starting with RAG (recommended for most teams):
□ Define your document sources and update frequency
□ Choose a chunking strategy (start with recursive, 512 tokens)
□ Select an embedding model (text-embedding-3-large for quality, MiniLM for cost)
□ Set up a vector store (pgvector if you have Postgres, Pinecone if not)
□ Implement hybrid search (vector + keyword)
□ Add a reranking step
□ Build an evaluation set (50+ question/answer pairs)
□ Measure retrieval recall and answer quality
□ Set up monitoring and logging
□ Implement incremental indexing for document updatesAdding fine-tuning (when you need it):
□ Collect 200+ high-quality input/output examples
□ Ensure examples cover edge cases, not just happy paths
□ Validate data quality (have domain experts review samples)
□ Start with gpt-4o-mini (cheapest, fastest to iterate)
□ Run 3 epochs, evaluate on held-out test set
□ Compare fine-tuned model vs base model + prompt engineering
□ Only deploy if fine-tuning meaningfully beats the baseline
□ Set up a retraining schedule (monthly minimum)
□ Monitor for model drift over timeCommon Mistakes to Avoid
1. Fine-tuning when you should RAG. If your main goal is adding factual knowledge, RAG is almost always better. Fine-tuning for facts leads to hallucination when the model "remembers" training data imprecisely.
2. RAG when you should fine-tune. If you're spending 50% of your prompt on format instructions and style examples, fine-tuning that into the model will save tokens, reduce latency, and improve consistency.
3. Over-engineering the first version. Start with RAG, measure quality, and only add fine-tuning when you hit specific quality bottlenecks. Many teams add fine-tuning "just in case" and waste weeks.
4. Neglecting evaluation. Without a test set and metrics, you're guessing. Build evaluation into your pipeline from day one, regardless of which approach you choose.
5. Ignoring cost at scale. A $0.005 difference per query is $1,500/month at 10K queries/day. Model your costs at production scale before committing to an architecture.
The Bottom Line
Here's the simplest version of our framework:
- Need up-to-date facts? → RAG
- Need consistent style/behavior? → Fine-tuning
- Need both? → Hybrid
- Not sure? → Start with RAG (it's faster to build, easier to iterate, and you can always add fine-tuning later)
The best AI teams don't pick one approach dogmatically. They understand the tradeoffs, measure what matters, and combine techniques where it makes sense.
Master Both at TrainIQ
Choosing between fine-tuning and RAG is just the beginning. At TrainIQ, we dedicate entire modules to each approach — Module 4 covers fine-tuning (LoRA, QLoRA, RLHF, DPO) and Module 5 covers RAG systems end-to-end. But the real value comes from learning them together, understanding the tradeoffs hands-on, and building production systems that combine both.
Our curriculum is designed for exactly this kind of architecture decision-making — not just "how to use the tools" but "when and why to use each one."
**See the full curriculum and enroll →**
The teams that ship the best AI products aren't the ones with the fanciest models. They're the ones who make the right architecture decisions. That's what TrainIQ teaches.