Large Language Model Engineering Guide — Internal Reference
LARGE LANGUAGE MODEL
ENGINEERING GUIDE
From Prompt Engineering to Production Deployment
AI Engineering Team | v4.1 | May 2025
Field Value
Audience ML Engineers, Prompt Engineers, Applied AI Teams
Scope LLM selection, integration, fine-tuning, evaluation, deployment
Document Type Technical Reference Guide
Prerequisite Knowledge Python, REST APIs, basic ML concepts
Version 4.1 (Breaking changes from v3.x noted inline)
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
1. Introduction to Large Language Models
Large Language Models (LLMs) are a class of generative AI system trained on massive text
corpora using transformer-based architectures. They exhibit emergent capabilities — including
reasoning, summarization, translation, code generation, and question answering — that arise
from scale rather than explicit programming.
This guide is a practical engineering reference for teams working with LLMs in a production
context. It covers the full stack from model selection and prompt design through fine-tuning,
evaluation, deployment, and monitoring. It is intentionally opinionated: where choices exist, we
recommend specific approaches based on our internal experience.
1.1 How LLMs Work
LLMs are trained using a self-supervised objective: given a sequence of tokens, predict the next
token. Through exposure to vast amounts of text, the model learns rich representations of
language, world knowledge, and reasoning patterns. At inference time, the model generates text
autoregressively — one token at a time — conditioned on the input prompt and all previously
generated tokens.
The transformer architecture underpinning modern LLMs uses a self-attention mechanism that
allows each token to attend to all other tokens in the context window. This enables long-range
dependency modeling and is responsible for much of the coherence and contextual sensitivity in
LLM outputs.
1.2 Model Taxonomy
Category Examples Strengths Typical Use Cases
Instruction-tuned GPT-4o, Claude 3.5, Following complex Chatbots, assistants, analysis
Gemini 1.5 instructions
Reasoning models o1, o3, DeepSeek-R1 Multi-step logical Math, coding, complex QA
reasoning
Code specialists Codestral, Code generation, Developer tools, IDEs
DeepSeek-Coder debugging
Embedding models text-embedding-3, Semantic Search, RAG, classification
BGE-M3 representation
Multimodal GPT-4o, Claude 3.5, Vision + language Document AI, image analysis
Gemini tasks
Open weights Llama 3, Mistral, On-premises, Privacy-sensitive, fine-tuning
Qwen2.5 customization
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
2. Prompt Engineering
Prompt engineering is the practice of designing, structuring, and iterating on the text inputs to
LLMs to reliably elicit desired outputs. It is both a craft and an empirical discipline — good
prompts are developed through systematic experimentation, not intuition alone.
2.1 Core Prompting Techniques
Zero-Shot Prompting
Zero-shot prompting provides the model with a task description and an input, without any
examples. This is the starting point for most use cases. Zero-shot works well for common tasks
that are well-represented in training data, such as classification, summarization, and translation.
Classify the following customer message as: COMPLAINT, INQUIRY, or
COMPLIMENT.
Message: 'Your support team resolved my issue in under 5 minutes.
Outstanding!'
Classification:
Few-Shot Prompting
Few-shot prompting provides between 2 and 10 demonstrations (examples) before the target
input. This is one of the most reliable techniques for improving output quality, particularly for
non-standard output formats or domain-specific classification tasks.
Key considerations for effective few-shot prompting: examples should be diverse and
representative, the order of examples can affect outputs, and the format of examples must
exactly match the desired output format.
Chain-of-Thought (CoT) Prompting
Chain-of-thought prompting encourages the model to reason step-by-step before producing a
final answer. This technique significantly improves performance on tasks requiring multi-step
reasoning, mathematics, and logical deduction. The simplest form adds 'Think step by step' to
the prompt. More structured forms provide a reasoning template.
Solve the following problem. Show your work step by step before giving the
final answer.
Problem: A factory produces 240 units per hour for 6 hours. 15% are
defective.
How many non-defective units are produced?
Step 1: Calculate total units produced...
System Prompts
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
System prompts define the model's role, persona, constraints, and output format at the
conversation level. They are processed before user messages and establish persistent context
across a conversation. Well-designed system prompts are the single highest-leverage
intervention for production LLM systems.
• Be explicit: state what the model should and should not do
• Define the output format in detail, including schema if JSON is expected
• Specify the persona, expertise level, and communication style
• Include safety guardrails appropriate to the application
• Test system prompt changes rigorously — small changes can have large effects
2.2 Prompt Design Principles
Principle Description Common Failure Mode
Clarity Use unambiguous, precise language Vague instructions produce
inconsistent outputs
Specificity Provide concrete constraints and Over-general prompts produce
examples over-general outputs
Format Describe exact output format desired Model uses unexpected structure
specification
Context grounding Provide relevant context before the task Model hallucinates missing facts
Role definition Assign a clear expert role to the model Model responds too broadly or too
conversationally
Negative Specify what NOT to do Model produces content you want
constraints to exclude
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
3. Retrieval-Augmented Generation (RAG)
RAG is a technique that combines LLM generation with dynamic retrieval of relevant information
from a knowledge base. It addresses the core limitations of standalone LLMs: knowledge
cutoffs, hallucination on factual queries, and inability to reason over private or proprietary data.
3.1 RAG Architecture
A standard RAG pipeline consists of: (1) an ingestion phase that processes documents into a
vector index, and (2) a retrieval-augmented inference phase that retrieves relevant chunks and
injects them into the prompt at query time.
Ingestion Pipeline
1. Document loading: ingest from PDFs, HTML, databases, APIs, or file stores
2. Chunking: split documents into segments (typically 256–1024 tokens) with controlled
overlap
3. Embedding: encode each chunk using an embedding model to produce dense vector
representations
4. Indexing: store embeddings in a vector database (Pinecone, Weaviate, Qdrant,
pgvector)
5. Metadata tagging: attach source, date, access level, and other metadata to each chunk
Query Pipeline
6. Embed the user query using the same embedding model
7. Retrieve the top-k most semantically similar chunks using approximate nearest neighbor
search
8. Re-rank retrieved chunks using a cross-encoder for precision improvement (optional but
recommended)
9. Construct the augmented prompt: system context + retrieved chunks + user query
10. Generate the response with source attribution
3.2 RAG Best Practices
• Use a dedicated embedding model — do not use the generation model for embedding
• Implement hybrid search combining dense vector search with BM25 keyword search
• Add query expansion or HyDE (Hypothetical Document Embedding) to improve retrieval
recall
• Implement chunk overlap (10–20% of chunk size) to preserve boundary context
• Regularly re-index knowledge base as documents are updated
• Monitor retrieval quality with hit rate and MRR metrics, not just generation quality
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
3.3 Advanced RAG Patterns
Pattern Description When to Use
HyDE Generate a hypothetical answer, When query-document mismatch is high
embed it, use for retrieval
Multi-query Rephrase the query multiple ways, Complex or ambiguous queries
retrieval merge retrieval results
Self-RAG Model decides when to retrieve vs. Mixed knowledge-intensive tasks
generate from memory
RAPTOR Recursive summarization creates Long document corpuses
hierarchical index
Graph RAG Knowledge graph combined with Highly relational knowledge domains
vector index
Agentic RAG Model iteratively retrieves until Research-style, multi-step tasks
sufficient context gathered
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
4. Fine-Tuning
Fine-tuning adapts a pre-trained base model to a specific task or domain using a relatively small
supervised dataset. It is appropriate when prompt engineering and RAG cannot achieve the
required performance, when a specialized style or format is needed, or when latency/cost
constraints require a smaller custom model over a large frontier model.
4.1 Fine-Tuning Decision Framework
Scenario Recommended Approach
Task is common, model is already capable Prompt engineering only
Need to inject private/proprietary knowledge RAG first, fine-tune if recall insufficient
Need custom output format or style consistently Fine-tuning (supervised)
Need to improve reasoning on domain-specific Fine-tuning with chain-of-thought data
tasks
Need the smallest possible model for Fine-tune a smaller open model
cost/latency
Need to align behavior with custom RLHF or DPO fine-tuning
policies/values
4.2 Supervised Fine-Tuning (SFT)
SFT trains the model on a dataset of (input, output) pairs. The model is trained to maximize the
likelihood of the target output given the input. Dataset quality is far more important than dataset
size — 500 high-quality, diverse examples typically outperform 10,000 noisy ones.
Data Preparation Guidelines
• Minimum recommended dataset size: 200 examples for format/style, 1,000+ for domain
knowledge
• Include diverse examples covering the full range of expected inputs
• Ensure examples demonstrate the exact output format required in production
• Decontaminate training data: remove examples that appear in your evaluation set
• Use human review for at least 10% of training examples to catch quality issues
4.3 Parameter-Efficient Fine-Tuning (PEFT)
Full fine-tuning updates all model parameters and is compute-intensive. For most applications,
parameter-efficient methods provide comparable performance at a fraction of the cost. LoRA
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
(Low-Rank Adaptation) is the industry-standard PEFT method and our recommended starting
point.
from peft import get_peft_model, LoraConfig, TaskType
lora_config = LoraConfig(
r=16, # LoRA rank
lora_alpha=32, # Scaling factor
target_modules=['q_proj', 'v_proj'],
lora_dropout=0.1,
task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(base_model, lora_config)
4.4 Reinforcement Learning from Human Feedback (RLHF)
RLHF trains a reward model on human preference data, then uses reinforcement learning to
optimize the language model to maximize the reward signal. It is particularly effective for
alignment tasks: improving helpfulness, reducing harmful outputs, and enforcing policy
compliance. DPO (Direct Preference Optimization) has emerged as a simpler and often equally
effective alternative.
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
5. Evaluation
Rigorous evaluation is the foundation of reliable LLM engineering. The gap between perceived
and actual model performance is one of the most common sources of production failure.
Evaluation must be conducted before deployment, after every significant model or prompt
change, and continuously in production.
5.1 Evaluation Taxonomy
Evaluation Type Method Tools Cadence
Accuracy / Exact match, F1, LM-Eval, HELM, Per model/prompt change
Correctness ROUGE, BERTScore custom harness
Instruction following Human eval, model- GPT-4o judge, Per major change
as-judge human reviewers
Hallucination rate Fact-checking against FactScore, custom Per model change
ground truth pipelines
Safety / Refusal Red team probes, Internal red team, Per deployment
harmful prompt sets Promptfoo
Latency / Cost P50/P95/P99 latency, LangSmith, internal Continuous
token cost monitors
Drift detection Distribution shift in Evidently, custom Daily in production
outputs monitors
5.2 Building an Evaluation Dataset
A strong evaluation dataset is the most valuable investment in LLM engineering. It should be
curated by domain experts, cover both typical and edge-case inputs, and be kept strictly
separate from training data. Aim for a minimum of 200 examples for initial evaluation and
expand over time as production traffic reveals new failure modes.
• Curate golden examples from real or realistic production traffic
• Include adversarial examples: inputs designed to elicit failures
• Stratify by difficulty, input length, and task type
• Version-control your evaluation dataset alongside your models
• Never use evaluation data for training or prompt development
5.3 LLM-as-Judge
Using a powerful LLM (such as GPT-4o or Claude 3.5 Sonnet) to evaluate outputs from another
model is a scalable alternative to human evaluation for many dimensions. LLM judges correlate
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
well with human judgments for dimensions like coherence, relevance, and instruction following.
They are less reliable for factual accuracy and domain-specific correctness.
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
6. Production Deployment
6.1 Deployment Patterns
Pattern Description Pros Cons
Managed API Use frontier model via API Fastest to deploy, Cost at scale, data privacy
(OpenAI, Anthropic, Google) no infra
Self-hosted open Deploy open-weights model Privacy, cost at Infra complexity, team
model on own infra scale burden
Model gateway Route requests to multiple Fallback, A/B Added latency, complexity
backends testing
Sidecar proxy LLM proxy in each service Observability, Deployment overhead
caching
6.2 Inference Optimization
LLM inference is compute- and memory-intensive. The following optimizations are
recommended for production systems where latency or cost is a concern:
• Quantization: use INT8 or INT4 quantization to reduce memory footprint and increase
throughput (4-8x speedup with minimal quality loss for most tasks)
• KV cache optimization: maximize KV cache reuse, especially for shared system prompts
• Continuous batching: use vLLM or TensorRT-LLM for dynamic batching of inference
requests
• Speculative decoding: use a small draft model to generate candidate tokens verified by
the main model
• Prompt caching: cache system prompt representations for endpoints with static context
6.3 Observability
Production LLM systems require observability beyond standard software metrics. In addition to
latency, error rate, and throughput, you must monitor LLM-specific signals:
• Token usage and cost per request (track separately for input and output tokens)
• Output length distribution (unexpected length changes signal prompt drift)
• Model-as-judge quality scores on sampled production traffic
• User feedback signals (thumbs up/down, corrections, escalations)
• Latency by model, prompt version, and input length
• Refusal rate (unexpected changes indicate prompt injection or adversarial usage)
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
6.4 Prompt Versioning
Prompt changes are code changes and must be managed accordingly. All prompts must be
version-controlled, reviewed before deployment, and associated with evaluation results that
justify the change. Prompt rollbacks must be possible within 15 minutes for production systems.
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
7. Security Considerations
7.1 Prompt Injection
Prompt injection is the most prevalent LLM-specific security vulnerability. An attacker provides
inputs designed to override the system prompt or hijack the model's behavior. Direct prompt
injection exploits the user input channel; indirect injection embeds malicious instructions in data
retrieved at inference time (e.g., via RAG).
Mitigation Strategies
• Strict input validation and sanitization for all user-controlled fields
• Use delimiters to clearly separate system content from user content
• Apply a secondary classifier to detect injection attempts before the main model
processes them
• Restrict model capabilities to the minimum required for the use case
• Never execute code or perform privileged actions based on unvalidated LLM output
7.2 Data Exfiltration
LLMs with tool-calling capabilities can be manipulated to exfiltrate sensitive data through crafted
outputs. Defense-in-depth is required: validate and sanitize all model outputs before acting on
them, apply output length limits where feasible, and audit tool-calling patterns in production.
7.3 Model Supply Chain Security
Open-weights models downloaded from public repositories may contain backdoors, malicious
behaviors, or training data that violates licenses. Before deploying any third-party model, verify
the model hash against the official repository, review the model card and training data
disclosure, and conduct internal red team evaluation.
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
8. Cost Management
LLM costs at production scale can be substantial. Cost optimization should be addressed at
architecture design time, not as an afterthought.
Technique Potential Implementation Notes
Savings Effort
Model right-sizing 50–80% Medium Use smaller model where
performance allows
Prompt compression 20–40% Medium Remove redundant tokens from
prompts
Output caching 30–70% Low Cache identical or semantically
similar queries
Batching 20–50% Medium Batch low-urgency requests for
throughput pricing
Streaming Perceived Low Stream tokens to improve UX without
latency reducing cost
Quantization 40–75% (self- Medium-High Significant for on-premises
hosted) deployments
Prompt caching 70–90% (cached Low Available from Anthropic, OpenAI,
portion) Google
8.1 Cost Attribution
All LLM API costs must be attributed to the consuming team and use case. Implement token-
level cost tracking in your observability stack, with dashboards showing cost per user, per
feature, and per model. Set up budget alerts at 70% and 90% of monthly allocation to allow
corrective action before overruns occur.
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
9. Appendix: Quick Reference
9.1 Model Selection Checklist
11. Define quality requirements: what level of accuracy/coherence is needed?
12. Define latency requirements: what is the acceptable p95 response time?
13. Define cost budget: what is the maximum cost per 1,000 requests?
14. Assess data privacy: can data be sent to a third-party API?
15. Benchmark at least 3 candidate models on your specific task
16. Evaluate safety and refusal behavior with adversarial prompts
17. Document model selection rationale in the Model Card
9.2 Pre-Deployment Checklist
18. Evaluation dataset created and evaluation results documented
19. Prompt versioned and stored in version control
20. Safety testing completed (adversarial prompts, injection testing)
21. Observability instrumented (latency, cost, quality, refusal rate)
22. Cost attribution configured
23. Kill switch and rollback procedure documented and tested
24. Model Card completed and approved by AI System Owner
9.3 Recommended Libraries and Tools
Category Tool Use Case
LLM Orchestration LangChain, LlamaIndex RAG pipelines, agentic workflows
Evaluation RAGAS, LM-Eval, RAG eval, benchmarks, regression testing
Promptfoo
Observability LangSmith, Arize, Tracing, monitoring, cost tracking
Helicone
Vector Database Qdrant, Weaviate, Semantic search, RAG retrieval
pgvector
Inference Server vLLM, TensorRT-LLM High-throughput self-hosted inference
Fine-tuning Axolotl, TRL, Unsloth SFT, LoRA, DPO training
Prompt Langfuse, PromptLayer Prompt versioning and A/B testing
Management
AI Engineering Team | Page
Large Language Model Engineering Guide — Internal Reference
— End of Document —
AI Engineering Team | Page