LARGE LANGUAGE MODEL (LLM) TRAINING
TUTORIAL
A Complete End-to-End Technical Guide: Tokenization, Pre-Training, SFT & LoRA
This guide walks through the three core pillars of modern Large Language Model engineering: Data
Tokenization, Pre-Training from scratch using Causal Language Modeling, and Instruction Fine-Tuning via
Low-Rank Adaptation (LoRA).
1. Environment & Prerequisites Setup
To begin training LLMs, ensure python 3.10+ and standard Hugging Face/PyTorch deep learning libraries
are installed:
pip install torch transformers datasets tokenizers peft trl accelerate bitsandbytes
📌 HARDWARE REQUIREMENTS: Hardware Note: Pre-training requires CUDA-enabled GPU
acceleration. SFT/LoRA can run on single consumer GPUs (e.g., RTX 3090/4090 or T4 GPUs).
2. Tokenization & Data Pipeline
LLMs do not process raw strings directly. Text is converted into discrete integer tokens using subword
algorithms such as Byte-Pair Encoding (BPE). The code below demonstrates loading a BPE tokenizer and
formatting input tensors.
from transformers import AutoTokenizer
# Load a standard BPE tokenizer (e.g., GPT-2 or Llama)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token
text = "LLMs predict the next token based on context."
tokens = tokenizer(text, padding="max_length", max_length=16, truncation=True,
return_tensors="pt")
print("Token IDs:", tokens["input_ids"])
print("Decoded:", [Link](tokens["input_ids"][0]))
3. Pre-Training from Scratch (Decoder-Only)
During pre-training, an uninitialized transformer processes massive text datasets and learns general
linguistic structures by predicting the next token in a sequence (Causal Language Modeling).
Step 3.1: Define Architecture Configuration
from transformers import GPT2Config, GPT2LMHeadModel
config = GPT2Config(
vocab_size=len(tokenizer),
n_positions=512, # Maximum sequence window
n_embd=256, # Embedding dimension
n_layer=4, # Transformer layer blocks
n_head=4 # Multi-head attention heads
)
model = GPT2LMHeadModel(config)
print(f"Model Parameters: {sum([Link]() for p in [Link]()) / 1e6:.2f}M")
Step 3.2: Prepare Dataset & DataLoader
from datasets import load_dataset
from transformers import DataCollatorForLanguageModeling
from [Link] import DataLoader
raw_dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
def tokenize_function(examples):
return tokenizer(examples["text"], truncation=True, max_length=512)
tokenized_dataset = raw_dataset.map(tokenize_function, batched=True, remove_columns=["text"])
tokenized_dataset = tokenized_dataset.filter(lambda x: len(x["input_ids"]) > 0)
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
train_dataloader = DataLoader(tokenized_dataset, batch_size=4, shuffle=True,
collate_fn=data_collator)
Step 3.3: Execute Causal Training Loop
import torch
from [Link] import AdamW
device = [Link]("cuda" if [Link].is_available() else "cpu")
[Link](device)
optimizer = AdamW([Link](), lr=5e-4)
[Link]()
for epoch in range(1):
for step, batch in enumerate(train_dataloader):
batch = {k: [Link](device) for k, v in [Link]()}
outputs = model(**batch)
loss = [Link]
[Link]()
[Link]()
optimizer.zero_grad()
if step % 50 == 0:
print(f"Step {step} | Loss: {[Link]():.4f}")
4. Supervised Fine-Tuning (SFT) using LoRA
Pre-trained base models excel at text completion but fail at structured conversation. Supervised Fine-
Tuning aligns the model using prompt-response pairs. To optimize efficiency, we apply Low-Rank
Adaptation (LoRA) to freeze main parameters and train small rank matrices.
from transformers import AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
from datasets import Dataset
# 1. Load Pre-trained Base Model
base_model_name = "gpt2"
model = AutoModelForCausalLM.from_pretrained(base_model_name)
# 2. Configure LoRA Parameters
lora_config = LoraConfig(
r=8, # Rank dimension
lora_alpha=16, # Scaling factor
target_modules=["c_attn"], # Target attention projections
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# 3. Instruction Dataset Formatting
formatted_data = {
"text": [
"### Instruction:
Explain gravity in simple terms.
### Response:
Gravity is a force that pulls objects toward each other.",
"### Instruction:
What is Python?
### Response:
Python is a high-level programming language known for readable syntax."
]
}
sft_dataset = Dataset.from_dict(formatted_data)
# 4. Execute SFT Trainer
training_args = TrainingArguments(
output_dir="./lora_llm_output",
per_device_train_batch_size=2,
learning_rate=2e-4,
num_train_epochs=3,
logging_steps=1,
fp16=[Link].is_available()
)
trainer = SFTTrainer(
model=model,
train_dataset=sft_dataset,
dataset_text_field="text",
max_seq_length=256,
args=training_args
)
[Link]()
5. Text Generation & Inference Strategy
After alignment, evaluation mode is enabled to perform inference. Sampling strategies such as
Temperature, Top-K, and Top-P (Nucleus) control diversity and accuracy.
[Link]()
prompt = "### Instruction:
What is Python?
### Response:
"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
output_tokens = [Link](
**inputs,
max_new_tokens=50,
temperature=0.7,
top_p=0.9,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
print([Link](output_tokens[0], skip_special_tokens=True))
6. Summary: Key Pipeline Stages Comparison
Stage Primary Objective Data Format Compute Complexity
Pre-Training Learn language Unstructured raw text Extremely High
representations & corpora (GPUs/TPUs clusters)
general knowledge
SFT (Fine-Tuning) Align model to follow Prompt-Response pairs Medium (1-8 GPUs)
instructions & task
formats
LoRA / PEFT Parameter-efficient Prompt-Response pairs Low (Single consumer
adaptation with GPU)
minimal weights