CSOC’26 – NLP Track Capstone Project
Paper Replication Study:
Scaling Laws for Neural Language Models
1 Introduction
Modern Natural Language Processing is driven by empirical scaling. In this capstone, you
will transition from student to researcher by undertaking a rigorous replication study of the
landmark OpenAI paper: ”Scaling Laws for Neural Language Models” (Kaplan et al., 2020).
Rather than completing a standard software engineering project, you will validate the core
hypothesis of scaling theory: that language modeling cross-entropy loss follows predictable
power-law relations with respect to model parameters (N ) and dataset size (D).
This replication study is organized into two sequential levels:
• Level 1: Architectural Implementation – Constructing a clean, decoder-only autoregres-
sive Transformer from foundational tensor operations in PyTorch.
• Level 2: Scaling Law Validation – Executing controlled training sweeps across parameter
sizes and dataset scales to empirically fit and analyze the power-law exponents αN and αD .
Core Guidelines:
• No High-Level Shortcuts: Utilizing high-level PyTorch abstractions such as [Link]
or [Link] is strictly prohibited. All attention mechanics, tensor shaping, and
masking projections must be coded from scratch.
• Strict Control of Variables: To claim a valid replication, all optimization parameters, learn-
ing rate decay profiles, and token throughput limits must be meticulously controlled.
1
2 Level 1: Foundational Architecture (Decoder-Only GPT)
Your first objective is to build a mathematically precise autoregressive (decoder-only) Trans-
former.
2.1 Mathematical Formulations to Implement
You must implement the following components as modular PyTorch layers:
1. Causal Multi-Head Attention (MHA): Given an input tensor X ∈ RB×T ×dmodel , project it to
Queries (Q), Keys (K), and Values (V ) using linear projection weights. Split into H heads of
dimension dk = dmodel /H and compute attention using causal masking:
QK T
Attention(Q, K, V ) = softmax √ +M V
dk
where the causal mask M ∈ RT ×T is defined as:
(
0 i≥j
Mij =
−∞ i < j
2. Position-wise Feed-Forward Network (FFN): A two-layer MLP mapping applied identi-
cally across sequence positions:
FFN(x) = GELU(xW1 + b1 )W2 + b2
where W1 ∈ Rdmodel ×dff and W2 ∈ Rdff ×dmodel , with dff = 4 × dmodel .
3. Pre-Layer Normalization (Pre-LN): To ensure stable gradient flow when scaling layer
depths, place Layer Normalization on the input branches:
x(k+1) = x(k) + SubLayer(LN(x(k) ))
4. Positional Encodings: Inject sequential order using fixed sinusoidal positional encodings
or learnable position embeddings.
5. Weight-Tied LM Head: An output projection mapping back to vocabulary dimension Vvocab .
To reduce parameter bloat, bind the weights of your token embedding matrix to the final
linear layer.
2.2 Progression Checklist – Level 1
Implement a custom data pipeline to pack raw tokens into batches of X, Y ∈ RB×T .
Build the causal attention head with an explicit lower-triangular mask.
Write tensor dimension verification unit tests for each intermediate layer.
Verify causal compliance: assert that gradients at sequence index t do not propagate to
inputs at indices > t.
Overfit a single batch of text to zero cross-entropy loss to verify mathematical complete-
ness.
2
3 Level 2: Replicating Neural Scaling Laws
With a verified architecture, you will now replicate the empirical experiments of Kaplan et
al. (2020). You will analyze scaling performance across two dimensions: parameters (N ) and
training token dataset size (D).
3.1 Parameter Counting Methodology
In order to focus on the expressive capacity of the core sequence model, embedding param-
eters are highly dependent on vocabulary size and must be excluded from your parameter
count variable N . For a network with L layers, hidden dimension dmodel , and dff = 4dmodel , the
non-embedding parameter count is defined as:
N ≈ L × 12 · d2model + 4 · dmodel
Implement the following programmatic validation in your training script:
def count_non_embedding_params(model):
return sum([Link]() for name, p in model.named_parameters()
if 'embed' not in name and 'lm_head' not in name)
3.2 Replication Sweep Design
To preserve compute accessibility on free-tier platforms (Google Colab, Kaggle, Lightning AI),
you will execute two distinct, low-overhead sweeps designed to complete in under 30 minutes:
1. Parameter Scaling Sweep (Finding αN ): Train three models of increasing dimensions (Ta-
ble 1) on the full dataset for exactly 3,000 steps. Track the best validation loss (L) achieved
by each model scale.
Table 1: Model Configurations for Parameter Sweep
Scale Layers (L) dmodel dff Heads (H) Context (T ) Approx. Parameters (N )
Tiny 2 64 256 2 128 ≈ 1.0 × 105
Small 4 128 512 4 256 ≈ 7.9 × 105
Medium 6 256 1024 8 256 ≈ 4.7 × 106
2. Data Scaling Sweep (Finding αD ): Keep the model size constant by utilizing the Small
model configuration. Train this model on four subset sizes of your dataset: 10%, 25%, 50%,
and 100% of total tokens. Record the minimal validation loss (L) for each data budget limit.
3.3 Dataset Options (No Custom Tokenizer Setup Required)
To eliminate the friction of building custom tokenizers, select one of the following ready-to-use
datasets:
• Tiny Shakespeare (Character-Level): A 1.1MB character corpus. Use a simple 65-character
Python dictionary lookup to map characters directly to integers. Training is extremely fast
and yields stable curves.
• TinyStories-10k (Subword-Level): Synthetically generated, grammatically simple stories.
Use the off-the-shelf pre-trained GPT-2 tokenizer directly via Hugging Face:
3
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
3.4 Fitting the Exponents
Fit power-law equations to your experimental data in log-log space:
log(L(N )) = constant − αN log(N )
log(L(D)) = constant − αD log(D)
Plot both lines of best fit to determine the empirical exponents αN and αD .
3.5 Progression Checklist – Level 2
Prepare the chosen dataset and create a standard 90/10 training/validation split.
Instantiate Tiny, Small, and Medium models, printing validation counts for non-embedding
parameters.
Run the parameter sweep, keeping optimizer hyperparameters, sequence lengths, and
step counts identical.
Run the data sweep using the Small model across the four token-budget configurations.
Transform results into logarithmic space and use [Link] to extract the slopes (αN
and αD ).
Calculate the scaling ratio: γ = αN /αD .
Generate log-log scaling plots (scaling_laws.png) containing the regression lines and raw
data points.
4 Analysis & Discussion
4.1 Comparing Exponents and the Scaling Ratio
Once you calculate your empirical parameters, analyze the scaling ratio:
αN
γ=
αD
In Kaplan et al. (2020), OpenAI reported αN ≈ 0.076 and αD ≈ 0.095, implying γ ≈ 0.80. A ratio
γ < 1.0 suggests that scaling data yields faster marginal performance gains than scaling pa-
rameters on modern architectures. Contrast your empirical ratio with OpenAI’s baseline and
discuss what your observed ratio indicates regarding token efficiency vs. parameter capacity
on your selected dataset.
4
5 Submission Guidelines
5.1 Repository Layout
Your repository must strictly follow this structure:
��� [Link] # Custom CausalAttention, FFN, and DecoderTransformer modules
��� train_scaling.py # Main training script accepting sweep configurations
��� plot_scaling.py # Exponent fitting and plot generation script
��� [Link] # Compiled research paper replication report
5.2 Research Report Requirements (Overleaf LaTeX)
Your replication report must read as a formal research paper. Document:
1. Mathematical Formalism: Define the exact formulas implemented in Level 1.
2. Replication Methodology: State dataset choices, token counts, and the optimization con-
figuration.
3. Scaling Curves: Embed your log-log validation scaling plots. Include the empirical regres-
sion equations and calculated values for αN , αD , and the ratio γ.
4. Theoretical Discussion: Address the discrepancies with Kaplan et al. using the specific
analysis criteria outlined above.
May your scaling curves be exponential, and your constants robust!