Scaling Laws & Emergent Abilities

Core Insight:

In 2020, researchers at OpenAI discovered that LLM performance follows predictable power laws. If you plot loss (how wrong the model is) against compute, data size, or parameter count on a log-log scale, you get straight lines. This means we can predict how good a model will be before training it.

The Three Variables:

• N (Parameters): Number of weights in the model (e.g., 7B, 70B, 405B)
• D (Dataset size): Number of tokens used for training (e.g., 2T tokens)
• C (Compute): Total FLOPs used for training (e.g., 10²⁴ FLOPs)

Loss ≈ (N/N₀)^(-αN) + (D/D₀)^(-αD) + irreducible loss

1. Kaplan et al. (2020) - OpenAI Scaling Laws:

• Performance depends most strongly on scale (compute, dataset, parameters)
• Larger models are more sample efficient - they learn more per token
• Architecture details (depth vs width) matter less than scale
• There are diminishing returns but they diminish slowly

2. Chinchilla Paper (Hoffmann et al., 2022):

• OpenAI's original analysis was wrong about the optimal ratio
• Chinchilla showed: for a given compute budget, you should train a smaller model on more data
• Optimal ratio: ~20 tokens per parameter (not 1-2 as OpenAI suggested)
• A 70B model trained on 1.4T tokens (Chinchilla) beats a 175B model trained on 300B tokens (GPT-3)
• This paper changed the industry - LLaMA, Mistral, Gemma all follow Chinchilla-optimal training

What Are Emergent Abilities?

Abilities that appear suddenly and unpredictably when models cross certain scale thresholds. The model can't do it at 10B parameters, barely does it at 50B, and does it well at 100B+.

• Chain-of-thought reasoning: Only works well at ~100B+ parameters
• Few-shot arithmetic: Near-zero at small scale, jumps to high accuracy at large scale
• Word unscrambling: Appears around 70B parameters
• Multi-step logical reasoning: Requires very large models

Debate: Are Emergent Abilities Real?

A 2023 paper by Schaeffer et al. argued that emergent abilities may be a mirage of the metrics - if you use continuous metrics instead of discrete accuracy, the "sudden jump" disappears and performance improves smoothly. This is still actively debated.

Why This Matters for You:

• Model selection: Know that a 70B model isn't just "10x bigger" than 7B - it can do qualitatively different things
• Cost planning: Scaling laws help predict training costs. 10x more compute ≈ 20% better performance
• Fine-tuning decisions: A well fine-tuned 7B model can beat a general 70B model on specific tasks
• Agent design: Complex multi-step reasoning (agents) needs large enough base models

The Compute Frontier (2024-2026):

• GPT-4: estimated ~10²⁵ FLOPs, ~$100M training cost
• Claude 3.5/4: similar or larger compute budgets
• Llama 3.1 405B: trained on 15T tokens - massively Chinchilla-optimal
• Next frontier: 10²⁶-10²⁷ FLOPs, $1B+ training runs

1. Q: What are scaling laws in LLMs?
   A: Empirical power-law relationships showing that model performance (loss) improves predictably with more compute, data, and parameters. Discovered by Kaplan et al. at OpenAI in 2020.
2. Q: What did the Chinchilla paper change?
   A: It showed the optimal compute allocation favors training smaller models on more data (~20 tokens per parameter), rather than making models as large as possible. This influenced all subsequent model training decisions.
3. Q: What are emergent abilities?
   A: Capabilities that appear suddenly at certain scale thresholds - like chain-of-thought reasoning only working well at 100B+ parameters. Though the definition is debated.
4. Q: How do scaling laws help in production?
   A: They help predict training costs, choose model sizes for tasks, and understand when to use larger vs smaller models for specific capabilities.