Skip to content
InfratoApp

Stanford University · Course

CS336: Language Modeling from Scratch

Stanford CS336, Language Modeling from Scratch, is an implementation-heavy course in which you build the pieces of a Transformer language model yourself. The page we read is the archived Spring 2025 offering.

Open at Stanford University ↗Common questions

Who it suits

  • Engineers and researchers who already know machine learning and want to understand language models by building one.
  • Not for beginners: the course page lists proficiency in Python, calculus and linear algebra, probability, and basics of machine learning and deep learning.

What it covers

  • Basics: a tokenizer, model architecture and optimiser to train a standard Transformer language model.
  • Systems: profiling, a Triton implementation of FlashAttention2, and memory-efficient distributed training.
  • Scaling: fitting a scaling law to project model scaling.
  • Data: turning raw Common Crawl dumps into pretraining data, with filtering and deduplication.
  • Alignment and reasoning RL: supervised fine-tuning and reinforcement learning on math problems.

Before you start

  • Proficiency in Python.
  • College calculus and linear algebra.
  • Basic probability and statistics.
  • Comfort with the basics of machine learning and deep learning.

Common questions about CS336: Language Modeling from Scratch

How heavy is the workload?
The page says it is a 5-unit class and a very implementation-heavy one, so allocate enough time.
Which offering is this?
The archived Spring 2025 website. The page points to a newer offering for the latest version.
What will I build?
Five assignments: basics, systems, scaling, data, and alignment and reasoning RL, per the course page.

More like this

Keep learning on Infrato

Where this comes from

Written from Stanford University’s own pages, read on 2026-10-13. Content, prices and availability change, so the provider’s page is the source. We are not affiliated with Stanford University and this is not an endorsement.