Skip to content

About

Self-study notes and labs for Stanford CS336: language modeling, transformers, GRPO, BPE

Topics

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

68 Commits

Folders and files

Repository files navigation

Stanford CS336 — Language Modeling from Scratch

Self-study implementations of Stanford CS336 (Spring 2026): build a language model from tokenizer through alignment, with staff tests as the quality bar.

This is course work, not a shipping product. Implementations live in this repo; large datasets, checkpoints, and GPU runs stay local.

Author: Mangesh Raut · Not enrolled for credit — no Gradescope or leaderboard submissions.

Course · Lectures (2026) · Staff assignments · Layout · Purpose

What’s inside

Five standalone uv projects under assignments/:

# Topic In this repo Tests (local)
1 — Basics Byte-level BPE, GPT-style transformer, AdamW, TinyStories / OWT training Val loss TS 2.07, OWT 5.38 47 pass, 1 xfail
2 — Systems Flash attention, DDP, FSDP, sharded AdamW CPU / gloo path on Mac 10 pass, 4 skip
3 — Scaling IsoFLOPs, Chinchilla-style fits Predicted L ≈ 7.17; curves below 7/7 + artifacts
4 — Data Extract, LID, PII, classifiers, dedup Required modules + tests 21/21
5 — Alignment GRPO, SFT packing, DPO, eval parsers Required modules + tests 26/26
bash scripts/verify_all.sh

Each assignment is verified from its own directory (uv run pytest -q and/or bash scripts/finalize_assignment.sh). See docs/LAYOUT.md.

Assignment 3 artifact (tracked)

IsoFLOPs curves from the scaling-law assignment (SVG in git, not a product screenshot):

IsoFLOPs: final loss vs parameters

Related local files (open on disk; this repo does not use GitHub Pages):

Docs

File Role
docs/PURPOSE.md Why this monorepo exists (self-study scope)
docs/LAYOUT.md Folder layout, verify commands, local-only paths
assignments/README.md Per-assignment index
Per-assignment PROJECT_STATUS.md, SOLUTION.md, LEARNINGS.md, writeup.md Status, implementation notes, writeups

Handouts (*.pdf) stay next to each assignment, as in the staff repos.

What belongs on GitHub

In git Local only
Source, tests, scripts, small artifacts Token .bin memmaps, checkpoints
Docs, scaling fits, run summaries local-shared-data/ (~2GB classifiers for A4)
A5 staff eval fixtures (~85MB) .venv/, Gradescope zips

Scope (honest)

All local unit tests listed above pass. Not included: Gradescope submissions, Modal GPU training (A4 full WET pipeline, A5 GSM8K GRPO), CUDA/B200 leaderboard runs, or the Stanford-hosted Hyperturing API (enrolled students only). The A3 dashboard HTML is a local example, not a deployed app.

License and citation

Original portfolio documentation in this repository is MIT (© 2026 Mangesh Raut). Staff assignment scaffolding under assignments/ remains MIT Copyright Stanford University as in each assignment LICENSE. See CITATION.cff if you cite this portfolio.

Course links

About

Self-study notes and labs for Stanford CS336: language modeling, transformers, GRPO, BPE

Topics

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Sponsor this project

Packages

Used by

Contributors

Languages