Self-study implementations of Stanford CS336 (Spring 2026): build a language model from tokenizer through alignment, with staff tests as the quality bar.
This is course work, not a shipping product. Implementations live in this repo; large datasets, checkpoints, and GPU runs stay local.
Author: Mangesh Raut · Not enrolled for credit — no Gradescope or leaderboard submissions.
Course · Lectures (2026) · Staff assignments · Layout · Purpose
Five standalone uv projects under assignments/:
| # | Topic | In this repo | Tests (local) |
|---|---|---|---|
| 1 — Basics | Byte-level BPE, GPT-style transformer, AdamW, TinyStories / OWT training | Val loss TS 2.07, OWT 5.38 | 47 pass, 1 xfail |
| 2 — Systems | Flash attention, DDP, FSDP, sharded AdamW | CPU / gloo path on Mac | 10 pass, 4 skip |
| 3 — Scaling | IsoFLOPs, Chinchilla-style fits | Predicted L ≈ 7.17; curves below | 7/7 + artifacts |
| 4 — Data | Extract, LID, PII, classifiers, dedup | Required modules + tests | 21/21 |
| 5 — Alignment | GRPO, SFT packing, DPO, eval parsers | Required modules + tests | 26/26 |
bash scripts/verify_all.shEach assignment is verified from its own directory (uv run pytest -q and/or bash scripts/finalize_assignment.sh). See docs/LAYOUT.md.
IsoFLOPs curves from the scaling-law assignment (SVG in git, not a product screenshot):
Related local files (open on disk; this repo does not use GitHub Pages):
- Scaling experiments dashboard (
examples/dashboard.html) - Training API client notebook
| File | Role |
|---|---|
| docs/PURPOSE.md | Why this monorepo exists (self-study scope) |
| docs/LAYOUT.md | Folder layout, verify commands, local-only paths |
| assignments/README.md | Per-assignment index |
Per-assignment PROJECT_STATUS.md, SOLUTION.md, LEARNINGS.md, writeup.md |
Status, implementation notes, writeups |
Handouts (*.pdf) stay next to each assignment, as in the staff repos.
| In git | Local only |
|---|---|
| Source, tests, scripts, small artifacts | Token .bin memmaps, checkpoints |
| Docs, scaling fits, run summaries | local-shared-data/ (~2GB classifiers for A4) |
| A5 staff eval fixtures (~85MB) | .venv/, Gradescope zips |
All local unit tests listed above pass. Not included: Gradescope submissions, Modal GPU training (A4 full WET pipeline, A5 GSM8K GRPO), CUDA/B200 leaderboard runs, or the Stanford-hosted Hyperturing API (enrolled students only). The A3 dashboard HTML is a local example, not a deployed app.
Original portfolio documentation in this repository is MIT (© 2026 Mangesh Raut). Staff assignment scaffolding under assignments/ remains MIT Copyright Stanford University as in each assignment LICENSE. See CITATION.cff if you cite this portfolio.
- CS336 course page — Spring 2026
- Lectures — current offering
- Spring 2025 lectures — previous offering
- Staff assignment org