Parameter Golf

Under a fixed compute and parameter budget, which single architectural or optimizer change actually moves validation bits-per-byte — and how many steps does a technique need before its signal is distinguishable from noise? Experiments change one idea at a time, sync result.json from remote GPUs (RunPod / MI300X), and rank on quantized validation BPB with float BPB alongside for reference. Run lineage records each experiment’s parent and its single change so unequal step counts remain readable.

Public surfaces are the technique explorer and run leaderboard notebooks over the open pggolf-experiments repository. Fourteen techniques are catalogued by signal timing: early (200–500 steps) includes SmearGate, BigramHash, XSA, Muon (~2× token efficiency over AdamW from the first steps), OrthoInit + LN scale, and QK-Norm + Z-loss; medium and late buckets reserve full-length runs for ideas that cannot show merit sooner. As of the pinned leaderboard snapshot, six runs sit on the board: ROCm-patched baseline 2.067 quantized val BPB after 251 steps (17.1M params, 600 s wallclock on one MI300X); current leader exp004_partial_rope_promo at 1.295 over 2,463 steps at the same parameter budget; depth recurrence’s best recorded result remains ~1.163 BPB but only after thousands of steps.

The notebooks are views over versioned experiment definitions and returned results, not substitutes for the training repository. Private GPU machines, raw checkpoints, and unsynced run trees stay off the public surface; new runs appear only after their result.json lands and the publication snapshot regenerates.