Parameter Golf — Technique Explorer

Question

Under a fixed Parameter Golf compute budget, which training techniques can show a measurable signal early enough to rank — and which only look like noise until thousands of steps?

Context

Working catalog for Parameter Golf, grounded in experiment prioritization and timing research and the meta-stack / convergence notes in 2 - Source Material/Research/.

The durable research idea is signal timing, not a leaderboard of final BPB:

• Short runs of late-signal techniques measure noise, not merit — burn a few hundred steps ranking early ideas first.
• Community and paper stacks cluster into free wins (BigramHash, SmearGate), core capacity, efficiency (Muon, QK-Norm), depth (recurrence, partial RoPE), and representation bets (STP-lite, JEPA-lite) that rarely pay before long horizons.
• Fixed wallclock or step budgets should sequence experiments by when an effect becomes distinguishable from noise.

Findings

• Early signal (200–500 steps): SmearGate, BigramHash, exclusive self-attention (XSA), Muon, OrthoInit + LN scale, QK-Norm + Z-loss. Muon shows roughly 2× token efficiency over AdamW from the first steps.
• Medium signal (500–2,000 steps): 3×MLP capacity, EMA weight averaging, attention-over-residuals variants, partial RoPE.
• Late signal (2,000+ steps or post-training): depth recurrence (best recorded ~1.163 BPB, only after thousands of steps), STP-lite, JEPA-lite, late quantization-aware training.

Fourteen techniques total; the interactive filter re-executes that catalog in-browser.

Method

Techniques are tagged by signal timing so a fixed experiment budget can be sequenced: burn a few hundred steps ranking early ideas; reserve full-length runs for techniques that cannot show value sooner. Timing bands come from literature synthesis and local run experience, not from a single controlled factorial.

Limits

• Timing bands are planning priors, not preregistered causal effects. A technique tagged “early” can still fail on a given stack.
• Catalog BPB anecdotes (e.g. depth recurrence ~1.163) are historical run notes — the sealed competitive standings live on the Parameter Golf Run Leaderboard.
• Interactive WASM notebook re-runs filter code; it does not retrain models.

Artifact

Fully interactive: Python via WebAssembly in the browser; the signal-timing filter re-runs code, not a canned view. Source notebook is public in pggolf-experiments; export pinned to its source commit.