Gigatoken Reproduction — SIMD Pretok and Hierarchical Cache

Question

Do Gigatoken’s order-of-magnitude speed claims vs HuggingFace and tiktoken survive on Apple M5 Max, and can the two systems levers be recreated enough to explain why?

Context

Independent reproduction of Marcel Rød’s Gigatoken, distilled in Gigatoken SIMD Pretok and Hierarchical Cache and the mechanism research note.

Two levers make mature BPE tokenizers ~10²× faster than HF/tiktoken (both already multithreaded Rust):

• Replace regex pretok with a SIMD mask scanner (64-byte classify → bitmask boundary algebra → branch-free span emit). Semantics match the GPT-2-family Unicode regex; throughput is the win.
• Memoize pretok→token encodings in a cache-hierarchy-aware short table plus a long map. OpenWebText sees ~99% hits; most pretoks encode to 1–2 tokens packed inline.
• SentencePiece families stay ~10–20× — the SP path does not get the same pretok win (upstream caveat).

Findings

• On a 200 MB OpenWebText slice, warm GPT-2 encode: 4453 MB/s Gigatoken vs 33.4 MB/s HuggingFace — 133×, validation OK on 20,401 documents.
• Vs tiktoken (same GPT-2 encoding): 42× (4953 vs 118 MB/s), zero token-ID mismatches in the first 50 documents. Native file API reaches 6.9 GB/s on the full 210 MB slice — the path that avoids Python object traffic.
• Other BPE-family tokenizers: Phi-4 100×, OLMo-2 98×, Qwen3 ~67× vs HuggingFace. SentencePiece TinyLlama only 16×, matching Gigatoken’s SP caveat.
• Absolute ratios below the published M4 Max / 11.9 GB table (1,268× HF, 140× tiktoken). Corpus size, cache residency, and API path move the number; the HF order-of-magnitude claim still holds.
• Lab remote: a3fckx/gigatoken-lab (private) tracks sealed notebook commit 42861c4; upstream remains the production library.
• In the RSI Lab map this is Club A systems reproduction: positive sealed claim with educational levers, not a self-improving agent.

Method

Official gigatoken bench vs HuggingFace with --validate, plus a separate tiktoken comparison. Corpus: first 200 MB of Stanford CS336 OWT sample, split on <|endoftext|>. Warm runs preferred on macOS after cold Gatekeeper. Notebook recreates ASCII GPT-2 regex vs hand FSM, Zipfian pretok-cache capacity sweeps, batch byte classification, and a live side-by-side encode against pinned results.json.

Limits

• Local M5 Max / 200 MB slice — not the published M4 Max / 11.9 GB table; absolute multipliers are not portable claims.
• Educational Python recreations in the notebook are not GB/s reimplementations; sealed benches come only from runs/results.json.
• This lab does not fork or replace upstream Gigatoken; production library remains Marcel Rød’s.
• Live §3 encode timings in the notebook are pedagogical — do not treat them as the published claim.

Artifact

Executed snapshot pinned to the Gigatoken Lab source commit. Canonical bench table: runs/results.json (hash-pinned above). Upstream Gigatoken remains the production library; this report claims a local reproduction and mechanism explanation only. Notebook: notebooks/gigatoken_levers.py.