Report

Agent Simulation Lab: What I Built and How

Agora as a state-first agent-based simulation lab: Markov worlds, scored forecasts, verifier-first shadow brains, and a sealed CompactCTM negative — with authority boundaries that keep Memory Store out of tick-level control.

Verdict Mixed

Published: Jul 2026
Verdict: Mixed
Verification: Provisional

Question: What does my agent-simulation work actually do, how is it built, and which ambitious claims are still forbidden?
Intent: Publish a plain map of the simulation stack, methods, sealed results, and cognitive-architecture boundaries so readers can navigate Agora without marketing overclaim.
Method: Synthesize the Agora project page, calibration runs, CompactCTM Stage-1 negative, and the verifier-first brain-shadow architecture already sealed in Club B reports.
Next test: Freeze a public metrics card and clear a held-out predictive gate before any learned controller earns authority over live transitions.

Agent Simulation Lab: What I Built and How

Question

What does my agent-simulation work actually do, how is it built, and which ambitious claims are still forbidden?

Context

Agora is the owning project: a state-first agent-based simulation (ABM) lab in the Recursive Self Improvement Lab’s Club B (measure / falsify). The job is not to ship a chat agent that “acts like society.” The job is to keep worlds, rules, runs, forecasts, and scores as plain inspectable state — then refuse promotion until an external verifier clears a held-out gate.

This sits beside — and must not be confused with — Memory Store, the Julep product I work on in the cognitive memory architecture domain. Memory Store is durable workspace memory for real agent workflows. Agora is a sealed simulation lab. Different authority, different evidence.

What I built

1. Vault-native simulation OS

Populations evolve under a Markov chain of rules. Worlds, rules, runs, channels, and targets live as OKF files (JSON / JSONL / Markdown / Canvas) so every step is diffable text. Named worlds include:

opinion_diffusion — adopt / undecided / reject macro-dynamics
sir_epidemic — susceptible / infected / recovered
rumor_debunk — rumor spread under corrective pressure

Operator loop:

\text{run} \rightarrow \text{predict} \rightarrow \text{evolve} \rightarrow \text{score against }\texttt{target.json}

2. Calibration before claims

First honest numbers are public in Agora Calibration Runs: opinion-diffusion and SIR trajectories, Markov+Kalman forecast error, and small rule-evolution improvements on named worlds. Verdict is mixed / provisional — local calibration demos only. No claim of survey replacement, external forecasting superiority, or commercial readiness.

3. Verifier-first shadow brain (no control)

A predictive “brain” layer was added behind run --brain-shadow with a hard rule: prediction must not change transitions, RNG, metrics, or final world state. Architecture keeps six jobs separate:

Job	Owner	Must not
World / observations / actions / replay	Agora sim	become the memory product
Observation encoding $Z_t$	representation layer	collapse into durable memory
Transient thought / sync	bounded CTM-style recurrence	own long-term context
Persistent predictive belief $S_t$	StateCore boundary	copy $Z_t$ or chat transcripts
Durable semantic / episodic context	Memory Store (product domain)	hold per-tick activations or decide reward
Promotion authority	external verifier (Brier / log loss)	trust internal confidence

Target flow (aspirational full loop; only the safe front half is implemented):

\text{World} \rightarrow Z_t \rightarrow \text{bounded thought} \rightarrow S_t \rightarrow \text{retrieval} \rightarrow \text{action} \rightarrow \text{external verifier} \rightarrow \text{trace}

Stage-0 proved the shadow does not perturb matched baseline runs.

4. CompactCTM Stage-1 — sealed negative

CompactCTM Shadow Stage-1 maps CTM-inspired recurrence onto Agora opinion-diffusion transitions with no control authority and no Sakana CTM paper parity. Held-out CompactCTM mean Brier 0.6277 vs empirical prior 0.3578 (Δ −0.270); log loss also worse. Independent audit: reject / do not promote.

That failure is the point of the lab: a brain-inspired proposer lost to a dumb prior under a frozen gate, so it stays unpromoted in public.

How (method discipline)

State first — if it is not in files, it did not happen.
Targets before stories — target.json trajectories beat narrative success.
Shadow before control — predictors may forecast; they may not steer until held-out credibility clears.
Publish negatives — CompactCTM reject is a first-class result, not a deleted experiment.
Keep Memory Store out of the tick — cognitive memory architecture (schemas, Briefs, checkin/recall/record) is the product domain at Julep; it must not become Agora’s per-tick activations or reward.

Findings

Agora is a real ABM lab with inspectable worlds and scored calibration, not a demo GIF.
Verifier-first shadow infrastructure works as a non-intervening measurement harness.
CompactCTM Stage-1 fails held-out prediction against an empirical prior — sealed and public.
Agent simulation and cognitive memory are complementary surfaces: simulation measures population dynamics under rules; Memory Store compounds workspace knowledge for agents that ship.

Limits

Source trees (agora, agora-agent-brain) remain private.
No claim that Agora forecasts real populations, beats commercial simulators, or earns control authority.
No claim of Sakana CTM paper parity from the Agora derivative.
No claim that Memory Store “is the Agora brain” or that product memory primitives were evaluated inside these sealed runs.
Full living-OS / “ask it anything” language stays thesis draft until held-out gates clear.

Artifact / next

Project: Agora
Reports: Calibration Runs · CompactCTM Stage-1
Memory product (employer): Memory Store · memory.store
Portfolio map: Cognitive Systems Portfolio Thesis
Next gate: freeze a public metrics card; clear held-out predictive credibility before any learned controller touches live transitions.

agent-based-simulation
multi-agent-systems
continuous-thought-machines
verifier-first
state-first-ai