Two research lines that turn out to be one: futarchy-fi's simulations repo ran LLM agents inside real decision-market mechanisms and measured where they fail; consistency-ft trains exactly that faculty. This page is self-contained — it assumes you haven't seen the futarchy experiments.
The experiment, in one screen
The futarchy-fi/simulations repo (private — ask Kelvin for access) contains a decision-market engine ("Proposal Poker"): each of 150 proposals has a hidden quality x ~ N(0,1); five agents each get one private noisy signal sj = x + noise (precision proportional to the agent's wealth), and stake real (simulated) money on APPROVE/REJECT across 3 sequential rounds of a binary staking market, with participation costs, a 1% fee, a winner subsidy, and a costly verification oracle. Every arm sees identical draws (common random numbers), so arms differ only in the decision process:
Arm
Decision process
Why it exists
A
coded rational-Bayesian agents + market
mechanism works with ideal agents
B
LLM agents (Claude Haiku 4.5) + market
the arm under study
C
poll: average the same LLMs' pre-market beliefs
does the market add anything over just asking?
D
one LLM manager sees all five signals
centralized baseline
F / G
one agent covertly bribed / bribed and instructed to lie coherently
manipulation and audit-evasion
Separately, the repo reproduces Galanis (2026) — LLMs trading in a prediction market over structured information — as an exact game solved with CFR+: the equilibrium aggregates well on every information structure, so when LLMs degrade on the hard structures, that's an agent limitation, not a market one.
The shared finding. The LLM traders are statically well-calibrated (Brier 0.076; stated 0.6 → realized 0.627) and stake in proportion to how informed they are (Spearman +0.86) — the two virtues markets rely on. But they do no within-market belief updating at all (precise claim below) — so the market's price aggregates worse than an unweighted poll of the same models (r = 0.882 vs 0.925 against truth), because stake-weighting adds noise while the beliefs never improve. The bottleneck is agent belief dynamics, not market design. Calibration-without-coherence is the exact failure consistency-ft exists to train away.
"Zero within-market updating" — the precise claim
It sounds unbelievable, so here is exactly what was measured. In round 0, each agent's prompt contains only its private signal — no market state (these answers double as the poll, Arm C). In rounds 1–2 the prompt additionally shows the public history: the running APPROVE/REJECT stake totals from the other agents' earlier moves. A Bayesian with an independent noisy signal should treat those totals as evidence — four other signals are worth roughly 4× their own — and move toward the market. What Haiku does instead, over 2,250 logged calls (raw JSONL in the repo):
Mean |belief change| from an agent's first to last stated belief: 0.027 on a 0–1 scale — and the changes are directionless: 241 drifted away from the market majority, 217 toward it. Coin-flip.
Nothing is learned across rounds: the round-0 mean belief predicts the hidden quality at Pearson r = 0.925; the last-round mean belief, after two rounds of watching the book, predicts it at r = 0.924.
It's not a too-easy-environment artifact. The v1 rerun deliberately de-saturated the environment (noisier signals, real headroom for pooling to help): same null, and the price (r = 0.882) still underperforms the naive belief average (0.925).
They even ignore a manipulated book. In Arm F a bribed agent skews the visible stakes 58:1; the other agents' belief drift stays at −0.002 to +0.003. They neither follow the distortion (good) nor bet against the mispricing (bad) — they just don't react to the book at all.
Caveats, honestly: one model tier (Haiku 4.5, thinking capped at 1024 tokens), one prompt format, five agents, 150 proposals. "Belief" = the number the agent states in strict JSON each round. A stronger model or a prompt that begs for updating might behave differently — untested. But within this setup the null is clean, replicated across v0 and v1, and consistent with Galanis's finding that aggregation degrades with reasoning-about-others complexity.
Why this matters for consistency-ft: "update your stated probability coherently when shown new evidence" is precisely what the conditional consistency checks (Cond, CondCond, ExpEvidence in Paleka et al.) measure, and what exploitability training (RLE, our RLEF variant) directly optimizes. An agent whose beliefs are Dutch-book-coherent including its conditionals cannot both call the book "evidence" and leave its belief unmoved.
Blue arrows: what consistency-ft ships into the futarchy stack. Violet arrows: what the futarchy research sends back. The capsule is the empirical finding both sides independently reached.
consistency-ft → futarchy
What we ship them
The missing market agent. Their traders never update; our conditional checks train updating. Drop a tuned checkpoint into proposal-poker and A/B market-vs-poll on their CRN shards.
A liar detector that works. Their Arm G "aligned liar" beat the stake-vs-belief audit 0/118 and profited; only slow outcome-based Brier caught it. A Dutch-book audit over related propositions is instantaneous — fabricated beliefs must stay coherent or pay a visible penalty.
Coherent quoting. bayes-market enforces cross-market coherence in the engine; a coherence-tuned model carries it in the head — an incoherent maker is a subsidy leak equal to our violation metric.
Soft inversion. Their batch mechanism is jammable by one inconsistent order under strict Bayesian inversion; projection-onto-nearest-coherent is the fix, and we build that machinery anyway.
futarchy → consistency-ft
What they ship us
Adversaries that adapt. Their best-response manipulators showed static analyses understate exploitability 3–5×; their evolution design is exactly the "adversarial check generation" Paleka names as the open anti-Goodhart fix.
An eval with economic units. ΣPnL = b[ln2 − LogLoss] means "can an adversary profit off your checkpoint in a venue" is a proper-scoring-rule eval needing almost no resolutions.
The saturation lesson. Their v0 environment was too easy to tell arms apart; same trap as training on checks the base model already passes (Consequence 0.03 vs Negation 0.39). Measure headroom first.
Ranked joint experiments
E1 · DAYS — Coherence audit vs Arm G liar, in their existing harness. Cheap, novel for both repos, no training required.
E2 · AFTER PILOT — RLEF-tuned checkpoint as market agent: does the market beat the poll? Our generalization result and their aggregation result, one experiment.
E3 · CHEAP — Galanis Hard-structure eval of tuned checkpoints; their CFR+ and LLM-empirics baselines already exist.
E4 · WHEN NEEDED — Evolution-based adversarial check generation, once the static LP adversary saturates.