Reinforcement learning from exploitation feedback, trained and judged on the Paleka forecasting-consistency suite: does removing Dutch-book exploitability make a forecaster genuinely better — more accurate, better calibrated — or only harder to arbitrage?
Kelvin (Futarchy Labs) · drafted with Fable · 2026-08-19 · for Jessica Taylor's review
What we're combining, and why it's new
Paper 1 — the measurement. Paleka et al. (arXiv:2412.18544) built 10 consistency checks over logically related forecasting questions (negation, conjunction bounds, conditional chains, expected evidence…) with an arbitrage-based violation metric, and released the instantiated tuples. Their own repair (inference-time arbitrage) improved only the optimized checks — no held-out transfer, no Brier gain. Training on the metric is listed as future work; no one has published it.
Paper 2 — the training signal. Matta/Naphade/Zou's RLE (ICML'26 workshop) post-trains against Dutch-book exploitability: an adversary LP extracts guaranteed profit from incoherent prices, and that profit is the negative reward. Shown only at adapter scale, on knowledge QA, anchored by ground-truth Brier. No public code.
Our addition. A third arm replaces the ground-truth anchor with a KL to the base model's own beliefs — coherentize what the model already believes instead of teaching labels. It needs no resolved outcomes, so it trains on unresolved forecasting questions, which is the regime forecasting actually lives in. To our knowledge this combination is unpublished.
Hypothesis under test: training a forecaster to minimize exploitability on logically related question tuples improves held-out consistency checks, Brier score on temporally held-out resolved questions, and calibration — beating both a Brier-only finetune and inference-time arbitrage repair. Falsifiable outcome: coherence improves on trained checks only, and accuracy/calibration are flat — consistency training is cosmetic.
The workflow
One pipeline, three reward arms, two approval gates. The trainer's reward is exploitability (the LP adversary's guaranteed profit on the episode's stated probabilities) plus an anchor term that differs per arm; every checkpoint runs the identical eval battery against two frozen baselines.
Experimental arms
Arm
Reward
Needs resolved outcomes?
What it isolates
control
Brier-only finetune (no exploitability term)
yes
Is coherence training better than plain accuracy training? (RLE's own comparison, reproduced)
The published recipe, moved from knowledge QA to real forecasting tuples
A₂ RLEF-KL
−Exploit(c, S) − λ·KL(c ‖ base model's probabilities)
no
Our contribution: label-free coherentization, trainable on unresolved questions; λ swept
What counts as a result
Primary: violation on the four held-out check types, vs base model and vs ArbitrageForecaster. Paleka showed inference-time repair does not transfer; if training does, that is the headline.
Ground truth: Brier on questions resolving after the training cutoff (no temporal leakage — their follow-up paper's main warning).
Calibration: ECE/RMSCE separately — coherence and calibration are empirically orthogonal, so we report both, never a blend.
Regression watch: per-check dashboard; outcome-RL is documented to improve conjunction checks while worsening negation. Any arm that regresses a check family gets flagged, not averaged away.
Decision gates (where Jessica signs off)
GATE 0 — this document. Approve the design: arms, splits, metrics, and the falsifiable framing. Changes are cheap now; everything below is mechanical.
GATE 1 — after the pilot (~1 week). Small model, ~5k episodes, one seed. Sane reward curves, A₀ collapsing as predicted, λ range identified. Cost ceiling ≈ $100. Go/no-go on the full run.
GATE 2 — after the full run (~2–3 weeks). All arms, 2–3 seeds. Decide: write up (workshop-length even on a null result — "coherence training doesn't transfer" is publishable), or extend to the 2028 long-horizon benchmark and larger models.
Logistics (decided at Gate 1, not before)
Managed path (pilot default): Fireworks RFT with a custom Python reward evaluator — gpt-oss-20b is both serverless and RFT-eligible today on our account; zero infra. Estimated $80–$500 for the pilot scale.
Self-hosted path (full run, and required if we stay on Qwen 3.5 9B): 1–8×H100 rental ($1.2–3/hr/GPU), TRL/verl GRPO, weights local — also unlocks probe readouts and custom losses later. Estimated $60 (single-GPU days) to ~$1,150 (8×H100 × 2 days).
Eval is cheap regardless: ~$1 per full 3,171-tuple sweep serverless; live dashboard already running at consistency.futarchy.ai.
Open account items: Fireworks serverless-training beta requested (unblocks managed GRPO on Qwen 3.5 9B); base-model choice is a Gate 1 decision with real trade-offs (no single model currently supports every recipe managed).
Known risks
Goodharting the LP: flat 0.5-everywhere answers pass several checks. Mitigation: the anchor terms, plus reporting discrimination (AUROC) so uninformative-but-coherent shows up.
Reward hacking the parser: probabilities are elicited from generated text; malformed outputs need a defined reward (zero), not a crash.
Small-model ceiling: if the base model's beliefs are noise, coherentizing them may just polish noise — the KL arm's λ sweep is also implicitly testing this.
Repo: futarchy-fi/consistency-ft (private) — README, LITERATURE, COSTS, and the v0 eval harness (live-verified) are already in place. GATE 0 CLEARED: approved by Jessica 2026-08-19, with one amendment — a knowledge-cutoff contamination audit (verify advertised cutoffs with high-salience post-cutoff questions) now gates the Brier evaluation windows. Full task DAG: plan.html; beads under epic hub-t8k9c.