Proposal · 21 September 2026 · Kelvin and Jessica · draft, unfunded

Self-Trade Run

Train a forecaster by letting it trade against a combinatorial market it built itself, week by week, with outcome-based RL in the style of Turtel et al. and Mantic. The market maker enforces coherence; settlements and dated evidence supply the information; the model's one-shot beliefs are distilled from the market it produced.

The one sentence to hold onto. Self-trade is a coherence-and-aggregation engine, not a knowledge engine. AlphaZero's self-play manufactures unlimited new experience; this game manufactures unlimited new consistency, while every bit of information still arrives through the evidence an agent reads at time T and through settlements. That decides most of the design below.

Five claims, each with a control

  1. Search improves beliefs. The market after K trades, WK, scores better on settlement log loss than the policy's one-shot output on the same evidence.
    frozen model, no RL
  2. Amortization works. Distilling WK into the one-shot output improves next round's one-shot. With claim 1 this is the policy-improvement operator that makes the loop AlphaZero-like rather than an ensemble.
    arm 3 vs arm 4
  3. Coherence transfers. One-shot outputs become more coherent on formula queries never trained on, without a penalty term, because the maker enforced coherence during search.
    Paleka checks, overlap loss
  4. Beats plain outcome RL at equal compute. On Turtel's released test set, against a single-question GRPO arm with the same model, data, tokens and reward.
    arm 1
  5. No memorization. Accuracy on retired questions with no evidence does not exceed fresh held-out accuracy by more than the base-rate gap.
    recall vs forecast probe

A round

Strict chronological weeks, one pass, no replay. Agents trade on evidence dated at or before T. Settlements arriving in week T+1 condition the joint, pay out every position that references them, and retire the question for good.

week T evidence dated ≤ Tarchive · dated search K turns on WLMSR trades, KL budget ε snapshot WTdistillation target week T+1 settlementscondition joint, pay, retire K turns on Wevidence dated ≤ T+1 snapshot WT+1reference for T's marks carry W reward for T's turnssettled: log-score payoff else: mark vs WT+1live, or frozen league policy updateGRPO w/o σ · KL anchor β distill W into one-shot head, every round

Solid arrows are the data flow inside one week and into the next. The dashed arrow is the amortization step: the end-of-round market becomes the supervised target for the model's one-shot belief head, which is what gives an explicit belief structure at inference time without running the market.

One trade, scored three ways

The LMSR makes every reward exact. A trade that moves the market from p to q pays b·(log q(ω) − log p(ω)) at outcome ω, a log-score difference against the pre-trade market. Marked against a later reference r instead of settlement, it pays b·[KL(r‖p) − KL(r‖q)], the "score the earlier belief against the later one" rule from the explicit-beliefs page. And the KL moved, KL(q‖p), is charged against the turn's budget ε.

0.50
0.70
0.60
0.05
if YES
if NO
marked vs r
KL moved

Expected payoff of your trade as a function of the later reference r, with b = 100. It is linear in r and crosses zero at r*: the trade pays if the market later believes more than r*. The dot is the r you chose; the dashed vertical lines are p and q.

TD bootstrap. If the question settles by evaluation time the reward is the realized payoff; otherwise it is the mark against r. Under a calibrated market the mark is the r-expectation of the settlement payoff, so mixing the two is the same trade-off as TD(λ): early and noisy against late and exact. The reference r is where the two arms differ: the live market WT+1 made by the current policy's own clones, or a frozen market made at T+1 by the previous checkpoint. Live is cheap and rewards herding, since clones score against clones. Frozen removes that channel at the cost of staleness. Settlement dominates in both arms because it is the only profit that is not zero-sum among clones.

A structure trade

The world model is the bayes-market factored LMSR: a junction tree, minimum-KL reweighting on every trade, treewidth cap 8, resolutions condition the joint. A link exists exactly when a conditional differs from its marginal, so adding a dependency is a conditional trade and removing one is trading it back. The rest of the joint adjusts by the I-projection, and the trader pays for it.

0.50
0.50
0.80
before
v=yesv=no
u=yes
u=no
after
v=yesv=no
u=yes
u=no
P(v) after
if u=yes, v=yes
if u=yes, v=no
if u=no0, called off

Complexity is a resource, not a price. The treewidth cap means the model cannot represent arbitrary structure; over-budget trades are rejected with the state untouched. So the agent plays a real structure-learning game: which dependencies deserve width. Two rules follow. Rejections are returned as feedback and masked, never scored as losses. And the move set collapses to two actions, trade-to-probability with an optional context, and a share trade on a formula.

A turn, and the learning loop

What one turn is

  1. State. A scope: variable v, its marginal, the conditionals and neighbours in its Markov blanket, candidate neighbours from the same family and BGE nearest neighbours, dated evidence, the week, the KL budget ε, remaining width.
  2. Action. Up to 1,500 tokens of reasoning, then a block of at most m trades. The engine executes them in order; a rejection comes back as feedback and the agent may retry once.
  3. Trust region. Total KL moved per turn ≤ ε. This is PPO's constraint applied to beliefs. It also caps extremization, Turtel's main failure mode, and it is the leakage budget.
  4. Coverage. K turns per round so every active variable is visited at least twice, scopes sampled by evidence volume and time to close.

Group rollouts on forked markets

turn statescope + evidence live marketone committed

G rollouts of the same state run on snapshot copies of the market, each scored against the reference or settlement, giving the group-relative advantage without division by σ (Turtel's modified GRPO, Mantic's choice). One rollout is committed: random in v1, on-policy; chosen by a value head in v2, which is where best-of-G search starts improving the market beyond the policy.

reward(turn) = Σtrades { settled: b·(log q(ω) − log p(ω)) | else: b·[KL(r‖p) − KL(r‖q)] } on the slice, called off if a context variable is unresolved
update: modified GRPO, no σ; KL anchor β to previous checkpoint; league of past checkpoints plays a fraction of turns and supplies the frozen r
distill: one-shot head ← KL(WT slice ‖ output); movement head ← E[KL(WT+1 ‖ WT)] scored against realized
no coherence penalty: the maker prices every formula by inference; there is no internal Dutch book to exploit

Leakage and memorization

  1. Retire on settle, single pass. A settlement is used once, as the scalar reward of past turns, and the question leaves the trading set. Turtel found multi-epoch training "learns the future"; here there is nothing to learn it from.
  2. Labels never enter as tokens. They shape the policy only through a KL-bounded update, so the information a settlement can write into the weights is bounded in nats per update by β and ε. That number can be reported.
  3. Most reward is the martingale term, market movement between rounds, which is not a label. Resolved questions are a small fraction of the active set each week.
  4. Hold out time, not questions. Train on weeks 1..N, freeze, evaluate on later weeks. Families are one scope, so a by-date resolution fixing its siblings is inference the model is supposed to do, not leakage.
  5. Measure it. Query retired questions with no evidence; report recall accuracy and fresh held-out accuracy as two numbers.
  6. Evidence is the remaining channel: the Manifold archive of comments, bets and edits is dated by construction; Exa needs the leak judge and its residual is reported, as Turtel does.

Two domains, four arms

DomainDataBase modelPurpose
A · PolymarketTurtel's released test set (1,265 questions, frozen Exa prompts, price at prediction) as held-out for every arm; a rebuilt 10k training set from the Gamma and CLOB APIs, resolved before February 2025, with dated Exa and a leak judge. Multi-outcome events give partitions, date series give monotone chains.DeepSeek-R1-Distill-Qwen-14B (faithful) or gpt-oss-120b. Qwen3 postdates the test window and is excluded.Comparability: ReMax 0.190, o1 0.202, market 0.151; 10k-only target 0.199
B · ManifoldThe local archive, explicit multiple-choice structure, 14k labeled propositions, comments and bets as dated evidence; families as scopes.Qwen3-8B, questions from May 2025The stage-0 gate and the coherence claims, where derived Boolean forms are far richer
ArmReward referenceMarketDistillationTests
1 · Turtel controlBrier on settlementnone, single questionnoneclaim 4 baseline
2 · market, livesettlement, else live WT+1yesnoneherding hazard measured
3 · market, frozensettlement, else league WT+1yesnoneclaims 1, 4, 5
4 · market, frozen, distilledas arm 3yesone-shot and movement headsclaims 2, 3

All arms share model, data and token budget. Baselines on every split: constant 50%, the per-form base-rate table that no supervised arm cleared on 20 September, market price at T as the ceiling, and the supervised A3 arm.

Stages and cost

StageDomainWhat it settlesComputeCostWeeks
0 · gateManifoldClaim 1 with a frozen 8B as trader on 4 weeks of data, no RL. If WK does not beat one-shot, the market is not a policy-improvement operator and stages 2 and 3 do not run.1×H100, ~8 h~$501
D · dataPolymarketRebuild the 10k training set with Exa and judge; verify test-set scoring against Table 1.API only$500–8001–2
1 · replicationPolymarketReMax and modified GRPO at 14B on the 10k set, scored on his test set. The control arm, and a result on its own.4×H100, ~40 h$400–6002
2 · market armsPolymarketArms 2 to 4 at 14B, same data and test set; claims 2, 4, 5.4×H100, ~60 h$700–1k1–2
3 · depthManifoldClaims 3 and 5 at depth: full 70-week pass at 8B on the winning arm.4×H100, ~5 days$1.5–2.5k1

H100 at $2.25 to $3.49 per GPU-hour, September 2026; rollouts dominate. About $2k and five to six weeks of engineering through stage 2, most of it the RL loop and the retrieval pipeline any outcome-RL work needs. Synthetic data comes in this order: derived Boolean forms (exact, free), the market's own end-of-round targets (manufactured each round), and only later generated questions from dated news at roughly $0.05 to $0.20 each.

Decisions before the run

Ordered by how much they change the work. Each carries a recommendation. Status as of 22 September: five decided, three open pending discussion with Jessica.

  1. Go or no-go on a phased budget, and who owns each piece.decidedAbout $2k through stage 2 with a hard stop per stage. Three pieces need an owner: the RL loop, the fork and snapshot API in bayes-market, and the dated retrieval pipeline. The Mac Studio plan gave Jessica the RL and exploitability work, so this needs her agreement.recommend fund stage 0 and stage D now, decide stages 1 and 2 on their results≈ $600 now
  2. Base model for the Polymarket domain.decidedDeepSeek-R1-Distill-Qwen-14B is faithful to Turtel, dense and well-trodden for GRPO. gpt-oss-120b gives a longer clean window and Mantic's recipe, but needs MoE LoRA engineering and has no verified cutoff. Qwen3 is excluded by the test window.recommend the 14B for stages 1 and 2
  3. Fund the training-set rebuild.decidedThe questions are mostly on disk already: Halawi's raw dump holds 14,246 Polymarket questions from 2020 to mid-2024 with resolutions, background, criteria and dates, and the free Gamma API fills the gap to January 2025. What still has to be bought is the dated evidence per training question; Turtel used Exa, and GDELT (free, titles and URLs only), Common Crawl News (free, heavy) or Perplexity's search API are the alternatives. Summaries and the leak judge cost about $300 whichever search is used. Turtel's test set ships with its Exa prompts frozen, so evaluation evidence is fixed regardless.recommend yes$500 to $800 in API spend
  4. Accept the gate rule now.openStage 0 asks the cheapest version of the core question: if a frozen model trades K times on the market, are the resulting beliefs better than asking it once? If not, the market cannot be the "search" that gets distilled into the policy, and the RL stages would train toward a target no better than the model's own guess. Agreeing the rule in advance means a weak result stops the later $1.5k rather than being explained away. Stage 0 also builds the engine integration every later stage needs.recommend yes≈ $50, one week
  5. Pre-register the evaluation.openWriting down before training which test set, which arms, which baselines, and how the reported checkpoint is chosen. The 20 September run is the example: validation picked epoch 2 and test preferred epoch 3; reporting epoch 3 would let the test set choose the number. Fixing "select by validation, score test once" beforehand removes that, and likewise stops picking the best of four arms after seeing them. It costs a paragraph.recommend yes, written into the doc before any RL spend
  6. Keep both reward references, or drop the live arm.openA trade made in week T is scored by where the market stands at T+1. In the live arm that market was moved by the same model's own clones, so the model can be rewarded for predicting its clones rather than the world. The frozen arm scores against a market made by last week's checkpoint, which closes the loophole but is a week stale. Running only the frozen arm never shows whether the simpler live design would have worked, or how large the clone-herding effect is. That is the $250.recommend keep both
  7. Commit rule and starting constants.decidedRandom on-policy commit in v1, value-guided best-of-G later. Liquidity b at about ten times one week's settlement payout, ε = 0.05 nats per turn, two visits per active variable per round. None of these are load-bearing at pilot scale.recommend take the defaults, tune in stage 0
  8. Publication of this page.decidedPublished 22 September at the request of Kelvin; the proposal doc is on the plan branch.

Known risks

  1. Bootstrapping on the policy's own market is the deadly-triad setting. The frozen reference and the KL anchor are the stabilizers; arm 2 exists to measure how bad the live version is.
  2. State-dependent actions. Treewidth rejections must reach the policy as feedback, never as losses.
  3. Engine latency at 14k variables is unmeasured; the module claims milliseconds per trade at width 8.
  4. A small pilot model may not read evidence well enough to show claim 1; the 0.6B run learned only base rates. If stage 0 fails at 4B, repeat once at 8B before concluding.