Does a Boolean consistency penalty improve held-out forecasting log loss when a model sees only one label per pair of propositions? Five arms, one seed, live numbers from the box as they sync.
These runs are invalid (found 23 September). The prompt builder dropped the option name of multiple-choice markets, believing it leaked the label. It is the proposition itself: the model saw “Which AI lab releases an LLM in October?” without “OpenAI”. About 79% of test propositions were undefined, which is consistent with every arm learning only base rates. Creator edits added after a market opened, which can reveal outcomes, were also left in the criteria for 23% of propositions. Both are fixed in training/supervised_pairs.py; the numbers below are kept as a record and are not a baseline.
Validation log loss on the observed form of each pair, in nats, lower is better. The best epoch is chosen by validation log loss, then consistency error as a tie-break; test is scored once on that checkpoint and only appears when an arm has finished.
How to read the numbers. Constant 50% scores 0.693. A rule that knows only the Boolean structure of the forms (0.5 for a literal, 0.25 for an AND, 0.75 for an OR) scores on this validation set, and a 12-number table of each form's training base rate scores on validation but on test. No trained arm beats that table on test yet. The two families of arms are trained on different pair graphs and label counts, so compare arms only within a family.
2,164 training pairs, one supervised label per pair. Validation on 422 pairs.
8,192 training pairs, all twelve labels (singles arm: P and Q only). Validation on 1,530 pairs.
Dashed lines are the constant-50% baseline (0.693) and the Boolean-aware baseline for that validation split. Epoch 0 is the frozen base model before any training, off the chart at about 1.5 nats. Hover a point for the exact value.
Where the finished arms are good and bad, by Boolean form, on the 1,066 held-out test pairs. Only arms whose test scoring has landed appear here; more columns fill in as arms finish.
Every kept epoch adapter was rescored on the test set after training, offline. Validation selected the bold epoch; the best test epoch is marked. Choosing a checkpoint by its test score is post-hoc, so both numbers are reported and neither is the headline.
Hollow ring = the epoch validation selected. Dashed lines: base-rate table 0.578, Boolean prior 0.607.
Two diagnostics computed from every arm's validation predictions. If the penalty only pulls compound predictions toward the 0.25/0.75 prior, the spread of p(AND) across pairs should fall with λ. If it teaches the model that ¬P is the complement of P, the correlation between p(P) and p(¬P) across pairs should approach −1; a model that has not learned negation gives them the same number, correlation near +1.
Standard deviation, at each arm's best epoch. Lower means the compounds sit closer to one value.
Across validation pairs, at each arm's best epoch. Coherent beliefs give −1.
Published Brier scores on binary forecasting questions, no retrieval unless noted. Brier 0.25 is constant 50%; a calibrated forecaster at Brier 0.20 scores about 0.58 to 0.62 in log loss. Papers report Brier only, so the mapping is approximate.
| Forecaster | Size | Set | Brier | Accuracy |
|---|---|---|---|---|
| Constant 50% | — | any | 0.250 | — |
| This run, A4 one-label accuracy | 0.6B | our test, Manifold | 0.236 | 63% |
| Base-rate constant | — | Halawi test | 0.234 | — |
| Llama-2-7B / Mistral-7B zero-shot | 7B | Halawi test | 0.353 / 0.237 | — |
| Llama-2-13B / 70B zero-shot | 13B / 70B | Halawi test | 0.226 / 0.283 | — |
| GPT-4-1106, no retrieval | frontier | Halawi test | 0.206 | 67% |
| GPT-4 + retrieval, on Manifold questions only | frontier | Halawi test, Manifold subset | 0.219 | 64% |
| Manifold crowd at question open | human | same subset | 0.200 | 68% |
| Halawi fine-tuned GPT-4 + retrieval | frontier | Halawi test | 0.179 | 72% |
| Fine-tuned 14B + news, RL on outcomes (Turtel 2025) | 14B | Polymarket | 0.190 | — |
| Llama-3-8B / 70B zero-shot | 8B / 70B | ForecastBench | 0.201 / 0.201 | — |
| o3 with scratchpad | frontier | ForecastBench | 0.146 | — |
| Superforecasters | human | ForecastBench | 0.104 | — |
What "decent" means here. No published model under 7B beats the base rate without retrieval. Clearing Brier 0.22 (log loss about 0.58) on Manifold with calibration intact would beat every published 7B zero-shot number and is the first target. Retrieval buys frontier models about 0.02; reasoning another 0.04; humans another 0.05. Sources: Halawi et al. 2024, Turtel et al. 2025, ForecastBench leaderboards.
An independent read-only review of the code, data and synced predictions, run while the arms were training. Its verdict: the run as designed cannot support the claim that the penalty improves forecasting.
| Threat | Evidence | Explains the A4–A5 gap? |
|---|---|---|
| The gap is shrinkage toward the structural prior, not learned coherence | All of A5's advantage sits on AND/OR forms; singles are slightly worse. A5's compound predictions are A4's with lower spread (SD 0.097 → 0.064) pulled toward 0.25/0.75. Blending A4 toward that prior after the fact reproduces A5 at 0.643 and beats it at 0.615. p(P) and p(¬P) move together in both arms, correlation +0.96, so neither learned negation. | Fully |
| No arm beats a lookup table | The per-form training base rate scores 0.578 on test; A4 scores 0.666. Within-form discrimination (AUC) is 0.52–0.58. | Removes the claim |
| Validation is biased toward the shrinking arm | 53% of validation pairs share a family (test 24%); 87% of validation criteria hit the 1,500-character cap (test 11%); validation's AND rate is 0.31 against 0.19 in train. The base-rate table loses on validation and wins on test, so early stopping selected against learning base rates. | Partly, and it may have picked the wrong A4 checkpoint |
| Confidence intervals are weak | Family links chain 917 of 1,066 test pairs into one component; the clustered interval on the validation gap is −0.02 to +0.05. | Partly |
| Arms not configuration-identical | A5 ran with gradient checkpointing after an out-of-memory relaunch; A4 did not. | No, but a confound |
| Leakage, checkpoint reload, tie-breaker | No cross-split family overlap found; reloaded adapter's test profile matches its validation epoch; the consistency tie-break never fires. | No |
What would settle it (about 2 GPU-hours): an L2-to-prior arm and a permuted-form penalty arm at λ=1, which A5 must beat; A4 and A5 at a higher learning rate so the models at least clear the base-rate table; and rescoring every kept checkpoint on test offline with per-form AUC. Claim only if A5 beats a prior-shrunk A4 and both controls on test, with a second seed agreeing in sign.