The model is asked about each proposition in its own fresh context and states a probability. One episode is that set of stated numbers. The reward is minus the squared violation of the logical constraint between them. Nothing is sampled, estimated or adversarially solved. Move the sliders to see the payout.
Reward R = −(c(A) + c(¬A) − 1)² over every pair of stated numbers. Darker is a worse payout. The dashed diagonal pays zero and is the coherent set. The ring is the current pair from Chapter 1.
One episode, three moves. Ask "P(A)?" in one context, ask "P(¬A)?" in another, read the two numbers, pay −(c₁ + c₂ − 1)². The separate contexts are the point: the model cannot copy its first answer, so coherence has to come from its beliefs, not from its scratchpad.
Because every query runs in its own context with the same weights, the episode is an imperfect-recall game: one policy, several forgetful copies, one shared payout. Stable points of policy gradient in such a game are its causal-decision-theory equilibria, and those are exactly the KKT points of the score. That is the bridge Jessica Taylor's dice-enterprise work makes precise.
RLE's exploitability is a linear program: a bettor picks stakes in [−1, 1] on each stated number and the worst consistent world is chosen against the model. Its value is |v|, piecewise linear with a kink at coherence. Squaring the violation replaces the saturated bettor with one whose stake is proportional to the violation. Both are legitimate rewards on stated numbers; the quadratic one has a gradient that vanishes as the model gets it right, which is what lets standard policy gradient settle instead of oscillating around the kink.
LP exploit |v| squared v². Same zero, same sign, different shape near coherence.
LP stake sign(v) quadratic stake 2v. The LP bettor goes all-in the instant the book is off. The quadratic bettor's stake shrinks with the violation, so the training signal is proportional to how wrong the numbers are.
Two propositions, one affine equality. The reward is symmetric: stating 0.01 for both is as bad as stating 0.99 for both, and every pair on the diagonal pays zero. The expansion below is only there to show the reward is a smooth polynomial in the stated numbers, not a rule the environment follows.
Both stated numbers receive the same gradient, −2v. The environment cannot tell the model which one is "wrong"; it only says the pair does not add up. Which number moves is up to the model's beliefs about A.
A parabola whose peak sits at 1 − c(A). Move c(A) and the peak slides. The dashed line marks the coherent answer for the current c(A).
An inequality. The feasible side is half the square, so the reward is zero on a region, not a line. The squared hinge is the standard smooth penalty: zero and flat on the feasible side, quadratic beyond it. On stated numbers this is computed directly; no approximation is involved.
On the feasible side the gradient is exactly zero: the environment is silent about how much smaller p(A∧B) is than p(A). Other constraints (p(A∧B) ≤ p(B), the chain in Chapter 4) supply that information.
The lower-right triangle pays zero. Above the diagonal the penalty grows with the square of the overshoot. Compare Chapter 1, where only a line pays zero.
Negation with more outcomes. Four mutually exclusive, exhaustive events, four fresh contexts, one payout on the total. Every stated number gets the same gradient, −2·(Σc − 1), so the environment pushes the whole set up or down together and leaves the split between events to the model.
The bar should end at the mark. Overshoot and undershoot are penalised alike.
One parabola in one number, the total. Four sliders, but the environment sees only their sum. Mutual exclusivity itself (no two events both true) is a separate constraint set, p(Eᵢ∧Eⱼ) = 0, and needs its own queries.
The constraint is a product, so the reward is quartic in the stated numbers and the gradients are no longer uniform. The conditional's gradient is scaled by c(B): when B is rare, the environment says almost nothing about p(A|B). That is the same weak-signal problem a Bayesian has with rare conditions, and it is a property of the constraint, not of the training method.
−2v·c(B) for the conditional, −2v·c(A|B) for the condition, +2v for the conjunction. Slide c(B) toward zero and watch the first bar vanish.
A parabola whose peak sits at c(A∧B) / c(B). If that ratio exceeds 1 no coherent conditional exists: the model's stated c(A∧B) is already larger than its stated c(B), and the Chapter 2 bound for B is what should fire.
Each constraint is a check on a small tuple of stated numbers. Each check is one episode: ask the tuple's propositions in separate contexts, compute the violation, pay minus its square. A training run mixes episodes across checks and tuples and adds one anchor query so the model cannot become coherent by becoming uniform.
| Constraint | Reward on stated numbers | Queries | Zero set | Gradient shape |
|---|---|---|---|---|
| Negation | −(c(A) + c(¬A) − 1)² | 2 | a line | same for both numbers |
| Conjunction bound | −max(0, c(A∧B) − c(A))² | 2 | a half-plane | zero when feasible, ±2v beyond |
| Partition (k events) | −(Σ c(Eᵢ) − 1)² | k | a hyperplane | same for all k numbers |
| Conditional chain | −(c(A|B)·c(B) − c(A∧B))² | 3 | a curved surface | scaled by the other factor |
| Anchor (any of the above) | −λ·(c(A) − cref(A))² | 1 | the reference | pulls toward resolution or the previous checkpoint |
What the sliders do not show. All-0.5 satisfies negation exactly and the conjunction bound trivially, and constant-uniform answers satisfy partitions. Coherence alone has degenerate optima. The anchor row is not optional; consistency-ft's choice is a KL-style pull toward the previous checkpoint's stated numbers rather than toward resolved ground truth, so the model coherentises what it already believes.
Everything above assumes the model states a number and the environment reads it. If instead the probability of interest lived inside the model's sampling distribution, say p(A) = π(yes | "Is A true?"), the environment could never read it directly, only draw yes/no tokens from it. Any reward on tokens has an expectation that is linear in π per query, so a squared violation, which is quadratic, would need two independent draws of the same question multiplied together, and a hinge or KL term would need a Bernstein-style approximation whose accuracy grows with the number of draws. That is the regime Jessica Taylor's dice-enterprise bounds cover: which score functions can be realised as expected rewards of a fixed memoryless game, and how many queries it takes. With verbalised probabilities the compile step is trivial and only the game-theoretic conclusion carries over: memoryless queries make the episode an imperfect-recall game, its stable points under RL are CDT equilibria, and those coincide with the KKT points of the score.