consistency-ft · original benchmark

For the historical Fireworks elicitation runs only, each proposition (title + resolution criteria) is asked twice, at temperature 0, reasoning_effort: "none" (falling back to "low" where a model rejects "none"), max_tokens: 2048.

1 · Verbalized probability

Estimate the probability that the proposition below is true.
Return only a number from 0.0 to 1.0, with no commentary.

Proposition:
{proposition}

The reply is parsed as a number in [0,1].

2 · True/False logprob probability

Decide whether the proposition below is true. Answer with exactly one word.

Proposition:
{proposition}

True or False:

Token logprobs (top_logprobs: 5) are scanned from the end of the completion (reasoning models mention true/false mid-analysis; the answer token is last). At the last position where both appear: P = exp(lp_True) / (exp(lp_True) + exp(lp_False)).

mean |verbal−logprob| = mean absolute gap between the two readouts over the same propositions; disagreement >0.2 = fraction with gap above 0.2; Pearson over the paired values. Historical consistency-check probabilities elsewhere on this page use readout 1 only; Jev uses its direct Noul probability.