consistency-ft · original benchmark details

Model ranking · all 25 model / reasoning settings

GLM-5.3, Grok 4.6, GPT-6 Astra, and GPT-5.6 Luna at none, low, medium, high, xhigh, and max are included alongside every historical model. Scores measure probability consistency, not accuracy. 264 shared tuples across 24 participating settings, from the original 300-tuple benchmark. All native settings and API runs with complete original coverage participate; incomplete historical settings remain visible and unranked. No topic or semantic subset is applied here. Missing original readouts reduce the shared set; selection by shared availability can change the ordering. Native results use the original native-20260912 campaign (numeric-v1). Native subscription runs and historical API runs differ in dates, prompts and reasoning settings. Coverage and methods →

Jev uses a distinct probability channel: TypeSafe Noul returns a direct yes-probability for each question, independently, with its full resolution criteria. No shared question context, logical hints, prompt tuning, verbal answers or token logprobs. Reasoning controls and an explicit abstention field are not exposed. The same algebraic formulas and shared tuple IDs are used; this does not isolate model quality from elicitation method. Research and protocol →

*Historical API runs requested none, with low as a fallback; the setting actually used was not recorded per proposition. Subscription runs were measured in September and historical API runs in August, with different native prompts, reasoning settings, and output limits.

Coverage and comparison limits

Original target: 300 tuples across 10 checks, using 591 distinct propositions. The ranking compares the same 264 tuples scored by all 24 participating settings. Each model’s original coverage and mean over its own available tuples are preserved below; those means can have different denominators. Missing original readouts reduce the shared set. A shared-set headline does not imply completion of the full original benchmark. Native results are fixed to native-20260912 (numeric-v1). The corrected subset and rerun are reported separately. Unanswered probabilities remain missing, never zero. Low violation measures coherence, not forecasting accuracy or calibration.

Subscription usage

Across all archived campaigns: 61,738,874 recorded native tokens and 8,753 CLI launches, including setup and interruptions. 76 interrupted launches have no native receipt; native usage is unknown for 106 launches in total. Unknown consumption is not zero. Imported observations are counted once. Quota percentages are not summed across campaigns or accounts.

Quota changes below cover native-20260913-scoped-final: 2026-09-13T04:30:48.606308+00:00 to 2026-09-13T04:55:52.120664+00:00. Account-wide deltas can include other fleet sessions; window resets and provider accounting prevent inference from tokens alone. Download complete usage report.