Calibration scorecard — model card

v1.0 · Tier 130d-warmup Brier + reliability tracker.

Purpose

Per Bible §1 Law 1 (named dimension = probability-first forecast + per-action playbook + cryptographically attested calibration) — this engine is the third leg. Every score / probability output gets logged with predicted distribution + realized outcome. Brier score computed weekly. Engines below 0.55 reliability flagged for re-weighting.

Inputs

Decision records from THV2DecisionTelemetry (R4)
Realized outcomes (price moves, regime transitions) lookup from THV2Store bars
Per-engine confidence values

Outputs

{
  perEngine: {
    'scoring':            { N: 1240, brier: 0.21, reliability: 0.78, status: 'ok' },
    'prescience-forecast':{ N:  890, brier: 0.18, reliability: 0.82, status: 'ok' },
    'playbook-action':    { N:  430, brier: 0.27, reliability: 0.61, status: 'warn' },
    'alerts-pipe':        { N:  680, brier: 0.24, reliability: 0.74, status: 'ok' }
  },
  perClass: { /* same shape but split by 14 asset classes */ },
  global: { brier: 0.22, reliability: 0.74, status: 'ok' },
  weeklyTrend: [ /* last 12 weeks */ ],
  alertThresholds: { brier: 0.30, reliability: 0.55 }
}

Math

Brier score: BS = (1/N) × Σ(pi − oi)² where pi = predicted probability, oi ∈ {0,1} = realized. Lower is better; perfect = 0; coin-flip = 0.25.

Reliability: 10-bin partition by predicted probability; within each bin, |mean(oi) − mean(pi)|. Reported as 1 − weighted-mean-deviation.

Limitations

What this is NOT:

Not yet populated with production data — needs ≥30 days of decision telemetry to surface meaningful Brier numbers
Not bias-corrected for regime — risk-on / risk-off / transition perform differently; per-class breakdown helps
Not survivorship-bias-aware — only tracks tickers present in today's universe

Source

engine/v104-calibration-scorecard.js + engine/calibrationLive.js + engine/decision-telemetry.js (R4)

Reviewed: 2026-04-27 · Next: 2026-07-27 · Per Bible §15.61 + §15.1