← All resultsA SEPARATE TEST OF THREE METHODS 419 / 600Graded comparisons 31 / 60UTC event dates 4Awaiting an eligible result 0Extra AI calls per comparison
Which method adds value?
Current TORK, a simple four-model average, and a calibrated average use the same events and saved information. Each is compared with Kalshi at the forecast cutoff.
Collecting new evidence. The current TORK decision rule remains in use.
What we are testing
Current TORK
The existing published decision. Its method and the original registered experiment are unchanged.
Simple average
Convert all four forecasts to the same outcome, then average their probabilities. Each model has equal weight.
Calibrated average
Apply one fixed correction learned from 153 earlier results across 14 dates. The correction is frozen before this test begins.
Correction factor: 1.0934. A factor of 1 leaves the average unchanged.
Kalshi benchmark
An exact matched price, already stored at the TORK lock and no more than 15 minutes old. The model average does not use this price as an input.
Same events. Same cutoff.
All methods must be recorded before the event starts. Late, missing, changed-version, or ambiguous inputs are excluded. Earlier forecasts cannot enter the new cohort.
Prospective scores remain hidden until at least 600 graded comparisons across 60 dates and completion of the full stopping day and all eligible outcomes through that day. This is an analysis checkpoint, not proof of adequate statistical power or a winning method.
Lower Brier and log loss scores mean better probability estimates. No early winner is declared.
Model contribution and calibration
For each model, we also save the average with that model removed. A higher Brier score after removal means that model helped. These comparisons use the same events and are exploratory.
Results will appear at the analysis checkpoint.
Coverage and exclusions
423 comparisons saved. 0 events have not started.
Saved comparisons by sport
MLB: 81
MMA: 17
NFL: 30
TENNIS: 295
Missed or excluded after start
ambiguous outcome mapping: 7
checkpoint timing mismatch: 4
missing kalshi lock: 581
no directional forecast: 181
outcome already known: 1
stale or late market: 15
The methods reuse saved model reports. Original model-generation cost is not recorded in this cohort; no dollar cost is inferred.
Why Ensemble v2 remains a challenger
Historical results available at registration, matched on the same model inputs and official outcomes. These are development evidence. They are not results from this new test.
| Sport | Matched events | Current TORK Brier | Ensemble v2 Brier |
|---|---|---|---|
| BOXING | 9 | 0.1979 | 0.2128 |
| MLB | 124 | 0.2325 | 0.2332 |
| MMA | 20 | 0.1890 | 0.2005 |
Lower is better. Small samples and different sports limit what these averages can establish.