Tellurion Referee: when a model disagrees with reality, say why.
The referee watches a learned world model as it runs. When its predictions stop matching the measurements, it decides whether the model is wrong, the instrument is wrong, or something new is acting on the system, and shows its evidence.
Result
Measured once, on a locked test of 1,000 simulated pendulum runs never seen during development.
| Metric | Referee | Model's confidence | Prediction gap |
|---|---|---|---|
| Real problems missed | 3.6% | 25.9% | 3.6% |
| Detection AUROC | 0.981 | 0.868 | 0.982 |
| False alarms, normal runs | 5.3% | 6.2% | 7.0% |
| Naming the cause (macro-F1) | 0.472 | 0.292 | 0.365 |
How it decides
- Does the prediction match the sensors? Yes: trust.
- Do the angle and speed sensors agree with each other? No: the instrument is wrong.
- Does textbook physics explain the motion? Yes: the model is wrong.
- Does it fit after re-measuring length and friction? Yes: the model is wrong, a setting changed. No: something new.
What doesn't work yet
The referee's alarm is the prediction gap, so at noticing problems it ties a simple gap alarm. Its added value is naming the cause: 99.7% right when the model is at fault, but 16.7% for sensor faults and 5.4% for new forces. It gives its verdict at the first alarm, when its window still holds mostly pre-fault data. Everything here is simulated.
Next
Judging a few steps after the alarm, real pendulum recordings with faults we introduce ourselves, and fits that describe the form of a missing force instead of only flagging it.