Independent verification and calibration for AI systems

Avenridge Institute conducted a pre-registered calibration evaluation of the publicly available model:

• distilbert-base-uncased-finetuned-sst-2-english
• 
  

using the SST-2 validation dataset.

The objective was not to evaluate raw classification accuracy alone, but to evaluate whether the model's probability outputs were statistically calibrated relative to observed outcomes.

The evaluation protocol was locked before execution using a documented pre-registration methodology.

The model achieved:

• Accuracy: 91.06%
• Brier Skill Score: +0.6664

However:

the model failed the pre-registered calibration criterion.

The strongest observed pattern was bimodal overconfidence:

• Predictions near 0% confidence were substantially more wrong than implied by the model probabilities.
• Predictions near 100% confidence were also materially overconfident.

The evaluation demonstrated that:

strong benchmark accuracy does not necessarily imply reliable probabilistic calibration.