LEARNING IN PUBLIC · EXPERIMENT 01
Can a probe catch
an arithmetic mistake?
A real model. A test set we held back. A result that needs more work. This is what learning with Dyno looks like.
Could a model's activations help flag numeric answers for external checking? We tested a small version of that question. The arithmetic questions are synthetic; the model answers and measurements are real. For addition, a calculator is still the right tool. Here, we're investigating an internal error signal.
The question and the setup
We generated 160 answers to three-digit addition questions and checked them with exact integer arithmetic. The first 112 questions trained the probe; the remaining 48 were held out. No question pair appeared in both splits. Format-pilot questions were excluded from the final dataset.
A logistic probe read layer 12's final input-token activation before the answer. A chat template and assistant equation prefix gave the small model a consistent answer format. Label 1 means an arithmetic error, and label 0 means a correct answer. Layer 12 and threshold 0.5 were fixed before fitting. There was no search for the best test score.
The result, including the uncomfortable part
AUROC 0.72, but accuracy below the majority baseline. There is some ranking signal in this sample. This operating threshold is not a dependable error detector.
| Measurement | Observed result |
|---|---|
| AUROC | 0.718 |
| Accuracy at threshold 0.5 | 64.6% |
| Always predict correct | 70.8% |
| Shuffled-label control accuracy | 64.6% |
| Errors caught / missed | 6 / 8 |
| False alarms | 9 |
AUROC measures ranking, not calibration. The score is not a calibrated error probability. A stratified bootstrap interval for AUROC was 0.563 to 0.849, using 5,000 resamples of this small test set. It does not measure variation across model choices or retraining.
One caught mistake. One missed mistake.
113 + 173: the model answered 386; the correct answer is 286. A probe score of 0.558 crossed the fixed threshold and flagged it.
288 + 361: the model answered 659; the correct answer is 649. A score of 0.498 did not cross the threshold. The probe missed the error.
These are the first held-out examples of each outcome, not the strongest examples selected for a demo. All answers and scores are in the download.
Try the experiment in Dyno
- Download
mlx-community/Qwen1.5-0.5B-Chat-4bitin Discover. - Open Lab → Experiments → Probes. Select Local MLX model and that model.
- Choose Import JSON and load the experiment settings below.
- Click Start lab & run experiment. Review the held-out metrics and controls together.
- Reopen the saved run in experiment history, or export it to share your evidence.
Download experiment JSONDataset, code, weights and results ZIPDownload video
The ZIP includes the Python SDK example, model revision, frozen protocol, generation script and result audit. The generic sentiment-sample description in the current app refers to its built-in example, not this imported dataset. The chart includes training and test examples; headline metrics use the test split only.
What this does not establish
This is not a reliable safety monitor, evidence of model awareness, or proof that a model uses a representation causally. It does not establish generalization to invoice wording, other operations, other models or unseen failure types. Input-only baselines and a larger, separately validated dataset are sensible next steps.
Inspired by Sun, Stolfo and Sachan, Probing for Arithmetic Errors in Language Models (EMNLP 2025). Their study explores arithmetic representations and error detection. Our smaller example uses a different model and Dyno's general logistic probe. It is not a replication of their method or scores.
Build, measure, share, repeat.
Dyno is an open-source workbench we're building to learn and make experiments easier to inspect. Honest negative results belong here too. Follow Danilo's learning-in-public series on Substack, explore the app and SDK, or see the Windows GPU worker.