Failure case
Model
MsBERT + span-ft-refined
Real held-out lacuna spans for researcher-facing review, plus benchmark probes for diagnosis.
Researcher View
Real held-out lacuna spans with per-slot predictions under the span benchmark regime. Hover a prediction slot to highlight the context words the model attended to when filling it — final-layer attention saliency, in the spirit of DeepMind's Ithaca.
Unknown Gaps
Strong uncertainty and loss markers from the TF layer. No oracle is available for these cases.
Oracle-Known Triage
Use this when the gold word is known and you want to classify why the model failed. Hover the predictions to highlight which context words drove the model's guess.
Benchmark Snapshot & Model Achievements
Held-out Hebrew-only benchmark comparing decoding architectures and researcher-assist signals.
Corrected July 2026 Benchmark
Agreement With Attributed Researcher Restorations
Reconstruction-free MsBERT, evaluated on 74 genuine single-word lacunae from held-out, non-biblical scrolls. The decoder keeps visible manuscript letters and approximate lacuna-derived word length (±1), but never receives the restored letters.
63.5%
Target-level Top-10
Any compatible attributed restoration · N=74
60.6%
Unique-reading Top-10
99 distinct target-reading pairs
9.5%
Without physical constraints
Same targets; diagnostic comparison
The 63.5% result measures the complete constrained decoder—not a 54-point improvement in the language model. The former 8.0% row-weighted result is superseded because it discarded preserved letters, malformed some editorial alternatives, and overweighted repeatedly published targets. Target-level Top-10 95% CI: 51.4%–74.3%.
Key Breakthrough
Autoregressive Sequence-Level Restoration (Sequence Accuracy)
Traditional MLMs predict slots independently (Parallel Decoding), which ignores syntax constraints and causes duplicates like אשר אשר.
Our Autoregressive Beam Search decodes tokens left-to-right, conditioning each step on previous predictions.
This results in dramatic relative accuracy improvements for full, grammatically coherent sequence restoration (Sequence Top-1).
Top-10 Slot-Level Accuracy by Gap Length
Independent slot recovery accuracy (percentage of individual gap words correct in place).
Biblical Contrast Set (Control Group)
Evaluated on 60 biblical scroll fragments. Since biblical texts share high similarity with known canonical manuscripts, this serves as an upper-bound sanity check on the models' classical Hebrew language proficiency.