Failure case
Model
MsBERT + span-ft-refined
Real held-out lacuna spans for researcher-facing review, plus benchmark probes for diagnosis.
Researcher View
Real held-out lacuna spans with per-slot predictions under the span benchmark regime. Hover a prediction slot to highlight the context words the model attended to when filling it — final-layer attention saliency, in the spirit of DeepMind's Ithaca.
Unknown Gaps
Strong uncertainty and loss markers from the TF layer. No oracle is available for these cases.
Oracle-Known Triage
Use this when the gold word is known and you want to classify why the model failed. Hover the predictions to highlight which context words drove the model's guess.
Benchmark Snapshot & Model Achievements
Held-out Hebrew-only benchmark comparing decoding architectures and researcher-assist signals.
Corrected July 2026 Benchmark
Agreement With Attributed Researcher Restorations
Reconstruction-free MsBERT, evaluated on 74 genuine single-word lacunae from held-out, non-biblical scrolls. The decoder keeps visible manuscript letters and approximate lacuna-derived word length (±1), but never receives the restored letters.
63.5%
Target-level Top-10
Any compatible attributed restoration · N=74
60.6%
Unique-reading Top-10
99 distinct target-reading pairs
9.5%
Without physical constraints
Same targets; diagnostic comparison
The 63.5% result measures the complete constrained decoder—not a 54-point improvement in the language model. The former 8.0% row-weighted result is superseded because it discarded preserved letters, malformed some editorial alternatives, and overweighted repeatedly published targets. Target-level Top-10 95% CI: 51.4%–74.3%.
Train-only RAG ablation
Exact-context retrieval uses only preserved words from non-biblical training scrolls. Its weight (α=0.5) was selected on dev scrolls, never on held-out targets.
| Held-out evaluation | Unit | MLM Top-10 | MLM + RAG Top-10 |
|---|---|---|---|
| Qumran Digital attributed readings | 74 single-word targets | 63.5% | 63.5% |
| Text-Fabric editorial labels | 25 single-word spans | 60.0% | 64.0% |
| Text-Fabric editorial labels | 440 slots in 100 multiword spans | 41.4% | 41.8% |
| Text-Fabric exact sequence | 100 multiword spans | 7.0% | 9.0% |
The Text-Fabric reconstructions are anonymous editorial evaluation labels, not physical truth. The balanced sample contains 25 spans in each length bucket; whole-sequence recovery requires every word to match in order. These modest gains are reported as an ablation, not as a universal RAG improvement.
Agreement by bibliographic source
Each source contributes at most one observation per manuscript target.
| Publication source | Independent targets | Top-1 | Top-10 |
|---|---|---|---|
| Study Edition | 24 | 20.8% | 62.5% |
| Qimron 2013 | 23 | 30.4% | 52.2% |
| PrCon I | 10 | 40.0% | 50.0% |
| Qimron 2020 | 9 | 44.4% | 66.7% |
| Wacholder/Abegg 1995 | 9 | 33.3% | 55.6% |
| DJD XXIX | 8 | 37.5% | 50.0% |
| Qimron 2014 | 8 | 50.0% | 75.0% |
“Source” means a bibliographic publication, sometimes with multiple authors—not an independent individual researcher. Rows below 10 targets are shaded and should not be interpreted as a ranking. When several publications propose the same reading at one target, that reading counts once in the headline metric.
Key Breakthrough
Autoregressive Sequence-Level Restoration (Sequence Accuracy)
Traditional MLMs predict slots independently (Parallel Decoding), which ignores syntax constraints and causes duplicates like אשר אשר.
Our Autoregressive Beam Search decodes tokens left-to-right, conditioning each step on previous predictions.
This results in dramatic relative accuracy improvements for full, grammatically coherent sequence restoration (Sequence Top-1).
Top-10 Slot-Level Accuracy by Gap Length
Independent slot recovery accuracy (percentage of individual gap words correct in place).
Biblical Contrast Set (Control Group)
Evaluated on 60 biblical scroll fragments. Since biblical texts share high similarity with known canonical manuscripts, this serves as an upper-bound sanity check on the models' classical Hebrew language proficiency.