Can generative AI edit ancient manuscripts? Over the past two years, claims that frontier multimodal models can “read historical manuscripts in seconds” have proliferated across tech media. For textual scholars, critical editors, and digital humanists, these claims demand rigorous, empirical evaluation rather than uncritical adoption or dismissive skepticism.
Preparing a primary textual witness for scholarly publication is a demanding, multi-phase endeavor. A digital scholarly edition requires far more than translation: it demands diplomatic transcription from damaged surfaces, multi-witness collation, constructing a formal critical apparatus, lemmatization and morphosyntactic parsing, arbitrating epigraphic lacunae, and encoding the entire apparatus into schema-valid TEI EpiDoc XML.
Under our SSHRC Insight Development Grant (Generative AI in Digital Humanities Research Methodology), our research team at St. Francis Xavier University developed a comprehensive benchmark suite to test 31 frontier and open-weights Large Language Models across 13 distinct scholarly editing tasks in Medieval Latin, Ancient Greek, and Dead Sea Scrolls Aramaic.
Here is what we learned from 400+ dual-blind graded evaluations, latency audits, and cost-efficiency trade-offs.
1. The Benchmark Architecture: 13 Tasks Across 3 Corpora
Rather than evaluating generic, out-of-context translations, our benchmark tests models across the entire end-to-end digital scholarly editing pipeline using three primary historical witnesses representing distinct scripts and preservation states:
- Medieval Latin Bookhand (Isidore of Seville, De ortu et obitu Patrum, Vat. lat. 629, f. 3v): Pristine 10th-century Caroline/early Gothic minuscule in two columns with standard scribal abbreviations, ligatures, and critical collation against Chaparro Gómez (1985).
- Continuous Greek Majuscule (Codex Marchalianus, Vat. gr. 2125, p. 11): 6th-century Egyptian uncial written in scriptio continua with lunate sigmas (
ϲ), Nomina Sacra contractions under supralinear strokes, and collation against Schermann’s 1907 Vitae Prophetarum. - Fragmentary Qumran Aramaic (Book of Giants, 4Q530 / 4Q531 / 6Q8): 1st-century BCE/CE Jewish Aramaic formal square script on darkened, lacunose leather. Tasks include consonantal transcription with DJD sigla, multi-fragment alignment, verbal stem parsing (Peal/Pael), and lacuna-disciplined translation.
Each model was evaluated under strict philological negative constraints at temperature zero and scored across a 0–100 scale by an automated dual-blind judging panel (MiniMax M3 and Grok 4.6).
2. Benchmark Results & Visualizations



🖱️ Interactive 3D View: Left-click and drag with your mouse to rotate the viewpoint. Right-click and drag to pan. Scroll to zoom. Hover over any node to view exact model scores, latency, and costs.
document.addEventListener(“DOMContentLoaded”, function() { const data = [ { type: “scatter3d”, mode: “markers+text”, x: [45.0, 88.0, 32.0, 181.0, 12.0, 2.8, 38.0, 82.0, 8.5, 1.6, 97.6, 75.0, 55.0, 18.5, 42.0, 22.0, 6.5, 48.0, 65.0, 6.5, 28.0, 211.9, 48.5, 95.0, 24.5, 35.0, 25.0, 40.0, 16.0, 45.0], y: [0.0169, 0.0188, 0.0065, 0.0392, 0.0016, 0.0009, 0.0026, 0.0191, 0.0005, 0.0008, 0.0221, 0.0162, 0.0142, 0.0035, 0.0146, 0.0135, 0.0009, 0.0115, 0.0127, 0.0001, 0.0012, 0.0026, 0.0139, 0.0081, 0.0020, 0.0026, 0.0008, 0.0085, 0.00001, 0.0014], z: [84.0, 79.8, 76.7, 58.1, 58.7, 82.1, 74.6, 72.9, 67.9, 61.7, 75.0, 78.1, 79.9, 77.8, 74.9, 61.0, 46.0, 79.1, 75.8, 70.2, 70.3, 64.5, 77.6, 58.0, 72.1, 58.7, 50.4, 60.1, 49.6, 60.7], text: [“Claude Opus 4.7”, “Claude Opus 4.8”, “Claude Sonnet 4.5”, “Claude Opus 5”, “Claude 3 Haiku”, “Gemini 3.7 Flash”, “Gemini 2.5 Pro”, “Gemini 3.1 Pro”, “Gemini 2.5 Flash”, “Gemma 4 31B”, “GPT-5”, “GPT-5.5”, “GPT-5.6 Terra”, “GPT-5.6 Luna”, “GPT-5.6 Sol”, “GPT-4o”, “GPT-4o mini”, “Grok 4.5”, “Grok 4.6”, “DeepSeek v4 Flash”, “DeepSeek v3.2”, “DeepSeek R1”, “Qwen 3.8 Max”, “Qwen 3 VL (th)”, “Muse-Spark 1.2”, “Llama 4 Maverick”, “Llama 3.3 70B”, “Mistral Large”, “Mistral Small”, “MiniMax M3”], textposition: “top center”, textfont: { size: 9, color: “#1e293b” }, marker: { size: 7, color: [“#7c3aed”,”#7c3aed”,”#7c3aed”,”#7c3aed”,”#7c3aed”,”#2563eb”,”#2563eb”,”#2563eb”,”#2563eb”,”#2563eb”,”#059669″,”#059669″,”#059669″,”#059669″,”#059669″,”#059669″,”#059669″,”#dc2626″,”#dc2626″,”#0891b2″,”#0891b2″,”#0891b2″,”#d97706″,”#d97706″,”#ea580c”,”#ea580c”,”#ea580c”,”#4f46e5″,”#4f46e5″,”#64748b”], opacity: 0.9, line: { width: 1, color: “white” } }, hovertemplate: “%{text}🏆 Quality Score: %{z:.1f}/100
⏱️ Avg Latency: %{x:.1f} s/task
💰 Avg Cost: $%{y:.4f}/task” } ]; const layout = { title: { text: “Interactive 3D Frontier: Score vs. Latency vs. Cost (31 Models)“, font: { size: 16 } }, scene: { xaxis: { title: “Avg Latency (s)”, backgroundcolor: “#f8fafc”, gridcolor: “#e2e8f0” }, yaxis: { title: “Avg Cost ($/Task)”, backgroundcolor: “#f8fafc”, gridcolor: “#e2e8f0” }, zaxis: { title: “Score (0–100)”, backgroundcolor: “#f8fafc”, gridcolor: “#e2e8f0” }, camera: { eye: { x: -1.7, y: -1.7, z: 1.3 } } }, margin: { l: 0, r: 0, b: 0, t: 40 }, paper_bgcolor: “#ffffff” }; Plotly.newPlot(“plotly-3d-benchmark”, data, layout, { responsive: true, displayModeBar: true }); });
3. Key Finding #1: The Workflow Difficulty Hierarchy
When analyzing performance across the 403 individual task evaluations, the workflow phases formed a steep, definitive difficulty hierarchy:
WORKFLOW PHASE DIFFICULTY HIERARCHY (0–100 Mean Score)
Translation 57.7 ──────────────────────────────
Transcription 48.9 ────────────────────
Lacuna Handling 45.9 ──────────────────
Annotation / MSD 39.0 ──────────────
TEI P5 XML Encoding 37.6 ─────────────
Multi-Witness Collation 32.6 ──────────
- Translation (57.7/100): Modern LLMs perform relatively well on literary translation from Greek and Latin, leveraging vast pre-training corpora.
- The Collation Bottleneck (32.6/100): Constructing a formal critical apparatus (
[lemma] reading Siglum ;) remains the steepest hurdle for AI. Models frequently hallucinate non-existent variants or fail to maintain strict lemma-bracket discipline across multiple manuscript witnesses. - TEI XML Metadata Failure (37.6/100): Models dropped precipitously when tasked with emitting schema-valid TEI P5 EpiDoc XML. They routinely hallucinate modern catalog schemas, invert nested tags, or fail XML validation, proving that structured digital encoding remains an active research challenge.
4. Key Finding #2: The Script & Damage Gradient
In our multimodal vision transcription benchmarks, character error rates (CER) did not correlate primarily with language family or script complexity. Rather, model performance mapped directly onto physical manuscript preservation and lacunosity:
THE SCRIPT & DAMAGE GRADIENT
0.0 ────────────────────────────────────────────────────────────────────────► 1.0 CER
Latin Minuscule (0.04) Greek Majuscule (0.12) Fragmentary Aramaic (0.75)
• Intact parchment • Faded ink, broken edges • Lacunose, darkened, cracked
• Clear word division • Scriptio continua • Carbon ink invisible in colour
• High training volume • Nomina Sacra contractions • Unseen fragmentary text
- Intact Latin Bookhand (CER 0.042 / WER 0.254): On clean, continuous parchment, frontier models (Gemini 3.1 Pro, Gemini 3.7 Flash) achieve single-digit error rates in under 40 seconds. At this tier, automated transcription is already an operational reality for digital scholarly editing.
- Greek Majuscule with Scriptio Continua (CER 0.121 / WER 0.281): Models like Claude Opus 4.8 and Grok 4.6 function effectively as primary drafting aids, requiring human collation to catch uncial diagonal confusions (
Δvs.Λ). - Fragmentary Dead Sea Scrolls (CER 0.754 / WER 0.980): Every model collapsed. With three out of four characters misidentified and word error rates near 100%, generative vision models are completely unusable for damaged epigraphic fragments.
5. Key Finding #3: Model Specializations & The Pareto Frontier
As illustrated in the heatmap (Figure 2) and 3D efficiency chart (Figure 3), no single model dominates every phase of digital editing:
- Anthropic (Claude Opus 4.7, 4.8 & 5): Dominates hermeneutics, literary translation, and lacuna restraint (scoring up to 93.2/100 on recognizing when damaged parchment makes textual certainty impossible).
- xAI (Grok 4.6 & 4.5): Demonstrates superior typographic precision and layout alignment in critical apparatus construction.
- Google (Gemini 3.7 Flash) & OpenAI (GPT-5 series): Lead in high-speed morphosyntactic parsing, grammatical tagging, and structured XML generation.
- The Cost-Efficiency Champions: Models like Gemini 3.7 Flash and DeepSeek v4 Flash achieve top-tier accuracy while reducing per-task costs by 95% compared to legacy frontier models (operating for fractions of a cent per page).
6. Methodological Pitfalls Every Digital Humanist Must Know
Our benchmark uncovered three systemic traps that researchers must account for when deploying AI in textual scholarship:
1. The Allograph Trap: Metrics Embody Editorial Theory
When we first evaluated Claude Opus 4.8 on Latin minuscule using raw string matching, it received a poor CER of 0.188. Detailed inspection revealed why: Claude had faithfully transcribed historical scribal allographs—long-s (ſ), r rotunda (ꝛ), and dotless-i (ı). When we applied palaeographical folding (LATIN_FOLD)—collapsing allographs to modern base characters—Claude’s error rate plummeted from 0.188 to 0.053. Naive string-matching metrics actively punish the most paleographically faithful models while rewarding models that silently modernize.
2. The Abstention Inversion: Hallucinations on Blank Leather
When seven models were fed visible-light colour images of Dead Sea Scroll fragments containing no visible ancient ink (readable only under infrared), Claude Opus 4.8 and GPT-4o correctly abstained by returning [...] or declaring the image illegible. In contrast, Gemini 3.1 Pro hallucinated four lines of fluent Aramaic on every blank fragment, inventing plausible paleographical arguments for ink strokes that did not exist. Standard CER metrics perversely ranked the hallucinating model at the top of the leaderboard because it produced text. Abstention Rate must be reported as an independent metric.
3. Commercial Safety Guardrails on Sacred Texts
Commercial safety filters represent a severe infrastructure vulnerability for religious studies. Gemini 3.7 Flash suffered a 100% safety block (PROHIBITED_CONTENT) on the 6th-century Greek martyrdom of Isaiah (being sawn in two under Manasseh), while Claude Opus 5 returned a 100% content-filter refusal across all 12 Aramaic Book of Giants fragments. Digital humanists cannot rely exclusively on closed commercial APIs whose safety filters classify historical religious literature as prohibited violent content.
7. The Methodological Blueprint: Human-in-the-Loop
Rather than viewing AI as an autonomous editor or a forbidden tool, digital scholarly editing requires a Human-in-the-Loop (HITL) framework where human expertise and algorithmic capability form a synergistic workflow:
- Tier 1 — High-Speed Parsing: Deploy lightweight, cost-effective models (e.g., Gemini 3.7 Flash) for initial morphological tagging and transcription drafting of intact bookhands.
- Tier 2 — Critical Synthesis: Deploy disciplined reasoning models (e.g., Claude Opus, Grok 4.6) with strict negative constraint prompts for multi-witness alignment and apparatus synthesis.
- Tier 3 — Human Curation & Validation: The text critic serves as the indispensable curator—arbitrating epigraphic lacunae, correcting uncial confusions, and validating schema compliance against master TEI EpiDoc standards.
In the digital age, our task is not to surrender the archive to the algorithm, but to use the algorithm to look more closely at the archive.
This research is supported by the Social Sciences and Humanities Research Council of Canada (SSHRC) Insight Development Grant (“Generative AI in Digital Humanities Research Methodology”, Fund R0253025). Benchmark datasets, prompt libraries, and scoring scripts are maintained in the project repository. Special thanks to Lead Undergraduate Research Assistant Demarquis Moss for his contributions to the ground-truth collation and benchmark audits.







