Evaluating OCR programs that convert PDFs or doc photographs into Markdown is much extra complicated than it seems. In contrast to plain textual content OCR, OCR-to-Markdown requires fashions to recuperate content material, format, studying order, and illustration decisions concurrently. Immediately’s benchmarks try to attain this with a mixture of string matching, heuristic alignment, and format-specific guidelines—however in apply, these approaches routinely misclassify appropriate outputs as failures.
This put up outlines why OCR-to-Markdown analysis is inherently underspecified, examines widespread analysis methods and their failure modes, highlights concrete points noticed in two extensively used benchmarks, and explains why LLM-as-judge is presently probably the most sensible option to consider these programs—regardless of its imperfections .
Why OCR-to-Markdown Is Laborious to Consider
At its core, OCR-to-Markdown doesn’t have a single appropriate output.
A number of outputs will be equally legitimate:
- Multi-column layouts will be linearized in numerous studying orders.
- Equations will be represented utilizing LaTeX, Unicode, HTML, or hybrids.
- Headers, footers, watermarks, and marginal textual content could or might not be thought-about “content material” relying on process intent.
- Spacing, punctuation, and Unicode normalization typically differ with out affecting that means.
From a human or downstream-system perspective, these outputs are equal. From a benchmark’s perspective, they typically are usually not.
Widespread Analysis Strategies and Their Limitations
1. String-Based mostly Metrics (Edit Distance, Precise Match)
Most OCR-to-Markdown benchmarks depend on normalized string comparability or edit distance.
Limitations
- Markdown is handled as a flat character sequence, ignoring construction.
- Minor formatting variations produce massive penalties.
- Structurally incorrect outputs can rating nicely if textual content overlaps.
- Scores correlate poorly with human judgment.
These metrics reward formatting compliance quite than correctness.
2. Order-Delicate Block Matching
Some benchmarks phase paperwork into blocks and rating ordering and proximity.
Limitations
- Legitimate various studying orders (e.g., multi-column paperwork) are penalized.
- Small footer or marginal textual content can break strict ordering constraints.
- Matching heuristics degrade quickly as format complexity will increase.
Right content material is commonly marked fallacious as a consequence of ordering assumptions.
3. Equation Matching by way of LaTeX Normalization
Math-heavy benchmarks sometimes count on equations to be rendered as full LaTeX.
Limitations
- Unicode or partially rendered equations are penalized.
- Equal LaTeX expressions utilizing completely different macros fail to match.
- Blended LaTeX/Markdown/HTML representations are usually not dealt with.
- Rendering-correct equations nonetheless fail string-level checks.
This conflates illustration alternative with mathematical correctness.
4. Format-Particular Assumptions
Benchmarks implicitly encode a most well-liked output type.
Limitations
- HTML tags (e.g.,
) trigger matching failures. - Unicode symbols (e.g.,
km²) are penalized towards LaTeX equivalents. - Spacing and punctuation inconsistencies in floor reality amplify errors.
Fashions aligned to benchmark formatting outperform extra basic OCR programs.
Points Noticed in Current Benchmarks
Benchmark A: olmOCRBench
Guide inspection reveals that a number of subsets embed implicit content material omission guidelines:
- Headers, footers, and watermarks which can be visibly current in paperwork are explicitly marked as absent in floor reality.
- Fashions skilled to extract all seen textual content are penalized for being appropriate.
- These subsets successfully consider selective suppression, not OCR high quality.
Moreover:
- Math-heavy subsets fail when equations are usually not absolutely normalized LaTeX.
- Right predictions are penalized as a consequence of illustration variations.
Because of this, scores strongly rely upon whether or not a mannequin’s output philosophy matches the benchmark’s hidden assumptions.
Instance 1
For the above picture, Nanonets-OCR2 accurately predicts the watermark to the correct facet of the picture, however within the floor reality annotation penalizes the mannequin for predicting it accurately.
{
"pdf": "headers_footers/ef5e1f5960b9f865c8257f9ce4ff152a13a2559c_page_26.pdf",
"web page": 1,
"id": "ef5e1f5960b9f865c8257f9ce4ff152a13a2559c_page_26.pdf_manual_01",
"sort": "absent",
"textual content": "Doc tu00e9lu00e9chargu00e9 depuis www.cairn.data - Universitu00e9 de Marne-la-Vallu00e9e - - 193.50.159.70 - 20/03/2014 09h07. u00a9 S.A.C.", "case_sensitive": false, "max_diffs": 3, "checked": "verified", "first_n": null, "last_n": null, "url": ""}
Kind absent implies that within the prediction knowledge, that textual content shouldn’t be current.
Instance 2
The benchmark additionally doesn’t take into account texts which can be current within the doc footer.
Instance on this doc, the Alcoholics Namelessu00ae and www.aa.org shouldn’t be current within the doc in accordance with the ground-truth, which is inaccurate
{
"pdf": "headers_footers/3754542bf828b42b268defe21db8526945928834_page_4.pdf",
"web page": 1,
"id": "3754542bf828b42b268defe21db8526945928834_page_4_header_00",
"sort": "absent",
"max_diffs": 0,
"checked": "verified",
"url": "",
"textual content": "Alcoholics Namelessu00ae",
"case_sensitive": false, "first_n": null, "last_n": null
}
{
"pdf": "headers_footers/3754542bf828b42b268defe21db8526945928834_page_4.pdf",
"web page": 1,
"id": "3754542bf828b42b268defe21db8526945928834_page_4_header_01",
"sort": "absent",
"max_diffs": 0,
"checked": "verified",
"url": "",
"textual content": "www.aa.org",
"case_sensitive": false, "first_n": null, "last_n": null}
Benchmark B: OmniDocBench
OmniDocBench reveals related points, however extra broadly:
- Equation analysis depends on strict LaTeX string equivalence.
- Semantically an identical equations fail as a consequence of macro, spacing, or image variations.
- Quite a few ground-truth annotation errors have been noticed (lacking tokens, malformed math, incorrect spacing).
- Unicode normalization and spacing variations systematically cut back scores.
- Prediction choice heuristics can fail even when the right reply is absolutely current.
In lots of instances, low scores replicate benchmark artifacts, not mannequin errors.
Instance 1
Within the instance above, the Nanonets-OCR2-3B predicts 5 g silica + 3 g Al$_2$O$_3$ however the floor reality expects as $ 5g mathrm{ s i l i c a}+3g mathrm{ A l}*{2} mathrm{O*{3}} $ . This flags the mannequin prediction as incorrect, even when each are appropriate.
Full Floor Reality and Prediction, and the take a look at case shared under:
'pred': 'The collected eluant was concentrated by rotary evaporator to 1 ml. The extracts have been lastly handed via a closing column stuffed with 5 g silica + 3 g Al$_2$O$_3$ to take away any co-extractive compounds that will trigger instrumental interferences durin the evaluation. The extract was eluted with 120 ml of DCM:n-hexane (1:1), the primary 18 ml of eluent was discarded and the remainder have been collected, which comprises the analytes of curiosity. The extract was exchanged into n-hexane, concentrated to 1 ml to which 1 μg/ml of inner customary was added.'
'gt': 'The collected eluant was concentrated by rotary evaporator to 1 ml .The extracts have been lastly handed via a closing column stuffed with $ 5g mathrm{ s i l i c a}+3g mathrm{ A l}*{2} mathrm{O*{3}} $ to take away any co-extractive compounds that will trigger instrumental
interferences in the course of the evaluation. The extract was eluted with 120 ml of DCM:n-hexane (1:1), the primary 18 ml of eluent was discarded and the remainder have been collected, which comprises the analytes of curiosity. The extract was exchanged into n - hexane, concentrated to 1 ml to which $ mumathrm{g / ml} $ of inner customary was added.'Instance 2
We discovered considerably extra incorrect annotations with OmniDocBench
Within the ground-truth annotation 1 is lacking in 1 ml .
'textual content': 'The collected eluant was concentrated by rotary evaporator to 1 ml .The extracts have been lastly handed via a closing column stuffed with $ 5g mathrm{ s i l i c a}+3g mathrm{ A l}*{2} mathrm{O*{3}} $ to take away any co-extractive compounds that will trigger instrumental interferences in the course of the evaluation. The extract was eluted with 120 ml of DCM:n-hexane (1:1), the primary 18 ml of eluent was discarded and the remainder have been collected, which comprises the analytes of curiosity. The extract was exchanged into n - hexane, concentrated to 1 ml to which $ mumathrm{g / ml} $ of inner customary was added.'
Why LLM-as-Choose Is the Least-Dangerous Possibility Immediately
Given these limitations, LLM-as-judge is presently probably the most sensible option to consider OCR-to-Markdown programs.
This isn’t as a result of LLM judges are excellent—however as a result of the issue is essentially semantic.
What LLM-as-Choose Handles Properly
- Semantic Equivalence Throughout Representations
LLMs can acknowledge that:- LaTeX, Unicode, and HTML equations will be equal
- Macro-level variations (
A^Tvsmathbf{A}^T) don’t change that means - Spacing and normalization variations are irrelevant
- Versatile Studying Order Reasoning
LLMs can assess whether or not content material is full even when:- Sections are reordered
- Multi-column layouts are linearized otherwise
- Context-Conscious Content material Inclusion
LLMs can purpose about whether or not:- Footers, headers, or watermarks ought to moderately be included
- Textual content inside logos or figures counts as content material
- Tolerance to Annotation Noise
When floor reality is incomplete or incorrect, LLMs can nonetheless choose correctness relative to the doc, quite than blindly implementing flawed annotations.
Why Metric Engineering Doesn’t Scale
Many benchmark failures are addressed by:
- Including normalization guidelines
- Increasing equivalence courses
- Introducing heuristic margins
These fixes don’t generalize. Each new doc sort—scientific papers, scanned books, multilingual PDFs, kinds—introduces new edge instances. LLMs generalize throughout these instances with out task-specific rule engineering.
Acknowledged Limitations of LLM-as-Choose
LLM-based analysis has actual drawbacks:
- Non-determinism
- Sensitivity to immediate design
- Increased price and latency
- Lowered reproducibility in comparison with static scripts
Nevertheless, these are operational limitations, not conceptual ones. In distinction, string- and rule-based metrics are conceptually misaligned with the duty itself.
Ultimate Takeaway
OCR-to-Markdown analysis is underspecified by nature. Current benchmarks conflate formatting, illustration decisions, and semantic correctness—typically penalizing fashions for being appropriate in methods the benchmark didn’t anticipate.
Till benchmarks explicitly embrace semantic equivalence, LLM-as-judge stays the closest approximation to human judgment and probably the most dependable analysis sign accessible right now. Benchmark scores ought to subsequently be handled as partial indicators, not definitive measures of OCR high quality.
