HackMD
GT4HistOCR model evaluation
Since GT4HistOCR has been published, various OCR models have been trained on all or subsets of the data for various engines. This is an account of how well these models fare:
- Tesseract models trained at UB Mannheim – published 2019-2020
- Calamari models trained by Qurator team – published 2019-2020
- Calamari models trained by ZPD / Uni Würzburg … TODO
The original corpus had various flaws, which have been semi-automatically corrected by UB Mannheim. Also, there have been multiple full training runs by both teams.
Methodology
-
OCR Prediction
-
Tesseract: uses multiprocessing single-line (PSM13) standalone CLI tesserocr-batch
find GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | parallel -u -X -j1 tesseract-batch -Q4 -l GT4HistOCR_2000000 -x .gt4histocr_old find GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | parallel -u -X -j1 tesseract-batch -Q4 -l GT4HistOCR -x .gt4histocr -
Calamari: uses standalone CLI
calamari-predictfind GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | parallel -u -X -j1 calamari-predict --batch_size 8 -j 4 --extension .gt4histocr_cala.txt --extended_prediction_data --extended_prediction_data_format pred --checkpoint ~/.local/share/ocrd-resources/ocrd-calamari-recognize/qurator-gt4histocr-1.0/\* --files
-
-
OCR Evaluation
-
String alignment (Needleman-Wunsch), distance calculation (unweighted Levenshtein on graphemes, i.e. after joining combining characters), rate calculation (denominator is length of alignment path, not GT length or max-length) and aggregation (micro-averaging and parallel/subsample aggregation of stddev by Chan et al. 1979) is done by cor-asv-ann-compare
cor-asv-ann-compare -o compare-gt-gt4histocr.json -n Levenshtein -c 20 -F <(find GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | sed s/.[nb][ri][mn].png/.gt.txt/) <(find GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | sed s/.png/.gt4histocr.txt/) cor-asv-ann-compare -o compare-gt-gt4histocr_cala.json -n Levenshtein -c 20 -F <(find GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | sed s/.[nb][ri][mn].png/.gt.txt/) <(find GT4HistOCR -type f "(" -name "*.nrm.png" -or -name "*.bin.png" ")" | sed s/.[nb][ri][mn].png/.gt4histocr_cala.txt/) -
For generalization we should actually only look at unseen validation data (or at least the test data used to select the checkpoint). But for Calamari, which uses a voting scheme over 5 models from cross-fold training, no split is applicable. So we currently test on the full dataset in both cases. (But should at least train/test both in a similar way in the future.)
-
Notation
We will refer to the
- GT4HistOCR_2000000 Tesseract model from 2019-08-22 as the old model
- GT4HistOCR Tesseract model from 2020-02-11 as the new model
- GT4HistOCR Calamari v0.3 model from 2019-07-22 as the old model
- GT4HistOCR Calamari v1.0 model from 2019-12-11 as the new model
- ZPD model
- original corpus as the old data
- corrected corpus as the new data
Results
Tesseract: old model, old data
- full dataset: 0.7% with no
I/Jerror but lots of whitespace at EOL errors - test dataset: 0.7%, same as above
- test dataset,
dta19subcorpus: 0.5% - test dataset, other subcorpora: 1.3%
Tesseract: old model, new data
- full dataset: 0.9% with lots of
J/Ierrors - test dataset:
Tesseract: new model, old data
- full dataset: 1.3% with lots of
I/Jand whitespace at EOL errors - full dataset,
dta19subcorpus: 0.9% - full dataset, other subcorpora: 2.5% (but no
I/Jconfusion there) - test dataset: 1.4%, same as above
- test dataset,
dta19subcorpus: 1.2% - test dataset, other subcorpora: 2.2%
Tesseract: new model, new data
- full dataset: 1.1% with lots of
r/ꝛerrors (because the model was trained before the respective corrections?) - test dataset:
Calamari: old model, old data
TODO
Calamari: new model, old data
- full dataset: 0.8% partly attributable to overaggressive normalization during training (
calamari-train --data.pre_proc.processors.6.replacement_groupsdefaults toextendedwhich among others rewrites double quotes to apostrophes and commas; for GT level 2, onlyspaces+roman_digits+ligatures-consonantalwould be appropriate) - When evaluating with stronger normalization (including Calamari's
quotesregularizer): 0.4% – quoting errors are gone
Calamari: new model, new data
- full dataset: 1.1% as above, but in addition lots of
J/Ierrors - When evaluating with stronger normalization (including Calamari's
quotesregularizer): 0.6% as above, but in addition lots ofJ/Ierrors
Calamari: ZPD model, old data
- full dataset: 0.8%, like new model above, but without normalization errors (the ambiguity around
⸗/—and⸗/-arises from inconsistencies between the subcorpora themselves)
Calamari: ZPD model, new data
Discussion
Transcription/normalization/regularization rules are extremely important in this high-accuracy regime.
…
