Flexible OCR workflows with OCR-D

Ocropy-based processors

• Many OCR-related operations available
  • Not cleanly wrapped
  • Not separately available
• Creating processors for
  • Binarization (on page/region/line level)
  • Deskewing (on page/region level)
  • Dewarping (on line level)
  • Clipping (on region/line level)
  • Resegmentation (on line level)
• Ultimate goal: Ocropy as an API
  • Code clean-up and improvement …
  • New operations …
  • Restructuring into ocrolib and processors …

Ocropy-based processors

Code clean-up and improvement

• move common Ocropy functions into common (from CLIs, from OLD/), add new ones:
  • PIL.Image vs np.ndarray conversions: array2pil, pil2array (integer vs normed float)
  • plausibility checks: check_page, check_region, check_line (but mix absolute and relative bounds, and make DPI-zoomable)
  • nlbin deskewing: estimate_skew_angle, estimate_skew (but resize when rotating, with minimum on variance drop)
  • nlbin binarization: estimate_local_whitelevel, estimate_thresholds, binarize (but keep exact pixel size, catch NaN)
  • remove connected components only contained in the margins: borderclean

Ocropy-based processors

Code clean-up and improvement

• disect and improve segmentation:
  • remove_hlines: add height threshold, reduce default width threshold
  • compute_separators_morph: reduce thresholds (because black colseps can be discontinuous), use only connected components fully inside zone
  • compute_gradmaps: reduce boxmap minsize (for chopped lines at margins), reduce horizontal blur (to avoid joining lines via as-/descenders)
  • compute_line_seeds: more robust top/bottom projection rules and no horizontal blur (to avoid joining lines via as-/descenders)
  • hmerge_line_seeds: new way to ensure horizontal label consistency
  • compute_segmentation:
    • fullpage switch (regions do not have hlines and colseps)
    • zoom parameter (thresholds must be DPI-relative)
    • use twice the estimated scale (blackletter has huge capitals and dense as-/descenders – avoid splitting lines)
    • before spreading line seeds, assign unlabelled connected components to their majority seed (instead of splitting)

Ocropy-based processors

New operation: Resegmentation

observation 1:

Ocropy dewarping and recognition is very sensitive to connected components intruding from neighbouring lines (e.g. ascenders and descenders)

observation 2:

GT line segmentation is very coarse (only bounding boxes, large overlap)

idea:

use Ocropy line segmentation to improve GT line segmentation via label majority rule, then annotate shrinked polygon

Ocropy-based processors

New operation: Resegmentation

Ocropy-based processors

New operation: Resegmentation

Ocropy-based processors

New operation: Resegmentation

Ocropy-based processors

New operation: Clipping

observation 1:

on GT, both regions and lines often overlap with their neighbouring regions and lines – not just in the background, but within connected components

idea 1:

remove connected components that are not fully contained in the segment but in a neighbour

observation 2:

many frequent cases of this will create interior islands or non-contiguous polygons (not allowed in PAGE-XML, usually not supported by implementations)

idea 2:

do not remove by shrinking the polygon, but by clipping to the background colour

Ocropy-based processors

New operation: Clipping

Ocropy-based processors

New operation: Clipping

• problem: regions overlapping for no good reason
  e.g. from Tesseract

Ocropy-based processors

Planned restructuring

• tmbdev/ocropy, forked under OCR-D/ocropy:
  • move non-UI functions from CLIs into ocrolib
  • package ocrolib under PyPI ocrolib
  • package CLIs under PyPI ocropus
  • add our ocrolib changes (one by one):
    • Python 3 port
    • additional ocrolib.common functions
    • improvements in segmentation
  • try to get upstream approval
• our OCR-D/ocrd_ocropus:
  • only for OCR-D wrappers
  • base on new ocrolib

→ Ocropy as API is under way!

Tesseract-based processors

• Many OCR-related operations available
  • Mostly available via API
  • Exposed to Python via tesserocr
• Often more robust segmentation and text recognition than Ocropy
• Creating Processors for
  • Binarization (on region/line level)
  • Cropping …
  • Segmentation (on page/region/line level) …
  • Deskewing (on page/region level) …
  • Text recognition (on line/word/glyph level)

Tesseract-based processors

New processor: Poor-man's cropping

• Cropping not implemented as separate function in Tesseract
• Idea: minmal rectangle around all detected regions as Border
• Side effect: quality improvement upon repeated region detection within Border
• Problems:
  • Facing pages
  • Empty or sparsely filled pages
  • Robustness (i.e. works good for most but really bad for some pages)

Tesseract-based processors

Basal segment classification

• Distinction of different region types in Tesseract
  • Text
  • Image
  • Separator
  • Table
  • …
• Mapped to (coarser) PAGE classification
• Switches:
  • crop_polygons (rectangles or polygons?)
  • find_tables (tables as tables?)

Tesseract-based processors

Basal segment classification

Tesseract-based processors

Deskewing and orientation

• 2 distinct APIs in Tesseract:

  	DetectOrientationScript()	AnalyseLayout() + Orientation()
  confidence	yes	no
  orientation	yes	yes
  script	yes	no
  deskewing	no	yes
  textline order	no	yes
  reading direction	no	yes

  → can yield contradictory results!

Tesseract-based processors

Deskewing and orientation

• resolve conflicts:

  1. accept script results from OSD
     (if very confident)
  2. accept orientation results from OSD
     (if very confident)
  3. ignore orientation results from AL
     (but warn if contradictory)
  4. apply deskewing results from AL
     (adding angle to orientation)
  5. apply order/direction results from AL

• available on page and region level

Olena-based binarization

• Binarization is still relevant!
  • No RGB-processing recognition engine(s) available
  • High influence on OLR and OCR results
• Multiple binarization implementations in Olena
  • Kim, Niblack, Wolf …
    → expose as parameter
  • Only as CLIs
    → ocrd bashlib as last resort
• PAGE processing (but only on page level)
• AlternativeImage support (as case study in xmlstarlet)

Olena-based binarization

Original	Tesseract

Ocropy	Olena-Wolf

Module Project-based processors

• Not very many workflow-fit
  • Interface compatibility
  • Result delivery
• Insufficient documentation
  • Short READMEs
  • Missing integration examples
  • Missing training facilities
• Low visibility

Module Project-based processors

ocrd-anybaseocr-crop

• Interface-compatible page border detection
  • Collaborated effort between module project and coordination project
  • Extremely important preprocessing step (due to DFG requirements on digitization)
  • Not yet implemented for other anybaseocr-based processors
• Very, very promising results
• Input: image
• Output: Border element with coordinates
• Not yet AlternativeImage-sensitive

Module Project-based processors

ocrd-anybaseocr-crop: example

Tesseract	DFKI

DPI relativity

• Most OLR operations are sensitive to image resolution:
  e.g. minimum/maximum segment size in pixels
• Some tools are DPI-aware
  e.g. Tesseract:

  ​​Warning: Invalid resolution 0 dpi. Using 70 instead.
  

  ​​tessapi.SetVariable('user_defined_dpi', str(200))
  

• Some implementations expect a certain value
  e.g. Ocropy: 300 DPI
• We (usually) know DPI from metadata/tags:

  ​​info = OcrdExif(pil_image)
  ​​if info.resolution != 1:
  ​​    # tag available
  ​​    dpi = info.resolution
  ​​    if info.resolutionUnit == 'cm':
  ​​        dpi = round(dpi * 2.54)
  ​​    ...
  

DPI relativity

• OLR processors that are DPI-aware already: pass DPI from OcrdExif

  e.g. Tesseract wrappers

• OLR processors that are not DPI-aware yet: modify to zoom:

  1. determine factor between expected and actual DPI
  2. multiply all relevant constants in the code

  e.g. Ocropy wrappers

Description vs. image

• PAGE: hierarchy of elements, each with descriptive and binary content
  • original /PcGts/Page/@imageFilename
  • derived //AlternativeImage/@filename
• Preprocessing steps: either

  1. producing derived images obligatory:
     binarization, despeckling, dewarping

  2. just describing the operation (images optional):
     cropping/segmentation (Coords/@points), deskewing (@orientation)

     operation must be applied at some point – preferably when descending to a lower hierarchy level (i.e. during segmentation)

• Consumers: must
  • respect AlternativeImage if present on their hierarchy level of interest
  • generate an image from the parent otherwise (which again could have AlternativeImage) – by:
    • cutting from Coords/@points
    • rotating by (Page|TextRegion)/@orientation

Description vs. image

AlternativeImage problems and solutions

• bounding box rectangles are too coarse (esp. in the presence of skew)
  → polygon coordinates must always be preserved fully (using polygon masking instead of simple cutting)
• coordinates are absolute (they reference the original image)
  → before cutting from the parent image, coordinates must be converted to relative
  → before adding new child elements, coordinates must be converted from relative
  → offsets must always be passed down the hierarchy
• not all image operations retain pixel positions:
  1. deskewing → annotate @orientation, but:
     • rotation generally increases the binary image at the margins
       → compensate by additional offset (half the increase in size)
     • rotation applies around the center of the image, not the origin
       → coordinates must be translated to center, rotated passive, then translated back
  2. dewarping → use Grid?
  3. rescaling → extend PAGE-XML with @scale?

Description vs. image

Summary (slightly more abstract)

• coordinates must be reproducible:

  Annotation must be sufficient to calculate pixel positions in AlternativeImage from those in @imageFilename (e.g. to cut parents) or vice versa (e.g. to add children).

• image preprocessing steps which alter the coordinate system must describe their transform appropriately:
  • linear coordinate transformations (translation/offset, rotation/angle, scale) can be made exact (up to rounding)
  • non-linear transformations (dewarping) are inexact …

Description vs. image

Remaining issues

• Strictness of @comments classification:
  • multiple AlternativeImage entries
    → rely on @comments, or always append/choose last?
  • if AlternativeImage and @orientation
    → rely on @comments, or expect the image to be deskewed already?
  • if AlternativeImage and Border
    → rely on @comments, or expect the image to be cropped already?
• Reproducibility:
  • if AlternativeImage is larger than Coords/@points rectangle
    → due to rotation (offset) or rescaling (zoom) or both!
    → introduce @scale or prohibit rescaling altogether?
  • (page/line-level) dewarping
    → use Grid?

Description vs. image

Implementation in OCR-D/core

High-level API:

• image/offset recursion:

  • Workspace.image_from_page: on the page level (original or derived)
  • Workspace.image_from_segment: all levels below page (derived)

  what it does:

  • get last AlternativeImage or generate from parent (including:
    • conversion of coordinates to parent-relative, which involves offset correction and possibly coordinate rotation,
    • possibly image rotation)
  • return the image and its absolute bounding box (compensating for resizing by additional offset)
  • (image_from_page only:) also return OcrdExif instance for original

  how to use:

  • not recursive itself – needs to be called recursively, passing down results:

  ​​​from ocrd_modelfactory import page_from_file
  ​​​...
  ​​​    page_id = input_file.pageId or input_file.ID # for logging
  ​​​    pcgts = page_from_file(workspace.download_file(input_file))
  ​​​    page = pcgts.get_Page()
  ​​​    page_image, page_xywh, page_image_info = workspace.image_from_page(
  ​​​        page, page_id)
  ​​​    ...
  ​​​    for region in page.get_TextRegion():
  ​​​        region_image, region_xywh = workspace.image_from_segment(
  ​​​            region, page_image, page_xywh)
  ​​​        ...
  ​​​        for line in region.get_TextLine():
  ​​​            line_image, line_xywh = workspace.image_from_segment(
  ​​​            line, region_image, region_xywh)
  ​​​            ...
  

Description vs. image

Implementation in OCR-D/core

High-level API:

• add image to METS:
  Workspace.save_image_file

  what it does:

  • export image file from PIL.Image object
  • make file path from fileGrp, ID and format
  • reference the file in METS via Workspace.add_file
  • return file path

  how to use:

  • needs to know image fileGrp, unique ID:

  ​​​...
  ​​​file_id = input_file.ID.replace(self.input_file_grp, 
  ​​​    'OCR-D-IMG-DEWARP')
  ​​​    ...
  ​​​        file_path = workspace.save_image_file(image, 
  ​​​            file_id + '_' + region.id + '_' + line.id,
  ​​​            page_id=input_file.pageId,
  ​​​            file_grp='OCR-D-IMG-DEWARP')
  ​​​        line.add_AlternativeImage(AlternativeImageType(
  ​​​            filename=file_path, 
  ​​​            comments=comments + ',' + 'dewarped')
  

Description vs. image

Implementation in OCR-D/core

Low-level API:

• convert from absolute coordinates to relative: ocrd_utils.coordinates_of_segment

  what it does:

  • get the points of the element's polygon outline
  • shift all points by the offset (top-left corner) of the parent towards origin
  • (in case the parent was rotated:) rotate all points with the center of the image as pivot

  how to use:

  ​​​line_polygon = coordinates_of_segment(line, region_image, region_xywh)
  ​​​line_polygon = resegment(line_polygon, region_labels, region_image_bin, line.id)
  ​​​line_polygon = coordinates_for_segment(line_polygon, region_image, region_xywh)
  ​​​line.get_Coords().points = points_from_polygon(line_polygon)
  

Description vs. image

Implementation in OCR-D/core

Low-level API:

• convert from relative coordinates to absolute: ocrd_utils.coordinates_for_segment

  what it does:

  • (in case the parent was rotated:) rotate all points with the center of the image as pivot in opposite direction
  • shift all points by the offset (top-left corner) of the parent away from origin

  how to use:

  ​​​...
  ​​​for word_no, word in enumerate(iterate_level(tessapi.GetIterator(), RIL.WORD)):
  ​​​    word_id = '%s_word%04d' % (line.id, word_no)
  ​​​    bbox = word.BoundingBox(RIL.WORD)
  ​​​    points = points_from_polygon(coordinates_for_segment(
  ​​​        polygon_from_x0y0x1y1(bbox),
  ​​​        None, # image not needed if element cannot have angle
  ​​​       line_xywh))
  ​​​    word = WordType(id=word_id, Coords=CoordsType(points))
  ​​​    line.add_Word(word)
  ​​​
  

Description vs. image

Implementation in OCR-D/core

Low-level API:

• only coordinate rotation (as numpy.ndarray): ocrd_utils.rotate_coordinates
• mask away exterior to background: image_from_polygon
• background-agnostic replacement for PIL.Image.crop: ocrd_utils.crop_image
• …

Description vs. image

Early adopters

• core Python implementation already used by:
  • Tesseract processors ocrd-tesserocr-*
  • Ocropy processors ocrd-cis-ocropy-*
• core Bash implementation WIP used by:
  • Olena binarization ocrd-olena-binarize

Multiple inputs or outputs

• specified by comma-separated list on CLI:

  ​​$ ocrd-olena-binarize -I OCR-D-GT-SEG-LINE -O OCR-D-SEG-PAGE,OCR-D-IMG-BIN
  ​​$ ocrd-cor-asv-ann-evaluate -I OCR-D-GT-SEG-LINE,OCR-D-OCR-TESS,OCR-D-COR-ASV-ANN
  

Multiple inputs or outputs

Use-case for multi-valued input: alignment

• TextLine/TextEquiv/Unicode alignment via
  • global sequence alignment (see ocrd-cis-align or ocrd-cor-asv-ann-evaluate)
  • neural attention mechanism (cf. Dong&Smith 2018 Multi-input attention)
• resegmentation
• possibly: text alignment informed by segmentation (coordinates)

Multiple inputs or outputs

Use-case for multi-valued output: PAGE and image

• image preprocessing produces PAGE with AlternativeImage references → generated images must also be added to METS
• bad solution: fixed fileGrp OCR-D-IMG-BIN, OCR-D-IMG-DESKEW, OCR-D-IMG-DEWARP etc.
• good approach:
  • use output_file_grp second position,
  • fallback to default if not given (e.g. ocrd-tesserocr-binarize or ocrd-olena-binarize)

Logging as a result

• not all operations have a natural (PAGE/image) output file group: e.g. OLR/OCR evaluation, model training
• some need to aggregate over multiple pages (or even workspaces): e.g. CER/WER

Running

• with individual CLIs combined in a custom bash script:
• with ocrd process as engine:
• with Taverna?
• with Kitodo?

Configuration

digraph G {
	node[shape=box];
	compound=true; 

	page_segmentation[label="Region segmentation"];
	line_segmentation[label="Line segmentation"];
	text_optimization[label="Text optimization"];

	subgraph cluster_preprocessing_page {
		label = "Page preprocessing";
		binarization_page[label="Binarization"];
		cropping[label="Cropping"];
		deskewing_page[label="Deskewing"];
		despeckling_page[label="Despeckling"];
		dewarping_page[label="Dewarping"];
		binarization_page -> cropping -> deskewing_page -> despeckling_page -> dewarping_page
		{rank=same; binarization_page, cropping, deskewing_page, despeckling_page, dewarping_page}
	}
	subgraph cluster_preprocessing_segment {
		label = "Region preprocessing";
		deskewing_segment[label="Deskewing"];
		despeckling_segment[label="Despeckling"];
		binarization_segment[label="Binarization"];
		binarization_segment -> despeckling_segment -> deskewing_segment
		{rank=same; deskewing_segment, despeckling_segment, binarization_segment}
	}
	subgraph cluster_preprocessing_line {
		label = "Line preprocessing";
		binarization_line[label="Binarization"];
		dewarping_line[label="Dewarping"];
		binarization_line -> dewarping_line
		{rank=same; binarization_line, dewarping_line}
	}
	subgraph cluster_ocr_line {
		label = "Text recognition";
		ocr_one[label="OCR 1"];
		ocr_two[label="OCR 2"];
		ocr_n[label="OCR n"];
		ocr_one -> ocr_two -> ocr_n[style=dotted,dir=none]
		{rank=same; ocr_one, ocr_two, ocr_n}
	}
	binarization_page -> page_segmentation[label="Pages",ltail=cluster_preprocessing_page]
	page_segmentation -> binarization_segment[label="Regions",lhead=cluster_preprocessing_segment]
	binarization_segment -> line_segmentation[label="Regions",ltail=cluster_preprocessing_segment]
	line_segmentation -> binarization_line[label="Lines",lhead=cluster_preprocessing_line]
	binarization_line -> ocr_one[label="Line images", ltail=cluster_preprocessing_line, lhead=cluster_ocr_line]
	ocr_one -> text_optimization[label="Line strings",ltail=cluster_ocr_line]
}

Available processors

Available processors

Available processors

Available processors

Available processors

From image to regions: commands

#
# create workspace from existing METS
ocrd workspace clone \
  https://digital.slub-dresden.de/data/kitodo/gottgott_38213401X/gottgott_38213401X_mets.xml .
#
# crop with anybaseocr
ocrd-anybaseocr-crop -I ORIGINAL -O CROPPED -m mets.xml
#
# binarize on page level
ocrd-cis-ocropy-binarize -I CROPPED -O BIN -p <(echo '{"level-of-operation": "page"}') -m mets.xml
#
# deskew on page level
ocrd-cis-ocropy-deskew -I BIN -O DESKEWED -p <(echo '{"level-of-operation": "page"}') -m mets.xml
#
# segment into regions
ocrd-tesserocr-segment-region -I DESKEWED -O REGIONS -m mets.xml

From image to regions: example

From regions to lines: commands

#
# clip regions
ocrd-cis-ocropy-clip -I REGIONS -O CLIPPED -p <(echo '{"level-of-operation": "region"}') -m mets.xml
#
# binarize on region level
ocrd-cis-ocropy-binarize -I CLIPPED -O RBIN -p <(echo '{"level-of-operation": "region"}') -m mets.xml
#
# deskew on region level
ocrd-cis-ocropy-deskew -I RBIN -O RDESKEWED -p <(echo '{"level-of-operation": "region"}') -m mets.xml
#
# segment into lines
ocrd-tesserocr-segment-line -I RDESKEWED -O LINES -m mets.xml

From regions to lines: example

From lines to text: commands

#
# clip lines
ocrd-cis-ocropy-clip -I LINES -O LCLIPPED -p <(echo '{"level-of-operation": "line"}') -m mets.xml
# 
# binarize on line level
ocrd-cis-ocropy-binarize -I LCLIPPED -O LBIN -p <(echo '{"level-of-operation": "line"}') -m mets.xml
# 
# dewarp on line level
ocrd-cis-ocropy-dewarp -I LBIN -O DEWARPED -p <(echo '{"level-of-operation": "line"}') -m mets.xml
# 
# recognize text
ocrd-tesserocr-recognize -I DEWARPED -O TEXT -p <(echo '{"model": "frk+deu+Fraktur+Latin"}') -m mets.xml

From lines to text: example

Observations

• Decent progress on the image preprocessing stage
• Severe problems with recognition of (complex) page structures
• Acceptable text quality (i.e on par, sometimes even better, than ABBYY FineReader)
• Running time (roughly 2 h) per book needs improvement (?)

Measurements on current GT

• Tesseract vs. Ocropy OCR, different models
• Tesseract vs. Ocropy{nlbin} vs. Olena{Kim/Wolf/Sauvola} binarization
• binarization on page level vs. region level
• impact of various preprocessors (deskewing, dewarping, clipping, resegmentation)
• resegmentation vs. clipping on line level
• segmentation accuracy of GT itself
• deskewing on page level vs. region level
• dewarping on page level vs. line level (not covered here)

graph LR
bin["BIN{PAGE}:olena{wolf}"]
clip["CLIP{BLOCK}"]
deskew["DESKEW{BLOCK}:tesserocr"]
reseg[RESEGMENT]
dew[DEWARP:ocropy]
ocr[OCR:*]
bin --> clip
clip --> deskew
deskew --> reseg
reseg --> dew
dew --> ocr

→ layout/preprocessing still not good enough

→ Ocropy suffers from Frakturwechsel (nearly no coverage of Antiqua/Arabic numerals, no way to mix models)

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:ocropy"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Ocropy deskews slightly better than Tesseract (the latter being more conservative)

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Deskewing helps, but not that much (i.e. either GT images already have little skew, or dewarping can compensate)

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> ocr
ocr[OCR:*] 

→ Ocropy is very sensitive against warped images, Tesseract nearly immune

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> reseg
reseg["RESEG"]
reseg --> ocr
ocr[OCR:*]

→ Deskewing appearently cannot replace dewarping, i.e. either deskewing is too bad, or dewarping cannot compensate missing deskewing in the first place.

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> ocr
ocr[OCR:*]

→ Resegmentation primarily helps Ocropy, i.e. Tesseract is much less sensitive to invading as-/descenders from neighbouring lines

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> deskew
deskew["DESKEW{PAGE}:tesserocr"]
deskew --> clip
clip["CLIP{BLOCK}"]
clip --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Deskewing (on average) works slightly better on the region level than on the page level

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> den
den["DENOISE{PAGE}:ocropy"]
den --> deskew
deskew["DESKEW{PAGE}:ocropy"]
deskew --> clip
clip["CLIP{BLOCK}"]
clip --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Denoising can improve Wolf binarization, esp. for Ocropy

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Clipping on the region level gives very minimal improvement (if resegmentation is used)

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Resegmentation primarily helps Ocropy, i.e. Tesseract is much less sensitive to invading as-/descenders from neighbouring lines

→ Resegmentation needs deskewing

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> clip2
clip2["CLIP{LINE}"]
clip2 --> ocr
ocr[OCR:*]

→ Clipping on the line level cannot quite replace resegmentation, with Tesseract it does not help at all

graph LR
bin["BIN{PAGE}:olena{kim}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Kim is noticeably worse than Wolf (on average)

graph LR
bin["BIN{PAGE}:olena{sauvola}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Basic Sauvola is slightly better than Wolf (on average)

graph LR
bin["BIN{PAGE}:olena{sauvola-ms-split}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ This Sauvola variant is even better (on average)

graph LR
bin["BIN{PAGE}:ocropy{nlbin}"]
bin --> clip
clip["CLIP{BLOCK}"]
clip --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Ocropy{nlbin} is really bad! (but what about perc / range / threshold / lo / hi parameters?)

graph LR
clip["CLIP{BLOCK}"]
clip --> bin
bin["BIN{BLOCK}:ocropy{nlbin}"]
bin --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Binarization on the region level is superior to the page level (if using Ocropy{nlbin}!)

graph LR
clip["CLIP{BLOCK}"]
clip --> bin
bin["BIN{BLOCK}:tesserocr"]
bin --> deskew
deskew["DESKEW{BLOCK}:tesserocr"]
deskew --> reseg
reseg["RESEG"]
reseg --> dewarp
dewarp["DEWARP:ocropy"]
dewarp --> ocr
ocr[OCR:*]

→ Ocropy cannot cope with Tesseract binarization.

→ But even Tesseract prefers Ocropy binarization!

graph LR
bin["BIN{BLOCK}:ocropy{nlbin}"]
bin --> ocr
ocr[OCR:*]

→ Without the extra preprocessors, Ocropy is lost on GT.

¹: "naîve" configuration of Ocropy

graph LR
bin["BIN{BLOCK}:tesserocr"]
bin --> ocr
ocr[OCR:*]

→ Without the extra processors, Tesseract still works.

²: "naîve" configuration of Tesseract (but how does this compare to the CLI?)

graph LR
bin["BIN{PAGE}:olena{wolf}"]
bin --> ocr
ocr[OCR:*]

→ Again, Ocropy requires the extra preprocessors, while Tesseract is suprisingly insensitive to invading neighbouring regions/lines, against skewed and warped images.

Measurements on GT4HistOCR

• directory / file name suffix structure instead of METS/PAGE
• pre-built (fixed) preprocessing with less effort (no polygons, no clipping / resegmentation / dewarping)
• automatic segmentation by alignment instead of manual segmentation
• representative?
• large enough for training (OCR, COR)

graph LR
bin["BIN{PAGE}:ocropy{nlbin}"]
deskew["DESKEW{PAGE}:ocropy{nlbin}"]
lseg["SEG{LINE}:ocropy{gpageseg}"]
bin --> deskew
deskew --> lseg
lseg --> ocr
ocr[OCR:*]
style bin fill:#ccc
style deskew fill:#ccc
style lseg fill:#ccc

→ Much better than best measured OCR-D configuration!

Striving for best-practice configurations

• Increasing number of processors
  • DFKI preprocessing and layout analysis
  • Würzburg layout analysis
  • Leipzig and Munich post correction
• Increasing number of models
  • Trainable processors?
  • Erlangen font detection + Leipzig training tools
  • OCR-D GT
• Increasing number of possible configurations
  • Recommendations for users are needed!
  • (Tools and workflows for text-based evaluation)
  • Tools and workflows for layout evaluation