Poor Rotation / Layout detection

[CanadianHusky]

Current Behavior

Are there any improvements for layout and rotation detection planned ?
No meaningful part is captured from the synthetic test image attached, no matter what psm mode is used

tesseract.exe --psm 1 -c min_characters_to_try=2 --dpi 300 -l eng "input.jpg" "output" hocr
tesseract.exe --psm 12 -c min_characters_to_try=2 --dpi 300 -l eng "input.jpg" "output" hocr

Also desipte -c min_characters_to_try=2 given, output complains
Too few characters. Skipping this page
OSD: Weak margin (0.00) for 23 blob text block, but using orientation anyway: 0

Output is just nonsense...at least the confidence is low enough

     <span class='ocr_line' id='line_1_1' title="bbox 100 450 120 718; textangle 90; x_size 27.333334; x_descenders 6.8333335; x_ascenders 6.8333335">
      <span class='ocrx_word' id='word_1_1' title='bbox 100 490 120 718; x_wconf 16'>NOILVYLSININGY</span>
      <span class='ocrx_word' id='word_1_2' title='bbox 100 450 120 482; x_wconf 70'>20</span>
     </span>

     <span class='ocr_line' id='line_1_2' title="bbox 62 1054 330 1074; baseline 0 0; x_size 27.333334; x_descenders 6.8333335; x_ascenders 6.8333335">
      <span class='ocrx_word' id='word_1_3' title='bbox 62 1054 290 1074; x_wconf 37'>NOILVYLSININGY</span>
      <span class='ocrx_word' id='word_1_4' title='bbox 298 1054 330 1074; x_wconf 0'>¢0</span>
     </span>

Synthetic image is minimized representative example of actual content

Expected Behavior

better detection for zones with rotated text.

Suggested Fix

No response

tesseract -v

tesseract v5.5.0.20241111
 leptonica-1.85.0
  libgif 5.2.2 : libjpeg 8d (libjpeg-turbo 3.0.4) : libpng 1.6.44 : libtiff 4.7.0 : zlib 1.3.1 : libwebp 1.4.0 : libopenjp2 2.5.2
 Found AVX2
 Found AVX
 Found FMA
 Found SSE4.1
 Found libarchive 3.7.7 zlib/1.3.1 liblzma/5.6.3 bz2lib/1.0.8 liblz4/1.10.0 libzstd/1.5.6
 Found libcurl/8.11.0 Schannel zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0

Operating System

Windows 10

Other Operating System

No response

uname -a

No response

Compiler

No response

CPU

AMD Ryzen 9, X3900

Virtualization / Containers

No response

Other Information

I looked at the recent updates for the last year and more, since version 5.0 release.
Many compiler fixes, code hygiene, cleanup and a few specific edge case bug fixes are done but I don't see any updates that really improve core OCR engine performance or accuracy for layout. I guess those are really difficult subjects.

[amitdo]

IMO, this post is very rude.

[stweil]

Are there any improvements for layout and rotation detection planned ?

Short answer: the current development is entirely community driven, so unless someone is interested in these issues and willing to invest efforts in programming, nothing will change. I don't know such plans.

[CanadianHusky]

@amitdo If I had mentioned this issue in person, verbally, I am confident you would have a different opinion. With the lack of body language and tone of voice, written communication is open to misunderstanding and misinterpretation. I had no intention of offending anyone of the community that invest their personal time in this project. I am aware how much effort goes into such projects and it is appreciated.
If I had enough C++ knowledge myself, I would gladly contribute to the code base. I can only contribute to this project by identifying issues, bring it to the attention of the community, with the hope and intention of improving an already great project. It would be a shame to see this project being abandoned. Despite its age, it is still one of the best OCR Engines (depending on content), especially after the major LSTM update.

In order to work around the issue of failed layout detection, I use another ML-Model that does text zone location detection. It returns all rectangle structures 99.99% correct and fairly fast (700ms on the sample below), no matter how nasty (mixed with strange graphics, regardless of language) the content is, even from natural images. My code then crops those individual rotated rectangle structs out and feeds them orthogonal into Tesseract because the OCR accuracy, especially for German and German Fraktur letters, in Tesseract is far better than any other model that I use. That intermediate step gets the job done, but is only a workaround.

If someone that knows the tesseract code base reads this message in the future, and is willing and able to improve this subject, it would be much easier to allow for an input file that feeds the tesseract engine directly with the zones to be OCR'ed - crop them out with the build-in leptonica engine, instead of trying to rework the entire layout engine and and let tesseract do the layout detection.

proposed option: tesseract.exe -c use_layout_file=(json filename with array of rotated rects.json)
the json file contains object array for rotated rectangles, similar to openCV's rotatedRectangle structure.
Tessearct psm mode skips OSD detection completely, or a new psm mode tells it to use layout from json.
The main thing for tessaract left to do is to read base image, crop out rotated zones from base image in an orthogonal way and perform OCR on individual zones. Since the zones geometry (bounding boxes) are already known, writing results out to hOCR or similar file should be no problem.

I have no expectation for any update for myself. I already have a working solution, although not the most efficient and clean - but it works.
My goal with the first post, as with this one was to make a great OCR Engine even better and bring this issue to the community teams attention.
Respectfully, I stand by my personal opinion that many recent updates were mostly the "low hanging fruits".

@stweil Understood. Thank you.

[amitdo]

The layout analysis in Tesseract was design to deal with books, newspapers and magazines. It has many limitations and is not trainable.

As said, this part of Tesseract will probably not be improved in the algorithm level by any member of the current team.

[CanadianHusky]

Understood.
But maybe my suggestion was misunderstood because I see tag "layout analysis".
My proposed improved does not require retraining or changing core detection algorithm in tesseract.
In fact it proposes to bypass the current algorithm that does layout, orientation and text zone detection as that can be outsourced to other libraries that do that part extremely well. Near perfect ML models exist that produce the required quadrilaterals.
tesseract.exe -c use_layout_file=(json filename with array of rotated rects.json)
with the above tesseract is only expected to crop out rotated regions from the input bitmap, unrotate to make horizontal, perform OCR matching and return hOCR or text. I have implemented this workflow by launching tesseract individually for each textline. The overhead of reading same bitmap and initializing the LSTM models is high, thats why it is not efficient but it works.
With the proposed method, input bitmap and LSTM model would be initialized only once; therefore run much faster and it would give Tesseract the flexibility to be used with any other ML Text detection library, or the option to do partial OCR because in some cases the layout is well defined (example ID Cards) and known in advance.
Best Regards

[stweil]

Do you have some documentation of your workflow with ML models for the layout detection? I'd like to try it myself.

[amitdo]

Your initial comment in this issue was certainly about layout analysis. you gave it the title "Poor Rotation / Layout detection". Your suggestion / feature request only appeared later.

[CanadianHusky]

Do you have some documentation of your workflow with ML models for the layout detection? I'd like to try it myself.

Certainly. It is very simple and just a few steps.
Below is the documentation and instructions how to create the attached json file using the pre-trained ML models available in https://github.com/PaddlePaddle/PaddleOCR (Licence = Apache 2.0 opensource)

1-
Prepare python infrastructure:
https://paddlepaddle.github.io/PaddleOCR/main/en/quick_start.html
I use the CPU model, but multiple GPU, Cuda based models for different version nvidia cards exist as well.

2-
execute python to analyze layout
2.1- either for text zone detection only, See https://paddlepaddle.github.io/PaddleOCR/main/en/quick_start.html#__tabbed_2_2
2.2- or text zone detection and text recognition (to compare confidence against tesseract and chose whichever is better)
PaddleOCR is often poor on German diacritics (Umlaute) , where Tesseract is excellent.
Also for good input image Tesseract is much faster (both engine init and running) on non-GPU (CPU only) systems when compared against Paddle's Server Model.
See https://paddlepaddle.github.io/PaddleOCR/main/en/quick_start.html#__tabbed_2_1

My sample is created with

from paddleocr import PaddleOCR
# this uses lightweight mobile ML model (small, fast, sometimes a bit less accurate) - larger server grade models exist that can be installed locally.
ocr = PaddleOCR(use_textline_orientation=True, use_doc_orientation_classify=False, use_doc_unwarping=False)
result = ocr.predict("input.jpg")
for res in result:
    res.save_to_img("output")
    res.save_to_json("output")

3- parse json (sample attached). Dictionaries of interest in the json created are:

"dt_polys": Detection zone polygon (array of quadrilaterals) these are rotated rectangle structures with 4 x,y points.
I crop these regions out of the original bitmap via code, deskew the quad to an orthogonal rect bitmap and feed that into tesseract.

"textline_orientation_angles": this is supposed to detect 0 vs 180, or 90 vs 270, or 45 vs 315 degrees etc (right to left or left to right) it works sometimes, but I found cases where it is not reliable enough. I use OSD detection in Tesseract to detect against 180 flipped text.

"rec_polys": only present if full OCR was done. Text recognition zone polygon (array of quad points) usually same as dt_polys. Not needed for the proposed workflow.

"rec_boxes": only present if full OCR was done. Text recognition zone bounding box as orthogonal rectangle. Not needed for the proposed workflow.

4-
the most important (and only needed) data for the layout is in "dt_polys":
code matches those coordinates up with the tesseract OCR text results (feeding line by line) and this is enough information (text, confidence and geometry) to rebuild a searchable PDF or output to whatever format is needed.

If something like
tesseract.exe --psm 14 -c use_layout_file=(json filename with array of rotated rects.json)
existed it would allow reading a simple json that contains "dt_polys": and tesseract can handle the rest, bypassing all layout detection completely.
One additional option needed may be something like -c check_for_180=True because the quad locations may contain 90/270 45/315 degree rotated text.
Internally PSM mode should probably always be 7 or 8 or 13 ( single line or single word ) because this workflow will never feed multiline into Tesseract.

If something is unclear, I'd be happy to provide more details.
Thank you.

300_mixed_rotation.json

[CanadianHusky]

Your initial comment in this issue was certainly about layout analysis. you gave it the title "Poor Rotation / Layout detection". Your suggestion / feature request only appeared later.

You are correct.

[CanadianHusky]

For reference, here a sample of what the proposed outsourcing of layout to another ML model could be capable of using my childhood comics as an example.
Not a single text object is missed, regardless if they are at an angle or inside a graphical image. It can hardly get any better.

[stweil]

Beeindruckend! Even although "Telefon" was missed.

[CanadianHusky]

Beeindruckend! Even although "Telefon" was missed.

only because I used the 'fast' mode of the detection and recognition model for the example above.
Here is a tuned version with slightly tweaked params. "TELEFON" perfectly fine.

What I find really impressive is that the angled word "MUSKELSCHMALZ" which has a partly cut off letter 'M' is detected correctly in the output now.

Ich hatte hunderte von diesen Comics. Gute alte Zeiten.