I followed the tutorial on the docs and tried finetuning jpn_vert. I added the font to lang_specific too. This is the output I get

Warning in pixScaleSmooth: ridiculously small scaling factor 0.010451
Image too small to scale!! (1x1 vs min width of 3)
Line cannot be recognized!!
Image not trainable

Here is the script I used

# generates training data
tesseract/src/training/tesstrain.sh \
        --fonts_dir ${FONTS_DIR} \
        --tessdata_dir  ${TESSDATA_DIR} \
        --langdata_dir ${LANG_DATA_DIR} \
        --linedata_only \
        --lang jpn \
        --fontlist "Tanuki Permanent Marker" \
        --noextract_font_properties \
        --output_dir train \
        --exposures "0"  

# extracts lstm from old traineddata
combine_tessdata -e ${TESSDATA_DIR}/jpn_vert.traineddata jpn.lstm

# finetunes model
lstmtraining \
    --continue_from jpn.lstm \
    --model_output output/jpn \
    --traineddata ${TESSDATA_DIR}/jpn_vert.traineddata \
    --train_listfile train/jpn.training_files.txt \
    --max_iterations 100

# combines into a .traineddata
lstmtraining --stop_training \
    --continue_from output/jpn_checkpoint \
    --traineddata ${TESSDATA_DIR}/jpn_vert.traineddata \
    --model_output output/jpn_vert.traineddata

What am I doing wrong here? It worked on jpn but did not work with jpn_vert.

<del>Notably, this defines (yet another) new venv for ocrd_calamari since it now requires tensorflow 2.4 and numpy >= 1.19</del>

ocrd_calamari doesn't require 2.4 yet, it uses 2.3 as does ocrd_pc_segmentation, so they can share venv/local/sub-venv/headless-tf2 for now.

thanks

With this set, Windows users will have text files converted from Windows style line endings (\r\n) to Unix style line endings (\n) when they’re added to the repository. https://www.edwardthomson.com/blog/git_for_windows_line_endings.html

thanks

thanks.

This reduces the size of those objects.

https://github.com/ocropus/ocropy was built in WSL2 using the following commands:

$ conda create -n ocropus_env python=2.7 $ source activate ocropus_env $ conda install --file requirements.txt $ wget -nd https://github.com/zuphilip/ocropy-models/raw/master/en-default.pyrnn.gz $ mv en-default.pyrnn.gz models/ $ python setup.py install

however I encounter some unexpected behavior...

1. the command ipython launches Python 3.8.3 and not the 2.7 version
2. from within the ipython interpreter my site-packages are referring to the wrong folder

In [1]: from distutils.sysconfig import get_python_lib In [2]: print(get_python_lib())

rather than the site-packages existing in the expected directory, which would look like this: /mnt/d/anaconda3/envs/ocropus_env/lib/python2.7/site-packages
running the code above shows it's location here: /mnt/d/anaconda3/lib/python3.8/site-packages

in fact the site-packages are contained within the folder: /mnt/d/anaconda3/envs/ocropus_env/lib/python2.7/site-packages

i cannot figure out what command to type to produce an ipython or python interpreter which will refer back to the correct site-packages directory.

i also cannot figure out how to test the recognizer with the prescribed ./run-test command. typing ./run-test only produces the output: : invalid option

The problem here is numpy compatibility:

tensorflow numpy 1.15.4 <1.19.0,>=1.16.0 2.1 <2.0,>=1.16.0 2.2 <2.0,>=1.16.0 2.3 <1.19.0,>=1.16.0 2.4.0rc3 ~=1.19.2

Wow. Had no idea it's that bad. There seems to be no limit to Tensorflow madness.

oh wait, ocrd_calamari does indeed require 2.3, so yes we can reuse the venv! I'll adapt the PR.

The problem here is numpy compatibility:

If you see a way to group the venvs more efficiently, let me know.

Notably, this defines (yet another) new venv for ocrd_calamari since it now requires tensorflow 2.14 and numpy >= 1.19

Is it really necessary to differentiate with ocrd_pc_segmentation's tf2 venv?

At least superficially, they could be compatible:

tensorflow >= 2.3.0rc2

tensorflow == 2.3.1

Of course, there might be other conflicts, I have not tried it.

The motivation for this is that each new TF installation

• takes a lot of bandwidth to build
• bloats the Docker image by ~ 2GB

Notably, this defines (yet another) new venv for ocrd_calamari since it now requires tensorflow 2.14 and numpy >= 1.19

It looks like there was a similar issue already logged under tesseract-ocr: https://github.com/tesseract-ocr/tesseract/issues/2828

Is the issue related to any missing leptonica development headers??? Are there separate build steps for building leptonica with development headers from source (Ubuntu18 package: libleptonica-dev -> "libleptonica-dev is already the newest version (1.75.3-3) )?

Thank you for the report.

This is not a leptonica issue. It might be a tesseract build issue, but I haven't seen it reported before. Please file the issue with tesseract if you haven't done already.

Dan

I built leptonica from source and everything built successfully. However, when I built tesseract, tesseract build process complains about a linker error (`LeptMsgSeverity' can not be used when making a shared object; recompile with -fPIC).

Is this a tesseract issue or leptonica issue?

Also, I did not see leptonica or tesseract library packages for Ubuntu20. When might these packages (e.g. libtesseract-dev, libleptonica-dev) become available?

Thanks.

(py37tf) kaisark@DESKTOP-L5GI3V9:~/Downloads/leptonica/build$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local -- The C compiler identification is GNU 9.3.0 -- The CXX compiler identification is GNU 9.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Could NOT find GIF (missing: GIF_LIBRARY GIF_INCLUDE_DIR) -- Found JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (found version "80") -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") -- Found PNG: /usr/lib/x86_64-linux-gnu/libpng.so (found version "1.6.37") -- Found TIFF: /usr/lib/x86_64-linux-gnu/libtiff.so (found version "4.1.0") -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Can not find: /webp/mux.h -- Setting build type to 'Release' as none was specified. -- Looking for include file dlfcn.h -- Looking for include file dlfcn.h - found -- Looking for include file inttypes.h -- Looking for include file inttypes.h - found -- Looking for include file memory.h -- Looking for include file memory.h - found -- Looking for include file stdint.h -- Looking for include file stdint.h - found -- Looking for include file stdlib.h -- Looking for include file stdlib.h - found -- Looking for include file strings.h -- Looking for include file strings.h - found -- Looking for include file string.h -- Looking for include file string.h - found -- Looking for include file sys/stat.h -- Looking for include file sys/stat.h - found -- Looking for include file sys/types.h -- Looking for include file sys/types.h - found -- Looking for include file unistd.h -- Looking for include file unistd.h - found -- Looking for include file openjpeg-2.0/openjpeg.h -- Looking for include file openjpeg-2.0/openjpeg.h - not found -- Looking for include file openjpeg-2.1/openjpeg.h -- Looking for include file openjpeg-2.1/openjpeg.h - not found -- Looking for include file openjpeg-2.2/openjpeg.h -- Looking for include file openjpeg-2.2/openjpeg.h - not found -- Looking for include file openjpeg-2.3/openjpeg.h -- Looking for include file openjpeg-2.3/openjpeg.h - not found -- Looking for fmemopen -- Looking for fmemopen - found -- Looking for fstatat -- Looking for fstatat - found -- Check if the system is big endian -- Searching 16 bit integer -- Looking for stddef.h -- Looking for stddef.h - found -- Check size of unsigned short -- Check size of unsigned short - done -- Using unsigned short -- Check if the system is big endian - little endian

-- General configuration for Leptonica 1.81.0

-- Build type: Release -- Compiler: GNU -- C compiler options: -- Linker options: -- Install directory: /usr/local

-- Build with sw [SW_BUILD]: OFF -- Build utility programs [BUILD_PROG]: OFF -- Used ZLIB library: /usr/lib/x86_64-linux-gnu/libz.so -- Used PNG library: /usr/lib/x86_64-linux-gnu/libpng.so;/usr/lib/x86_64-linux-gnu/libz.so -- Used JPEG library: /usr/lib/x86_64-linux-gnu/libjpeg.so -- Used JP2K library: -- Used TIFF library: /usr/lib/x86_64-linux-gnu/libtiff.so -- Used GIF library: -- Used WEBP library:

-- -- Configuring done -- Generating done -- Build files have been written to: /home/kaisark/Downloads/leptonica/build

libtool: link: g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginS.o src/api/.libs/libtesseract_la-baseapi.o src/api/.libs/libtesseract_la-altorenderer.o src/api/.libs/libtesseract_la-capi.o src/api/.libs/libtesseract_la-hocrrenderer.o src/api/.libs/libtesseract_la-lstmboxrenderer.o src/api/.libs/libtesseract_la-pdfrenderer.o src/api/.libs/libtesseract_la-renderer.o src/api/.libs/libtesseract_la-wordstrboxrenderer.o src/arch/.libs/libtesseract_la-intsimdmatrix.o src/arch/.libs/libtesseract_la-simddetect.o src/ccmain/.libs/libtesseract_la-applybox.o src/ccmain/.libs/libtesseract_la-control.o src/ccmain/.libs/libtesseract_la-linerec.o src/ccmain/.libs/libtesseract_la-ltrresultiterator.o src/ccmain/.libs/libtesseract_la-mutableiterator.o src/ccmain/.libs/libtesseract_la-output.o src/ccmain/.libs/libtesseract_la-pageiterator.o src/ccmain/.libs/libtesseract_la-pagesegmain.o src/ccmain/.libs/libtesseract_la-pagewalk.o src/ccmain/.libs/libtesseract_la-paragraphs.o src/ccmain/.libs/libtesseract_la-paramsd.o src/ccmain/.libs/libtesseract_la-pgedit.o src/ccmain/.libs/libtesseract_la-reject.o src/ccmain/.libs/libtesseract_la-resultiterator.o src/ccmain/.libs/libtesseract_la-tessedit.o src/ccmain/.libs/libtesseract_la-tesseractclass.o src/ccmain/.libs/libtesseract_la-tessvars.o src/ccmain/.libs/libtesseract_la-thresholder.o src/ccmain/.libs/libtesseract_la-werdit.o src/ccmain/.libs/libtesseract_la-adaptions.o src/ccmain/.libs/libtesseract_la-docqual.o src/ccmain/.libs/libtesseract_la-equationdetect.o src/ccmain/.libs/libtesseract_la-fixspace.o src/ccmain/.libs/libtesseract_la-fixxht.o src/ccmain/.libs/libtesseract_la-osdetect.o src/ccmain/.libs/libtesseract_la-par_control.o src/ccmain/.libs/libtesseract_la-recogtraining.o src/ccmain/.libs/libtesseract_la-superscript.o src/ccmain/.libs/libtesseract_la-tessbox.o src/ccmain/.libs/libtesseract_la-tfacepp.o src/ccstruct/.libs/libtesseract_la-blamer.o src/ccstruct/.libs/libtesseract_la-blobbox.o src/ccstruct/.libs/libtesseract_la-blobs.o src/ccstruct/.libs/libtesseract_la-blread.o src/ccstruct/.libs/libtesseract_la-boxread.o src/ccstruct/.libs/libtesseract_la-boxword.o src/ccstruct/.libs/libtesseract_la-ccstruct.o src/ccstruct/.libs/libtesseract_la-coutln.o src/ccstruct/.libs/libtesseract_la-detlinefit.o src/ccstruct/.libs/libtesseract_la-dppoint.o src/ccstruct/.libs/libtesseract_la-imagedata.o src/ccstruct/.libs/libtesseract_la-linlsq.o src/ccstruct/.libs/libtesseract_la-matrix.o src/ccstruct/.libs/libtesseract_la-mod128.o src/ccstruct/.libs/libtesseract_la-normalis.o src/ccstruct/.libs/libtesseract_la-ocrblock.o src/ccstruct/.libs/libtesseract_la-ocrpara.o src/ccstruct/.libs/libtesseract_la-ocrrow.o src/ccstruct/.libs/libtesseract_la-otsuthr.o src/ccstruct/.libs/libtesseract_la-pageres.o src/ccstruct/.libs/libtesseract_la-pdblock.o src/ccstruct/.libs/libtesseract_la-points.o src/ccstruct/.libs/libtesseract_la-polyaprx.o src/ccstruct/.libs/libtesseract_la-polyblk.o src/ccstruct/.libs/libtesseract_la-quadlsq.o src/ccstruct/.libs/libtesseract_la-quspline.o src/ccstruct/.libs/libtesseract_la-ratngs.o src/ccstruct/.libs/libtesseract_la-rect.o src/ccstruct/.libs/libtesseract_la-rejctmap.o src/ccstruct/.libs/libtesseract_la-seam.o src/ccstruct/.libs/libtesseract_la-split.o src/ccstruct/.libs/libtesseract_la-statistc.o src/ccstruct/.libs/libtesseract_la-stepblob.o src/ccstruct/.libs/libtesseract_la-werd.o src/ccstruct/.libs/libtesseract_la-fontinfo.o src/ccstruct/.libs/libtesseract_la-params_training_featdef.o src/classify/.libs/libtesseract_la-classify.o src/classify/.libs/libtesseract_la-adaptive.o src/classify/.libs/libtesseract_la-adaptmatch.o src/classify/.libs/libtesseract_la-blobclass.o src/classify/.libs/libtesseract_la-cluster.o src/classify/.libs/libtesseract_la-clusttool.o src/classify/.libs/libtesseract_la-cutoffs.o src/classify/.libs/libtesseract_la-featdefs.o src/classify/.libs/libtesseract_la-float2int.o src/classify/.libs/libtesseract_la-fpoint.o src/classify/.libs/libtesseract_la-intfeaturespace.o src/classify/.libs/libtesseract_la-intfx.o src/classify/.libs/libtesseract_la-intmatcher.o src/classify/.libs/libtesseract_la-intproto.o src/classify/.libs/libtesseract_la-kdtree.o src/classify/.libs/libtesseract_la-mf.o src/classify/.libs/libtesseract_la-mfdefs.o src/classify/.libs/libtesseract_la-mfoutline.o src/classify/.libs/libtesseract_la-mfx.o src/classify/.libs/libtesseract_la-normfeat.o src/classify/.libs/libtesseract_la-normmatch.o src/classify/.libs/libtesseract_la-ocrfeatures.o src/classify/.libs/libtesseract_la-outfeat.o src/classify/.libs/libtesseract_la-picofeat.o src/classify/.libs/libtesseract_la-protos.o src/classify/.libs/libtesseract_la-shapeclassifier.o src/classify/.libs/libtesseract_la-shapetable.o src/classify/.libs/libtesseract_la-tessclassifier.o src/classify/.libs/libtesseract_la-trainingsample.o src/cutil/.libs/libtesseract_la-emalloc.o src/cutil/.libs/libtesseract_la-oldlist.o src/dict/.libs/libtesseract_la-context.o src/dict/.libs/libtesseract_la-dawg.o src/dict/.libs/libtesseract_la-dawg_cache.o src/dict/.libs/libtesseract_la-dict.o src/dict/.libs/libtesseract_la-stopper.o src/dict/.libs/libtesseract_la-trie.o src/dict/.libs/libtesseract_la-hyphen.o src/dict/.libs/libtesseract_la-permdawg.o src/textord/.libs/libtesseract_la-alignedblob.o src/textord/.libs/libtesseract_la-baselinedetect.o src/textord/.libs/libtesseract_la-bbgrid.o src/textord/.libs/libtesseract_la-blkocc.o src/textord/.libs/libtesseract_la-blobgrid.o src/textord/.libs/libtesseract_la-ccnontextdetect.o src/textord/.libs/libtesseract_la-cjkpitch.o src/textord/.libs/libtesseract_la-colfind.o src/textord/.libs/libtesseract_la-colpartition.o src/textord/.libs/libtesseract_la-colpartitionset.o src/textord/.libs/libtesseract_la-colpartitiongrid.o src/textord/.libs/libtesseract_la-devanagari_processing.o src/textord/.libs/libtesseract_la-drawtord.o src/textord/.libs/libtesseract_la-edgblob.o src/textord/.libs/libtesseract_la-edgloop.o src/textord/.libs/libtesseract_la-fpchop.o src/textord/.libs/libtesseract_la-gap_map.o src/textord/.libs/libtesseract_la-imagefind.o src/textord/.libs/libtesseract_la-linefind.o src/textord/.libs/libtesseract_la-makerow.o src/textord/.libs/libtesseract_la-oldbasel.o src/textord/.libs/libtesseract_la-pithsync.o src/textord/.libs/libtesseract_la-pitsync1.o src/textord/.libs/libtesseract_la-scanedg.o src/textord/.libs/libtesseract_la-sortflts.o src/textord/.libs/libtesseract_la-strokewidth.o src/textord/.libs/libtesseract_la-tabfind.o src/textord/.libs/libtesseract_la-tablefind.o src/textord/.libs/libtesseract_la-tabvector.o src/textord/.libs/libtesseract_la-tablerecog.o src/textord/.libs/libtesseract_la-textlineprojection.o src/textord/.libs/libtesseract_la-textord.o src/textord/.libs/libtesseract_la-topitch.o src/textord/.libs/libtesseract_la-tordmain.o src/textord/.libs/libtesseract_la-tospace.o src/textord/.libs/libtesseract_la-tovars.o src/textord/.libs/libtesseract_la-underlin.o src/textord/.libs/libtesseract_la-wordseg.o src/textord/.libs/libtesseract_la-workingpartset.o src/textord/.libs/libtesseract_la-equationdetectbase.o src/viewer/.libs/libtesseract_la-scrollview.o src/viewer/.libs/libtesseract_la-svmnode.o src/viewer/.libs/libtesseract_la-svutil.o src/wordrec/.libs/libtesseract_la-tface.o src/wordrec/.libs/libtesseract_la-wordrec.o src/wordrec/.libs/libtesseract_la-associate.o src/wordrec/.libs/libtesseract_la-chop.o src/wordrec/.libs/libtesseract_la-chopper.o src/wordrec/.libs/libtesseract_la-drawfx.o src/wordrec/.libs/libtesseract_la-findseam.o src/wordrec/.libs/libtesseract_la-gradechop.o src/wordrec/.libs/libtesseract_la-language_model.o src/wordrec/.libs/libtesseract_la-lm_consistency.o src/wordrec/.libs/libtesseract_la-lm_pain_points.o src/wordrec/.libs/libtesseract_la-lm_state.o src/wordrec/.libs/libtesseract_la-outlines.o src/wordrec/.libs/libtesseract_la-params_model.o src/wordrec/.libs/libtesseract_la-pieces.o src/wordrec/.libs/libtesseract_la-plotedges.o src/wordrec/.libs/libtesseract_la-render.o src/wordrec/.libs/libtesseract_la-segsearch.o src/wordrec/.libs/libtesseract_la-wordclass.o -Wl,--whole-archive ./.libs/libtesseract_ccutil.a ./.libs/libtesseract_lstm.a ./.libs/libtesseract_native.a ./.libs/libtesseract_opencl.a ./.libs/libtesseract_avx.a ./.libs/libtesseract_avx2.a ./.libs/libtesseract_fma.a ./.libs/libtesseract_sse.a -Wl,--no-whole-archive -L/usr/local/lib -lleptonica -lpthread -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/9/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crtn.o -fopenmp -g -O2 -fopenmp -Wl,-soname -Wl,libtesseract.so.5 -o .libs/libtesseract.so.5.0.0 /usr/bin/ld: /usr/local/lib/libleptonica.a(colorspace.c.o): relocation R_X86_64_PC32 against symbol `LeptMsgSeverity' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: bad value collect2: error: ld returned 1 exit status make[2]: *** [Makefile:2216: libtesseract.la] Error 1 make[2]: Leaving directory '/home/kaisark/tesseract' make[1]: *** [Makefile:4358: all-recursive] Error 1 make[1]: Leaving directory '/home/kaisark/tesseract' make: *** [Makefile:1595: all] Error 2