- Organizing development in the coordination project (Paul Pestov)
- Vision for the workflow engine by GWDG (Triet Doan)
We meet at 10-11 am in a BBB room. At the beginning of each meeting, participants have the opportunity to suggest topics of interest to them, which can then be discussed and debated in small groups. Discussion points can also be entered in advance in the HedgeDoc. In addition to the Open TechCall, where the OCR-D community discusses technical topics, OCR(-D) & Co also offers participants without in-depth OCR-D knowledge the opportunity to contribute their own questions and ideas to the discussion and to openly exchange ideas with each other.
We have deployed and tested this in a practical system performance-wise:
- in TF, set
per_process_gpu_memory_fractionand control via environment variable (how many processes are meant to share the same physical GPU?): https://github.com/OCR-D/ocrd_segment/blob/b27131bcadf563c947fee51aabe7656d1c185b8c/ocrd_segment/classify_formdata_layout.py#L107 - in TF, activate
use_unified_memoryand thus prevent OOM (in case you misestimated peak allocation): https://github.com/OCR-D/ocrd_segment/blob/b27131bcadf563c947fee51aabe7656d1c185b8c/ocrd_segment/classify_formdata_layout.py#L109 - at the start of the server, configure processors such that as many instances get GPU access as do actually fit and set the envvar accordingly: https://git.informatik.uni-leipzig.de/smarthec/smarthec_webservice#docker-compose
@/all https://pad.gwdg.de/s/uHnvC1wUW
Wie auf dem OCR-D-Workshop angekündigt, möchte OCR-D ein Forum für Ground Truth organisieren. Ich schlage vor, dass dies am Donnerstag zwischen 13:00 - 14:00 Uhr stattfindet. Unser Plan ist: Wir beginnen am 16. September.
Themen:
- Nutzung der Ground Truth Guidelines (https://ocr-d.de/de/gt-guidelines/trans/)
- GT Speicherung, GT Nutzung, GT Kurration
- Transformationen von ABBYY, ALTO, ... nach Ground Truth format PAGE
- Fragen zu PAGE
As announced at the OCR-D workshop, OCR-D would like to organize a forum for Ground Truth. I propose that this take place on Thursday between 1:00 - 2:00 pm. Our plan is: we will start on September 16.
Topics
- Ground Truth Guidelines usage (https://ocr-d.de/en/gt-guidelines/trans/).
- GT Storage, GT useing, curration.
- Transformations from ABBYY, ALTO, ... to Ground Truth format PAGE
- Questions about PAGE
main for the default git branch of all repositories (see tesseract-ocr/tesseract#3554). I suggest to do that for the OCR-D repositories as well.
ocrd-olena-binarize -I OCR-D-IMG -O OCR-D-BIN -P impl sauvola from the user guide on a small subset of my data with 400 pages or so and it seems that each page takes like 4 seconds to process including two times writing of the mets.xml. At the same time I am seeing almost no cpu / memory utilization. I am running inside docker on a Ubuntu server with 8 cores/64GB of ram. Importing was slow too. Even for this small test run it is painfully slow with binarization taking 26min, but I'd potentially would want to process multiple 100k documents and it's just out of question given the speed. Is this expected or am I likely doing something wrong? Any tips would be much appreciated.
9 replies
- GPU for OCR(-D)
- What models of GPU are you using?
- What should we recommend for potential buyers?
- What GPU-benefitting processors are you using?
- Best Practices and experiences around efficient data transfer from/to GPU/CPU/RAM
- Techniques for efficient scaling (multi-device) and sharing (multi-kernel/client)
- specification/conventions for
TextEquiv/@indexin and outside of OCR-D, cf. OCR4all/LAREX#282
We meet at 10-11 am in a BBB room. At the beginning of each meeting, participants have the opportunity to suggest topics of interest to them, which can then be discussed and debated in small groups. Discussion points can also be entered in advance in the HedgeDoc. In addition to the Open TechCall, where the OCR-D community discusses technical topics, OCR(-D) & Co also offers participants without in-depth OCR-D knowledge the opportunity to contribute their own questions and ideas to the discussion and to openly exchange ideas with each other.
Meet @NWeichselbaumer & Christoph Reske from the OCR-D module project Font Group Recognition for Improved OCR and other interesting guests tomorrow and the day after tomorrow at (German language) expert talk Typometrische Daten in Drucken des 17. Jahrhunderts: https://www.hab.de/event/typometrische-daten/
Registration for online participation by e-mail at: beyer@hab.de / cboveland@hab.de
calamari-recognize does not run on GPU, even with
maximum-cudaIt should? Could you open a GitHub issue for that as well?
OCR-D/ocrd_calamari#68 @jbarth-ubhd
Feel free to join us in https://meet.gwdg.de/b/eli-ufa-unu !
@/all Join us tomorrow, 1–2 pm (Berlin Time) in our next OCR-D-GT-Call. Tomorrow's discussion will be focused on problems transcribing difficult cases. Link to our Big Blue Button: https://meet.gwdg.de/b/eli-ufa-unu
PS: You can easily look up upcomning events at our website now: https://ocr-d.de/en/community.html
frk or gt4hist (https://github.com/ulb-sachsen-anhalt/ulb-zeitungsprojekt-hp1)
3 replies
dinglehopper the OCR evaluation tool just got 50x faster!
Many thanks to @maxbachmann of the RapidFuzz string matching library who implemented all my API wishes (support for all hashable types and an implementation of editops()). This is particulary interesting for anyone using the python-Levenshtein library: python-Levenshtein is GPL-licensed and makes it incompatible with your own non-GPL license - while RapidFuzz is MIT licensed. (Other advantages of RapidFuzz: support for all hashable types and it's also actively maintained: I had a bug fixed and fix released within an hour!)
1 reply
@/all We would like to use this week's Open TechCall (Wednesday, 2–3pm Berlin Time) again to collect feedback:
- Documentation: What is missing, outdated or lacking in detail?
- CLI: What features of processor CLI you use most often, which seldom?
- CLI: Are there missing
ocrdsubcommands for tasks you do regularly? - CLI: How do you parallelize workflows? Do you know the
..operator andocrd workspace mergecommands? - Python API: Do you use it directly in scripts, e.g. instantiating a workspace or using the PAGE API? How could we improve it?
- Evaluation: How do you evaluate the results, what kind of tools and measures do you use?
Feel free to join us in https://meet.gwdg.de/b/eli-ufa-unu
In 2020 I built an updated Ubuntu (and Debian) package for olena using OCR-D's version, because it was painful to build Olena from source. Is there still interest in this? ocrd_olena seems to build it from source. I'm trying to figure out what to do with the package, e.g. setup up an APT repo or not. For myself, having the .deb online is good enough.
Ubuntu: https://qurator-data.de/~mike.gerber/olena_2.1-0+ocrd-git+1/
Debian: https://qurator-data.de/~mike.gerber/olena_2.1-0+ocrd-git+1-debian10/
dinglehopper the OCR evaluation tool just got 50x faster!
Many thanks to @maxbachmann of the RapidFuzz string matching library who implemented all my API wishes (support for all hashable types and an implementation of editops()). This is particulary interesting for anyone using the python-Levenshtein library: python-Levenshtein is GPL-licensed and makes it incompatible with your own non-GPL license - while RapidFuzz is MIT licensed. (Other advantages of RapidFuzz: support for all hashable types and it's also actively maintained: I had a bug fixed and fix released within an hour!)
Same author (@maxbachmann) seems to have taken up the maintenance of python-Levenshtein at https://github.com/maxbachmann/Levenshtein, which is also a great development. (But it still has the "GPL problem", of course)
1 reply
We meet at 10-11 am in a BBB room. At the beginning of each meeting, participants have the opportunity to suggest topics of interest to them, which can then be discussed and debated in small groups. Discussion points can also be entered in advance in the HedgeDoc. In addition to the Open TechCall, where the OCR-D community discusses technical topics, OCR(-D) & Co also offers participants without in-depth OCR-D knowledge the opportunity to contribute their own questions and ideas to the discussion and to openly exchange ideas with each other.
- PAGE Viewer in browse-ocrd
- Algorithms for evaluating reading order performance?
- Bugfix release of OCR-D/core
--location cwd, models are looked for directly in the current working directory, i.e. a resource <fname>, is looked for in <cwd>/<fname>, not in <cwd>/ocrd-resources/<processor>/<fname> as is the case for the other location options. While it has always been implemented this way, the spec have been corrected to reflect this.
4 replies
We just released a new minor version 2.28.0 of OCR-D/core, with these changes:
Added:
- Store parameterization of processors in METS for provenance, #747
ocrd workspace find --download: Add a--waitoption to wait between downloads, #745- bashlib: Check fileGrps when parsing CLI args, #743, OCR-D/ocrd_olena#76
- Dockerfile: Install
timeto have/usr/bin/timein the image, #748, OCR-D/ocrd_all#271
Fixed:
ocrd-dummy: Also set pcGtsId, v0.0.2, #739
Thanks to @bertsky for his contributions.
If you have any question or comments, feel free to respond here or in the release announcement discussion on GitHub.
A new release of ocrd_all will follow later today or tomorrow.
We now also released a new version v2021-11-30 of ocrd_all.
It includes the brand new tesseract 5.0.0, the latest OCR-D/core v2.28.0, as well as fixes to workflow_configuration and a OCR-D/ocrd_segment@bdc6771 in ocrd_segment.
Thanks to @bertsky, @stweil and all contributors!
The Docker images are being rebuilt and should be deployed and available in a few hours.
- Bi-weekly standup
- Releases v2.28.0 and v2.29.0
- Focus for next two weeks:
- Issues around resource manager and workspace tooling
- Increase Image.MAX_IMAGE_PIXELS - any reason against?
- Replace PIL with ImageMagick callout for pixel density
- Distributed workflows with makefiles (@anguelos)
- Consistent pytest-based unit tests (@m3ssman)
- Layout evaluation with ocrd_segment's ocrd-segment-evaluate (@bertsky)