tldr-pages/tldr

I'll try to keep this concise to it's not an essay. 😆

OPUS is a website that hosts free public datasets for mapping natural language. (i.e. translations)
These datasets vary in size, though some are quite small.
They are used mostly for aiding projects that need large amounts of translation data.

Two projects I like called Argos Translate and LibreTranslate use these datasets to train models for machine translations, and made a cloud service which is comparable to Google Translate, except it's free, open-source, self-hostable, and works offline.

I would like to generate a dataset which matches the schema for OPUS, using tl;dr as the source. Then this can be used to train models as well by Argos Translate.

My request was, if I cleaned up the script I made above, could we commit that to the repo and actually make the dataset an artefact in CI (GitHub Actions) as well, similar to the tldr PDF/book?

If not, I'll just provide the script and dataset to OPUS directly.

The way the script works, it reads all English pages, and finds where we have a translated page.
Then it parses the markdown, and creates pairs of command descriptions to commands. (During the clean up, I should have it process the overall page description as well, currently only taking command descriptions.)

When comparing descriptions between languages, I use the command as an ID so we know we're getting the correct thing.
Since our translations can be inconsistent, i.e. the order changes or commands are added/removed without the translation being updated, we can't just assume everything matches out of the box.

Then we remove all translatable elements from commands, i.e. {{...}} so we can use the command as a nice ID. (because we translate {{path/to/file}}, for example)
We then remove all descriptions/commands that are duplicates, since we have no way of 100% selecting the right translation to map it too.

Then find the same command between the English page, and the translated page, and save the description to the dataset.