I'd have to make the classifier support it
Interesting note though, training the classifier on huge chunks of text takes an exorbitant amount of time, but training it on smaller chunks is extremely fast.
At first I tried to do
classifier.train on the entire courpus for each of the languages and it was taking hours with no end in sight. But I just now tried training line by line instead and it finished in minutes.
Too bad I can't spell
I would love to figure out a way to multithread the workflow and have one progress bar per thread, but idk if it's possible
188 languages
I don't see why not, don't apt-get do this when dowloading several packages. ?
Anyway, it's so cool to train your own models with your own code. Congrats !
That's why
PR ready for
cadmiumcr/classifier and one waiting for cadmiumcr/distance :-)
the one you used in your POC language detector
I'm going to modify the
Case tokenizer
Definitely a little slower though
I guess there were some communication missteps because I have a question and mixed feelings about this : Why was it directly moved to a new
You set up a great way to discuss additions/changes to the cadmiumcr ecosystem : the
Concerning the language identification module, what are your feelings, plans, thoughts ?
To be clear : if you're not satisfied with
Just say it ;-)
cadmiumcr repo (without discussing it first) ?You set up a great way to discuss additions/changes to the cadmiumcr ecosystem : the
cadmiumcr/rfcs. Why not use it ?Concerning the language identification module, what are your feelings, plans, thoughts ?
To be clear : if you're not satisfied with
cadmium_language_detector, I'm open to replace it with you solution, wrap our algos as engines to a common API, etc.Just say it ;-)
Yeah I suppose it could've been RFCd, but I just wanted something more accurate and different from what currently exists (since I haven't actually seen any language detectors made using a classifer; they all rely on algorithms that are pretty finicky). The
cadmium_language_detector project is cool, and we can probably do some API merging, but personally I'd like to go with whichever is more accurate for the actual algorithm.
I'm planning on writing some tests tomorrow, but in my testing so far
cadmium_lang is pretty dang accurate, even with small text samples
It will be useful to evaluate some algos (Language identification, POS tagging, sentiment analysis) and let our users check for themselves the accuracy of our tools
Forgot about that haha. Feel free to submit a proposal :)