hm
may be you right.
But i start train task with this command: " CUDA_VISIBLE_DEVICES=0 onmt-main --model_type TransformerBig --config config.yml --auto_config train --num_gpus 1"
Ok. I see. Don't load cuda library. Thanks ^)
So strange. My PC go to in reboot, while in saving checkpoint phase on 2 video card. May be any ideas?
May be some problem in parallel tasks?
May be one dataset worse than second.
But may be we need remake some parameters )
Hello, I have trained OpenNMT-tf models with version 2.1.1 in a Linux/Ubuntu 18.04 GPU set up . I have now installed latest version 2.6.0 on a new VM which has CPU-only. In principle, is it possible to run inference from the latter CPU-only machine on the above-mentioned GPU trained models? Btw, when running inference, I do get a couple of error messages reg. missing libraries (please see screenshot) but I am not sure whether I should ignore them if I only use CPU
(Yes, you should ignore warnings about missing NVIDIA libraries when running on CPU)
Is this what you mean, or the contents of data.yml
Yes, I do. But this one refers to the averaged checkpoint of 27K steps (i.e. the 20K checkpoint file was overwritten by this one). But I decided to use the 20K checkpoint for the inference - is this OK?
Well, in https://github.com/OpenNMT/OpenNMT-tf I see that "The project is production-oriented and comes with backward compatibility guarantees." so please ignore my previous message
Hello again, for adding more train data sets to an existing model (checkpoint) and train it further, not necessarily for fine-tuning but rather for adding more parallel examples to the supervised learning, is the best practice to add the new train data sets to the config file (e.g. data.yml) or do update-vocab (merge) or both?
Also, once a model is trained and evaluated., and has met expectations, is it safe to add the excluded from the original training validation and test data sets?
1) I use SentencePiece, and I see your point. 2) So concatenate the new data to the existing src/tgt train files instead of adding the new files in the config 3) Yes, when testing of final model is through and results satisfactory, to use the test set as train data. I guess not the val set, since when retraining it is needed and should not be included in the train data.
Hello guys,
I'm currently working on a wrapper for the OpenNMT Tokenizer in order to use it as TensorFlow ops.
I wanted to know if :
a) is there any interest for it in the community ? it's still in an early phase, so it would be awesome if i could get some feedback on it
b) could i be able to publish the code and distribute it as a pip package ? since it embeds a custom build of the onmt tokenizer i wanted to ask before doing anything
Thanks !
I'm currently working on a wrapper for the OpenNMT Tokenizer in order to use it as TensorFlow ops.
I wanted to know if :
a) is there any interest for it in the community ? it's still in an early phase, so it would be awesome if i could get some feedback on it
b) could i be able to publish the code and distribute it as a pip package ? since it embeds a custom build of the onmt tokenizer i wanted to ask before doing anything
Thanks !
Hi Emmanuel,
a) I think there are some interests. We had similar plans in the past but this proved to be a bit complex to build and maintain over time. Other initiatives such as https://github.com/tensorflow/text could be an alternative for some use cases.
b) Sure. The Tokenizer is MIT licensed so you can do anything you want as long as you credit the original project.
Please keep me updated and let me know if there are any changes in the Tokenizer that would make your work easier.
Awesome ! Thanks for your reply Guillaume.
I just created the repo (https://github.com/eohana/tensorflow-onmttok-ops). Wheels are not yet published (i'll try to setup the ci pipeline asap), but for those who want to check it out, there's a Dockerfile provided for building the wheels.
To answer you about the tokenizer, i think that providing support for building in a Bazel environment could be a nice addition, as it could facilitate the ops integration when building TF Serving. I'll try to investigate myself, but i don't think it's a priority for now.
Feel free to give me you feedback, as for now i still consider it as a POC
Thanks for your help !
I just created the repo (https://github.com/eohana/tensorflow-onmttok-ops). Wheels are not yet published (i'll try to setup the ci pipeline asap), but for those who want to check it out, there's a Dockerfile provided for building the wheels.
To answer you about the tokenizer, i think that providing support for building in a Bazel environment could be a nice addition, as it could facilitate the ops integration when building TF Serving. I'll try to investigate myself, but i don't think it's a priority for now.
Feel free to give me you feedback, as for now i still consider it as a POC
Thanks for your help !
Hi Guillaume, I am training a transformer model(default config) with about 48million training examples. I am trying to figure a few things that I could not understand, could you please help? 1. some discussions above mentioned 'Accumulate gradients of x iterations to reach effective batch size of 25000', in my case it was 2 iterations. How do we interpret this? Is one iteration one step? 2. every 4600 steps, I saw a message 'tensorflow/core/kernels/data/shuffle_dataset_op.cc:143] Filling up shuffle buffer (this may take a while): 2689207 of 48195269', the shuffle usually takes 2-3 minutes, not a big deal in terms of time, but I am wondering can we control the frequency of shuffling, how 4600 steps was calculated for my case? 3. My original data set was 50m, I split it to 48m as training, 1m for dev, and 1m for test. evaluating 1m seems taking a long time, would you recommend use a smaller size for dev and test, for instance 100k or 50k, does that seem to be too less compared with the amount of training examples?