May be some problem in parallel tasks?
May be one dataset worse than second.
But may be we need remake some parameters )
Hello, I have trained OpenNMT-tf models with version 2.1.1 in a Linux/Ubuntu 18.04 GPU set up . I have now installed latest version 2.6.0 on a new VM which has CPU-only. In principle, is it possible to run inference from the latter CPU-only machine on the above-mentioned GPU trained models? Btw, when running inference, I do get a couple of error messages reg. missing libraries (please see screenshot) but I am not sure whether I should ignore them if I only use CPU
(Yes, you should ignore warnings about missing NVIDIA libraries when running on CPU)
Is this what you mean, or the contents of data.yml
Yes, I do. But this one refers to the averaged checkpoint of 27K steps (i.e. the 20K checkpoint file was overwritten by this one). But I decided to use the 20K checkpoint for the inference - is this OK?
Well, in https://github.com/OpenNMT/OpenNMT-tf I see that "The project is production-oriented and comes with backward compatibility guarantees." so please ignore my previous message
Hello again, for adding more train data sets to an existing model (checkpoint) and train it further, not necessarily for fine-tuning but rather for adding more parallel examples to the supervised learning, is the best practice to add the new train data sets to the config file (e.g. data.yml) or do update-vocab (merge) or both?
Also, once a model is trained and evaluated., and has met expectations, is it safe to add the excluded from the original training validation and test data sets?
1) I use SentencePiece, and I see your point. 2) So concatenate the new data to the existing src/tgt train files instead of adding the new files in the config 3) Yes, when testing of final model is through and results satisfactory, to use the test set as train data. I guess not the val set, since when retraining it is needed and should not be included in the train data.
Hello guys,
I'm currently working on a wrapper for the OpenNMT Tokenizer in order to use it as TensorFlow ops.
I wanted to know if :
a) is there any interest for it in the community ? it's still in an early phase, so it would be awesome if i could get some feedback on it
b) could i be able to publish the code and distribute it as a pip package ? since it embeds a custom build of the onmt tokenizer i wanted to ask before doing anything
Thanks !
I'm currently working on a wrapper for the OpenNMT Tokenizer in order to use it as TensorFlow ops.
I wanted to know if :
a) is there any interest for it in the community ? it's still in an early phase, so it would be awesome if i could get some feedback on it
b) could i be able to publish the code and distribute it as a pip package ? since it embeds a custom build of the onmt tokenizer i wanted to ask before doing anything
Thanks !
Hi Emmanuel,
a) I think there are some interests. We had similar plans in the past but this proved to be a bit complex to build and maintain over time. Other initiatives such as https://github.com/tensorflow/text could be an alternative for some use cases.
b) Sure. The Tokenizer is MIT licensed so you can do anything you want as long as you credit the original project.
Please keep me updated and let me know if there are any changes in the Tokenizer that would make your work easier.
Awesome ! Thanks for your reply Guillaume.
I just created the repo (https://github.com/eohana/tensorflow-onmttok-ops). Wheels are not yet published (i'll try to setup the ci pipeline asap), but for those who want to check it out, there's a Dockerfile provided for building the wheels.
To answer you about the tokenizer, i think that providing support for building in a Bazel environment could be a nice addition, as it could facilitate the ops integration when building TF Serving. I'll try to investigate myself, but i don't think it's a priority for now.
Feel free to give me you feedback, as for now i still consider it as a POC
Thanks for your help !
I just created the repo (https://github.com/eohana/tensorflow-onmttok-ops). Wheels are not yet published (i'll try to setup the ci pipeline asap), but for those who want to check it out, there's a Dockerfile provided for building the wheels.
To answer you about the tokenizer, i think that providing support for building in a Bazel environment could be a nice addition, as it could facilitate the ops integration when building TF Serving. I'll try to investigate myself, but i don't think it's a priority for now.
Feel free to give me you feedback, as for now i still consider it as a POC
Thanks for your help !
Hi Guillaume, I am training a transformer model(default config) with about 48million training examples. I am trying to figure a few things that I could not understand, could you please help? 1. some discussions above mentioned 'Accumulate gradients of x iterations to reach effective batch size of 25000', in my case it was 2 iterations. How do we interpret this? Is one iteration one step? 2. every 4600 steps, I saw a message 'tensorflow/core/kernels/data/shuffle_dataset_op.cc:143] Filling up shuffle buffer (this may take a while): 2689207 of 48195269', the shuffle usually takes 2-3 minutes, not a big deal in terms of time, but I am wondering can we control the frequency of shuffling, how 4600 steps was calculated for my case? 3. My original data set was 50m, I split it to 48m as training, 1m for dev, and 1m for test. evaluating 1m seems taking a long time, would you recommend use a smaller size for dev and test, for instance 100k or 50k, does that seem to be too less compared with the amount of training examples?
Hi,
- Maybe iterations is a bit misleading here, but in "Accumulate gradients of x iterations to reach effective batch size of 25000": x iterations = x batches = 1 training step. Does that make sense?
- The parameter
sample_buffer_sizecan be used to configure the shuffle buffer size. Small values mean faster filling but worse shuffling of the training data, while large values mean slower filling but improved shuffling. The default buffer size is the dataset size. - You don't need more than 5k sentences in your dev and test sets.
Sure, thanks so much for your response! for #1, the default effective size was 25000 and batch_size was 3072 tokens, so if it says 'Accumulate gradients of 2 iterations to reach effective batch size of 25000', that means 2 batches = 25000 tokens, and then 1 batch = 12500 tokens. regardless of the batch_size 3072 in the config?
for#2, every 4600 steps, it logs 'filling up shuffle buffer', does that mean 4600 steps is one epoch of data in my case?
for #3, thanks so much. i will use smaller size for dev and test to speed up the training
- If you are training with N GPUs you are also processing N batches in parallel. So we want: 3072 * N * x >= 25000 where x in the number of accumulated iterations.
- Not necessarily an epoch but a point where the TensorFlow runtime determines that it should fill the buffer with more data.
Can you please give me example on how to employ my tokanized data and how to use it with transformer model?
Hi! You can find more information on the default file format here: https://opennmt.net/OpenNMT-tf/data.html#text