Is everything comparable? How much slower in your benchmark?
not everything, but the definition of slow are: 1) how fast the loss going down (in terms of number of steps to reach), the overall loss after ~50k steps still 1-ish something while t2t are 0.5-ish something. 2) how fast to stepping (per 100 steps), This one might 5x slower.
These measurement is oversimplified, however i just can feel it.
Actually there are still some odd things going on... Still investigating and might need to refine the behavior in a patch version. It's unclear what TensorFlow is doing under the hood.
Noted
The training loss is not comparable with Tensor2Tensor. They are subtracting a constant to offset the label smoothing.
For the time to do 100 steps, I would not expect it to be 5x slower (unless TF2 is 5x slower than TF1 which I did not find to be the case). Is it possible to compare tokens per second?
For the time to do 100 steps, I would not expect it to be 5x slower (unless TF2 is 5x slower than TF1 which I did not find to be the case). Is it possible to compare tokens per second?
score is usually used to score an existing prediction. You could define a custom model that extends the base SequenceClassifier, something like:class MyClassifier(onmt.models.SequenceClassifier):
def __init__(self):
super(MyClassifier, self).__init__(...)
def call(self, *args, **kwargs):
logits, _ = super(MyClassifier, self).super(*args, **kwargs)
predictions = dict(probs=tf.nn.softmax(logits))
return logits, predictions
def print_prediction(self, prediction, params=None, stream=None):
print(prediction["probs"], file=stream)
Hello, does the "train_alignments" work in opennmt-tf v2? when I add it in the config file (under data) I get this error :
When I comment it out, training runs fine. Thanks in advance
I checked but I see no error. This is a test train with 100 source and 100 target segments (in separate files), transformer model, sentencepiece, mixed precision. As for the alignments, I have used the OpenNMTTokenizer in conservative mode to tokenize them, pasted source/target with " ||| " delimiter, and run fast_align to produce the pharaoh format. The alignment file also comprises 100 lines.
Possibly the error lies on the different tokenization methods i.e. that the alignment (by fast_align) was run on the (same) training files but tokenized by OpenNMTTokenizer (conservative) while the training files, for training the model, are sent as raw text to sentencepiece inside OpenNMT-tf. Any ideas on this?
I understand. The thing is that tokenization of training files happens on-the-fly inside OpenNMT-tf which is more straightforward in my case, because when I do it offline, 1) I have to then de-tokenize myself 2) for protected tokens inside the ｟｠ I get funny encoding for these special parenthesis as we have previously discussed in http://forum.opennmt.net/t/issue-with-special-character-u-ff5f/2965/5
If you want to train with alignments you would need to tokenize offline. Did you look into using the OpenNMT Tokenizer directly? It is fast and have a convenient Python API. There should not be any encoding issues.
Thanks, I will look into it. So to clarify furher: 1) to train an NMT model with alignments, for the alignment I have to use the same type of tokenization that I do for the training, but I need to do it offline 2) For the training can I still keep the online tok method? or do I have to use the (offline) tokenized files? 3) Finally, does the same apply for word embeddings, if I want to train with embeddings as well?
One more doubt: suppose I use the offline tokenized files for the training. For inference, could I use the online tokenization (with the same tok method, obviously) to avoid tokenizing source and detokenizing the prediction? If yes, how to specifiy this ( that is, applying tok only to infererence) in the config.yml file? Could I specify "source_tokenization:" under "infer:" ?
You could however define the tokenization configuration in a separate file and use it only for inference using the Multiple configuration files approach
@guillaumekln Ah, OK, good point! I will try the multiple config option for inference. Thanks!
Also if the OpenNMT-tf model v2x is trained on GPU, can inference still be run on CPU-only?
I have a different question: While training, with omnt-tf v2x, I get this message in the log which does not hinder the training , however I wonder about its meaning/consequencies. Any idea?
I read that this is probably due to the fact that the number of iterations performed on the dataset is greater than the number of batches in the dataset. Is this a problem? Or should I ignore the message?
Hi, @guillaumekln, I meet the same problem that @emesha92 mentioned. The GPU utils is very low，most of the time 0%. sample_buffer_size is 500000, gpu num is 8, Accumulate gradients of 2 iterations to reach effective batch size of 25000. I wonder if the preparation of data is too slow?