Hello, does the "train_alignments" work in opennmt-tf v2? when I add it in the config file (under data) I get this error :
When I comment it out, training runs fine. Thanks in advance
I checked but I see no error. This is a test train with 100 source and 100 target segments (in separate files), transformer model, sentencepiece, mixed precision. As for the alignments, I have used the OpenNMTTokenizer in conservative mode to tokenize them, pasted source/target with " ||| " delimiter, and run fast_align to produce the pharaoh format. The alignment file also comprises 100 lines.
Possibly the error lies on the different tokenization methods i.e. that the alignment (by fast_align) was run on the (same) training files but tokenized by OpenNMTTokenizer (conservative) while the training files, for training the model, are sent as raw text to sentencepiece inside OpenNMT-tf. Any ideas on this?
I understand. The thing is that tokenization of training files happens on-the-fly inside OpenNMT-tf which is more straightforward in my case, because when I do it offline, 1) I have to then de-tokenize myself 2) for protected tokens inside the ｟｠ I get funny encoding for these special parenthesis as we have previously discussed in http://forum.opennmt.net/t/issue-with-special-character-u-ff5f/2965/5
If you want to train with alignments you would need to tokenize offline. Did you look into using the OpenNMT Tokenizer directly? It is fast and have a convenient Python API. There should not be any encoding issues.
Thanks, I will look into it. So to clarify furher: 1) to train an NMT model with alignments, for the alignment I have to use the same type of tokenization that I do for the training, but I need to do it offline 2) For the training can I still keep the online tok method? or do I have to use the (offline) tokenized files? 3) Finally, does the same apply for word embeddings, if I want to train with embeddings as well?
One more doubt: suppose I use the offline tokenized files for the training. For inference, could I use the online tokenization (with the same tok method, obviously) to avoid tokenizing source and detokenizing the prediction? If yes, how to specifiy this ( that is, applying tok only to infererence) in the config.yml file? Could I specify "source_tokenization:" under "infer:" ?
You could however define the tokenization configuration in a separate file and use it only for inference using the Multiple configuration files approach
@guillaumekln Ah, OK, good point! I will try the multiple config option for inference. Thanks!
Also if the OpenNMT-tf model v2x is trained on GPU, can inference still be run on CPU-only?
I have a different question: While training, with omnt-tf v2x, I get this message in the log which does not hinder the training , however I wonder about its meaning/consequencies. Any idea?
I read that this is probably due to the fact that the number of iterations performed on the dataset is greater than the number of batches in the dataset. Is this a problem? Or should I ignore the message?
Hi, @guillaumekln, I meet the same problem that @emesha92 mentioned. The GPU utils is very low，most of the time 0%. sample_buffer_size is 500000, gpu num is 8, Accumulate gradients of 2 iterations to reach effective batch size of 25000. I wonder if the preparation of data is too slow?
@newger18 This sounds odd. Are you training a Transformer model? What is the reported "words per second"?
Thanks so much! @guillaumekln
last question, If I want to change configuration. Can I put all configuration in data.yml ?? like this?
"!onmt-main --model_type Transformer --config data.yml --auto_config train --with_eval"
In this way it will take auto config or the configs in my data.yml file?
last question, If I want to change configuration. Can I put all configuration in data.yml ?? like this?
"!onmt-main --model_type Transformer --config data.yml --auto_config train --with_eval"
In this way it will take auto config or the configs in my data.yml file?
Hello i try to train TransformerBig model. WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 5008 vs previous value: 5008. And get warning You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
What can be it?
OK, so I resolved most of my issues and even got the output, but here's the thing. Apparently TF's bindings for Go doesn't support/implement
GatherTree which means I have to set beam_width to 1 when exporting my model. The thing is that the result of beam_width set to 1 is utter nonsense compared to when beam_width is set to say 5.
Any suggestions/tips/resources would be appreciated. Thanks
@guillaumekln Thanks for the answer man. Actually I'm at 10k steps of training, but you saying that reaching the end of training makes it better was a relief, because I was thinking about implementing
GatherTree myself, which not only is beyond my expertise, I was afraid that that wasn't the only operation missing from Go's bindings.
Hello. I train model with opennmt-tf 1.24.0 version. And try to serving this model by tensorflow/serving:2.1.0-rc1-gpu. And get this kind error. What can be it?
2020-01-08 09:57:52.021820: E external/org_tensorflow/tensorflow/core/grappler/optimizers/meta_optimizer.cc:561] function_optimizer failed: Not found: Op type not registered 'Addons>GatherTree' in binary running on model-serve-dev-fast-658d8d96b9-vqmdt. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.)
2020-01-08 09:57:52.132840: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at partitioned_function_ops.cc:113 : Not found: Op type not registered 'Addons>GatherTree' in binary running on model-serve-dev-fast-658d8d96b9-vqmdt. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.)
2020-01-08 09:57:52.021820: E external/org_tensorflow/tensorflow/core/grappler/optimizers/meta_optimizer.cc:561] function_optimizer failed: Not found: Op type not registered 'Addons>GatherTree' in binary running on model-serve-dev-fast-658d8d96b9-vqmdt. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.)
tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.2020-01-08 09:57:52.132840: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at partitioned_function_ops.cc:113 : Not found: Op type not registered 'Addons>GatherTree' in binary running on model-serve-dev-fast-658d8d96b9-vqmdt. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.)
tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed