score is usually used to score an existing prediction. You could define a custom model that extends the base SequenceClassifier, something like:class MyClassifier(onmt.models.SequenceClassifier):
def __init__(self):
super(MyClassifier, self).__init__(...)
def call(self, *args, **kwargs):
logits, _ = super(MyClassifier, self).super(*args, **kwargs)
predictions = dict(probs=tf.nn.softmax(logits))
return logits, predictions
def print_prediction(self, prediction, params=None, stream=None):
print(prediction["probs"], file=stream)
Hello, does the "train_alignments" work in opennmt-tf v2? when I add it in the config file (under data) I get this error :
When I comment it out, training runs fine. Thanks in advance
I checked but I see no error. This is a test train with 100 source and 100 target segments (in separate files), transformer model, sentencepiece, mixed precision. As for the alignments, I have used the OpenNMTTokenizer in conservative mode to tokenize them, pasted source/target with " ||| " delimiter, and run fast_align to produce the pharaoh format. The alignment file also comprises 100 lines.
Possibly the error lies on the different tokenization methods i.e. that the alignment (by fast_align) was run on the (same) training files but tokenized by OpenNMTTokenizer (conservative) while the training files, for training the model, are sent as raw text to sentencepiece inside OpenNMT-tf. Any ideas on this?
I understand. The thing is that tokenization of training files happens on-the-fly inside OpenNMT-tf which is more straightforward in my case, because when I do it offline, 1) I have to then de-tokenize myself 2) for protected tokens inside the ｟｠ I get funny encoding for these special parenthesis as we have previously discussed in http://forum.opennmt.net/t/issue-with-special-character-u-ff5f/2965/5
If you want to train with alignments you would need to tokenize offline. Did you look into using the OpenNMT Tokenizer directly? It is fast and have a convenient Python API. There should not be any encoding issues.
Thanks, I will look into it. So to clarify furher: 1) to train an NMT model with alignments, for the alignment I have to use the same type of tokenization that I do for the training, but I need to do it offline 2) For the training can I still keep the online tok method? or do I have to use the (offline) tokenized files? 3) Finally, does the same apply for word embeddings, if I want to train with embeddings as well?
One more doubt: suppose I use the offline tokenized files for the training. For inference, could I use the online tokenization (with the same tok method, obviously) to avoid tokenizing source and detokenizing the prediction? If yes, how to specifiy this ( that is, applying tok only to infererence) in the config.yml file? Could I specify "source_tokenization:" under "infer:" ?
You could however define the tokenization configuration in a separate file and use it only for inference using the Multiple configuration files approach
@guillaumekln Ah, OK, good point! I will try the multiple config option for inference. Thanks!
Also if the OpenNMT-tf model v2x is trained on GPU, can inference still be run on CPU-only?
I have a different question: While training, with omnt-tf v2x, I get this message in the log which does not hinder the training , however I wonder about its meaning/consequencies. Any idea?
I read that this is probably due to the fact that the number of iterations performed on the dataset is greater than the number of batches in the dataset. Is this a problem? Or should I ignore the message?
Hi, @guillaumekln, I meet the same problem that @emesha92 mentioned. The GPU utils is very low，most of the time 0%. sample_buffer_size is 500000, gpu num is 8, Accumulate gradients of 2 iterations to reach effective batch size of 25000. I wonder if the preparation of data is too slow?
@newger18 This sounds odd. Are you training a Transformer model? What is the reported "words per second"?
Thanks so much! @guillaumekln
last question, If I want to change configuration. Can I put all configuration in data.yml ?? like this?
"!onmt-main --model_type Transformer --config data.yml --auto_config train --with_eval"
In this way it will take auto config or the configs in my data.yml file?
last question, If I want to change configuration. Can I put all configuration in data.yml ?? like this?
"!onmt-main --model_type Transformer --config data.yml --auto_config train --with_eval"
In this way it will take auto config or the configs in my data.yml file?
Hello i try to train TransformerBig model. WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 5008 vs previous value: 5008. And get warning You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
What can be it?
OK, so I resolved most of my issues and even got the output, but here's the thing. Apparently TF's bindings for Go doesn't support/implement
GatherTree which means I have to set beam_width to 1 when exporting my model. The thing is that the result of beam_width set to 1 is utter nonsense compared to when beam_width is set to say 5.