BTW, I also noticed that you mentioned in a closed topic that 'You could instead split your file and run separate inference processes to leverage multiple GPUs.'. I also splitted the large data set to be 6 pieces, so each contains about 25m data entries. But starting the 2nd inference job on the same node threw exception and failed. Is there anything I missed which I should specify to leverage multiple GPUs to run inference?
Hello @guillaumekln, I wanted to ask about the effectiveness of transformer model on small dataset(200K). I have been training a small transformer model (1 layer, 512 dim, 4 heads). I am trying to extract amount, date from OCR text from receipts, so the source sentences are very long(~500 words) and target sentences are just one word. I have run the training for 100K iterations, but the loss seems very high (~1.5). Should I keep running it for longer? Is a transformer model as effective as an RNN on a small dataset like mine?
@yutongli Most likely the inference is running on GPU otherwise you would have a higher CPU usage. If you want to run multiple inference jobs, you should restrict the GPU visibility for each process with
@renjithsasidharan Hi. I see that you posted the same question on the forum. Let's continue the discussion there.
CUDA_VISIBLE_DEVICES (you'll find more info on Google). As for the original issue (killed job), is the memory usage increasing?@renjithsasidharan Hi. I see that you posted the same question on the forum. Let's continue the discussion there.
Hey @guillaumekln, I'm interested in understanding translation errors made by trained models, specifically by (1) looking at attention weights from transformer heads and (2) finding training examples with similar hidden state vectors (I can compute similarity myself, just need to know how to access the raw numbers at different parts of the network). Any advice? :)
You probably need to dive into the model code and place print statements when you need them. Just remember that the model is executed in graph mode so you need to use TensorFlow print function: https://www.tensorflow.org/api_docs/python/tf/print
@nslatysheva There is no such tutorial, but the code is not that big.
@guillaumekln Thanks very much! After some research, I managed to make the inference job to only run on CPUs, by controlling the GPU Visibility via Nvidia CUDA environment variable. (Now the CPU usage shows ~2700%, other than 150% previously. Also the GPU usage remains 0% per monitoring.) However, the inference output(predictions) does not seem to be dumped gradually and incrementally. It seems that the job keeps working hard behind the scene, holding output in memory for a very long time, without dumping at a regular pace. (Per my observation, the regular dumping happens in the last 2 hours before the job completed, given the duration of the entire job is about 30 hours). Can we specify any parameters to control the dumping during inference? if so, would that speed up the entire processing?
You can control this behavior but disabling it will actually make the overall decoding slower. See the parameter
infer > length_bucket_width in https://opennmt.net/OpenNMT-tf/configuration.html. It is set to 5 with auto_config but you can disable it with 0.
@guillaumekln I have trained a transformer model using opennmt-tf and want to serve it in production for real time inference. I am considering https://github.com/OpenNMT/CTranslate2, is Intel MKL the minimum requirement for building CTranslate2? If so and we end up with not being able to have CTranslate2 in production environment, would you please advise anything else, all i want to target is to bring the trained transformer model into production environment, so anything that can better the real time inference will be highly helpful! Thanks
I have trained a transformer model, Now I'm doing the inference. But the inference is stuck after translating some 20 sentences. what could be the problem ? has anyone faced this?. Plus it does not even throw any error. Its just stick after translating 20th sentence.
Hello, does OpenNMT-tf support protected sequences/placeholders i.e. ｟URL：http://www.opennmt.net｠as described in https://opennmt.net/OpenNMT/tools/tokenization/#special-characters? Provided that SP/BPE or unigram is deployed through OpenNTM tokenizer. Thanks in advance
Hi, I tried to tag (featured) both source and target languages with OpenNMT but didn't succeed. PyTorch just crashed at translation step (training was ok); and TensorFlow docs says that "target_inputter – A opennmt.inputters.Inputter to process the target data. Currently, only the opennmt.inputters.WordEmbedder is supported." which means (to me) that we can't tag the target. How could I try to tag both source and target with OpenNMT ? Is this possible ? May someone help me ?
Hi, tags are not supported on the target side in OpenNMT-tf. In OpenNMT-py, there is an open pull request for this feature: OpenNMT/OpenNMT-py#1710
Thanks @guillaumekln. In OpenNMT-py what puzzled me is that the training was perfect (at least no error) but at the translation step it seems that the crash was due to a non uniform word added by the system at the end of the sentence (looks like a end-of-sentence word that is of course? non featured).
Sure. Just run 2 separate training processes, each running on a different GPU. You can restrict the GPU visible to each process with this approach: https://opennmt.net/OpenNMT-tf/faq.html#how-can-i-restrict-the-tensorflow-runtime-to-specific-gpu
If you have more questions regarding CTranslate2, please post in https://gitter.im/OpenNMT/CTranslate2 or in the forum
Hello, I want to ask some question about embedding. I wonder how does the Embedding from WordEmbedder class trained? I looked at the source code but still don't understand. And is it possible to export the embedding from the trained WordEmbedder class into the pretrained word embedding format? Thanks in advance
Edit: As an additional information, I trained my model using the Transformer class
Edit: As an additional information, I trained my model using the Transformer class
Hi @guillaumekln I have trained a good transformer model but I saw some lengthy inference examples. for example, the source query is 10-characters long, but the generated inference by the model can be 200 characters, some portions are just repeating. Is there a way to control the length of the inference to be generated by the transformer model?
(BTW, I had the sequence_length set to 80 during training. ）
Hi. Look for
maximum_decoding_length in the parameters: https://opennmt.net/OpenNMT-tf/configuration.html. Maybe length_penalty can also help.
However, when this issue happens it usually means the input was unexpected for the model and the training may lack this type of examples.
Hi @guillaumekln , thanks so much for your response, it's very helpful!! Just a couple of quick follow-up questions: 1. I think the maximum_decoding_length and length_penalty are parameters affecting both training and inferencing, am I correct? 2. does the 'length' here refer to the number of characters, the number or subwords, or the number of words?
@guillaumekln for another parameter 'maximum_features_length', does the length here also refer to the number of tokens, as defined by the tokenization?