OpenNMT/CTranslate2

Santosh Boina

@santoshboina

(engineNMT) C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github\tests\data\models\v2>docker run -it --rm -v cd/data --entrypoint python opennmt/ctranslate2:1.6.1-ubuntu16-gpu
Python 2.7.12 (default, Oct 8 2019, 14:14:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import ctranslate2
translator = ctranslate2.Translator("/data/ende_ctranslate2/", device="cpu")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: failed to load the model /data/ende_ctranslate2//model.bin
translator = ctranslate2.Translator("C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github\tests\data\models\v2\aren-transliteration-i16", device="
cpu")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: failed to load the model C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github ests\data\models2ren-transliteration-i16/model.bin

This sis the code I have tried , but i am getting an error saying failed to load the model

Guillaume Klein

@guillaumekln

Within the Docker image, the path should be /data/<MODEL> where <MODEL> is a directory in C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github\tests\data\models\v2

Santosh Boina

@santoshboina

Thanks Guillaume. Stil no luck with model inference.I am trying the following code as you mentioned:
Step 1:
docker run -it --rm -v /data/customEngine_Inference/github/tests/data/models/v2/aren-transliteration-i16/ --entrypoint python opennmt/ctranslate2:latest-ubuntu18
Step2: Python instance opened where

import ctranslate2
translator = ctranslate2.Translator("/data/ende_ctranslate2/", device="cpu")
RuntimeError: failed to load the model /data/ende_ctranslate2//model.bin

Guillaume Klein

@guillaumekln

Are you familiar with Docker volume mounting? I'm not sure you are using it right.

VishalKakkar

@VishalKakkar

Hi @guillaumekln does it support custom transformer model having different no of layer, heads as well as embedding and model size?

Guillaume Klein

@guillaumekln

Hi, yes it does. If you use the conversion tool that is integrated in the training framework (e.g. release_model in OpenNMT-py), these parameters will be correctly taken into account.

VishalKakkar

@VishalKakkar

@guillaumekln Thanks.

Santosh Boina

@santoshboina

Hi @guillaumekln , how do I get tokens/sec details for each inference i make using ctranslate2?
I am using --log_throughput as argument while passing converted model and test data as volume mount to docker image available in documentation. I am getting inference but I do not see any log details of tokens/sec. Please help me.
docker run --rm --volume "//c/Users/santoshboina/Documents/ctranslate_github/CTranslate2/python/convrtd_en_de:/data" opennmt/ctranslate2:latest-ubuntu18 --model /data/ --src "/data/input_en.txt" --tgt "/data/output5.txt" --log_throughput

Guillaume Klein

@guillaumekln

It is outputted on the standard error: https://github.com/OpenNMT/CTranslate2/blob/master/cli/translate.cc#L122

Santosh Boina

@santoshboina

I am using vs code ide, I am getting console messages when i miss a model or other arguments , but not able to see anything for log throughput argument.

Guillaume Klein

@guillaumekln

You should check in your client or Docker and see if it is not discarding the standard error output. I just checked, and the value is correctly displayed in the console at the end of the translation.

Santosh Boina

@santoshboina

I am new to docker usage, not sure what is overlooked. I have tried couple of ways but no luck with log_throughput.

likerainsun

@likerainsun

Hello, I am trying to build the docker image or pull the image, run it. But, for both ways of doing that, it says "missing model" and the container does not start. Has anyone experienced the same issue with me?

Guillaume Klein

@guillaumekln

How did you run the image? You should pass a model as shown in this example: https://github.com/OpenNMT/CTranslate2#with-the-translation-client

likerainsun

@likerainsun

@guillaumekln Thanks for the tip. I think I didn't pass the model.

chengduo

@chengduoZH

Hi @guillaumekln, I am curious about how do you generate the vocabulary map to accelerate the translation program further.

Guillaume Klein

@guillaumekln

Hi. There is a link at the very bottom of the README: https://github.com/OpenNMT/CTranslate2/blob/master/README.md#how-do-i-generate-a-vocabulary-mapping-file

chengduo

@chengduoZH

Thank you.

alex-bustamante

@alex-bustamante

hi, I used to pull the docker image to use it with singularity and it worked just fine up until version 1.5.1. Starting 1.6.0, whenever I try to execute (--help as example below) I get the following output: singularity run ctranslate2_1.6.1-centos7-gpu.sif --help
/.singularity.d/runscript: line 38: set: --: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]

Has anyone had this problem before?

Guillaume Klein

@guillaumekln

Hi, sorry we removed the translation client in CentOS 7 images starting from version 1.6.1. Could you use a Ubuntu-based image instead?

alex-bustamante

@alex-bustamante

I can't as the server runs CentOS 7, I'm unable to go for that option. I guess I'll keep using 1.5.1 then. Thanks

Guillaume Klein

@guillaumekln

I'm not familiar with Singularity, but if it just runs the Docker image then the OS does not matter.

alex-bustamante

@alex-bustamante

the host's kernel version does matter with singularity. I guess it's a very particular case

chengduo

@chengduoZH

Hi Guillaume, Have you tested how much the speed has increased after setting CT2_USE_EXPERIMENTAL_PACKED_GEMM=1?

And how can I get the 'weight_packed' when converting the model to ct2?

chengduo

@chengduoZH

And how can I get the 'weight_packed' when converting the model to ct2?

I see, just set CT2_USE_EXPERIMENTAL_PACKED_GEMM=1.

Guillaume Klein

@guillaumekln

Hi, it does not always increase performance so you should try with your own setup (and yes, it's a runtime flag). I find this to help for single core decoding of models that are not too big.

chengduo

@chengduoZH

I tested the CT2 on a 24 core machine and set CT2_USE_EXPERIMENTAL_PACKED_GEMM=1, the result shows that the latency seems to be reduced by 18%.

Guillaume Klein

@guillaumekln

@alex-bustamante The latest version of the CentOS 7 images (1.11.0) include again the translation client.

Hobson Mokaya

@Atuti

I have no NVidia gpu on my laptop, so I use the cpu for training a model which relativley slow. I bumped into CTranslate2 this morning and learned that it could hasten the training, but I am unable to install.
I have installed docker, and an ubuntu docker image, but could not be able to continue installing ctranslate2.
I have openNMT-py on windows. Any help please?

Guillaume Klein

@guillaumekln

CTranslate2 only supports inference, not training.

Hobson Mokaya

@Atuti

Thank you @guillaumekln.

Hobson Mokaya

@Atuti

I am new to onmt, I have so far learnt that only NVIdia gpus are recommended. On my laptop I have an intel HD graphics 620, is there any chance that I can use that for training?
@guillaumekln

Guillaume Klein

@guillaumekln

No, I don't think you can train on Intel Graphics.

chengduo

@chengduoZH

hi @guillaumekln, it seems that there only have the performance on the Intel CPU in the readme doc, do you make benchmarks on AMD CPU?

Guillaume Klein

@guillaumekln

Hi. I did not run a full benchmark on AMD yet.

angelicgoyal

@angelicgoyal

Hi @guillaumekln , I'm getting varying scores for same words in the test data during batch translation, trying to resolve where might the problem be. Where do you think the issue might be at the inference or training end?

Guillaume Klein

@guillaumekln

Hi. Can you give an example? I'm not sure what you are referring to. Consider posting on the forum if there are many details: https://forum.opennmt.net/

angelicgoyal

@angelicgoyal

Here's an example @guillaumekln :

source [[‘v’, 'm', 'o', 'n']]
translate_batch [[{‘tokens': ['व', 'म', 'ो', 'न'], 'score': -2.462678909301758}, {'tokens': ['व', 'म', 'न'], 'score': -3.114733934402466}, {'tokens': ['व', 'ी', 'म', 'न'], 'score': -3.5103278160095215}]]

source [[‘v’, 'm', 'o', 'n'], ['v', 'm', 'o', 'n']]
translate_batch[[{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.4626777172088623}, {'tokens': ['व', 'म', 'न'], 'score': -3.1147348880767822}, {'tokens': ['व', 'ी', 'म', 'न'], 'score': -3.510328769683838}], [{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.457219123840332}, {'tokens': ['व', '्', 'म', 'न'], 'score': -2.4626777172088623}, {'tokens': ['व'], 'score': -3.2032346725463867}]]

source [[‘v’, 'm', 'o', 'n'], ['v', 'm', 'o', 'n'], ['v', 'm', 'o', 'n']]
translate_batch[[{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.4626779556274414}, {'tokens': ['व', 'म', 'न'], 'score': -3.114734411239624}, {'tokens': ['व', 'ी', 'म', 'न'], 'score': -3.5103282928466797}], [{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.457218885421753}, {'tokens': ['व', '्', 'म', 'न'], 'score': -2.4626779556274414}, {'tokens': ['व'], 'score': -3.2032341957092285}], [{'tokens': ['व', '्', 'म', 'न'], 'score': -2.457218885421753}, {'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.7263221740722656}, {'tokens': ['व'], 'score': -3.3992576599121094}]]

for the same output tokens, the scores fluctuate based on input's repetition in the batch call

Guillaume Klein

@guillaumekln

Can you try setting the environment variable MKL_CBWR=AUTO,STRICT, for example when running Python:

MKL_CBWR=AUTO,STRICT python ...

Guillaume Klein

@guillaumekln

@angelicgoyal Did this help solving your issue?

angelicgoyal

@angelicgoyal

No @guillaumekln it didn't work

Guillaume Klein

@guillaumekln

Is your example running on CPU or GPU?

angelicgoyal

@angelicgoyal

On CPU

Guillaume Klein

@guillaumekln

Is it an Intel CPU? I suppose not otherwise I'm pretty sure the flag above would fix the small numerical differences.

angelicgoyal

@angelicgoyal

yes it's intel based CPU. These differences are introduced on adding length/coverage penalties.

Where communities thrive

People

Repo info

Activity