Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 21 18:29
    guillaumekln closed #458
  • Apr 21 15:05
    guillaumekln opened #458
  • Apr 21 14:55
    guillaumekln edited #457
  • Apr 21 14:55
    guillaumekln edited #457
  • Apr 21 14:54
    guillaumekln edited #457
  • Apr 21 14:53
    guillaumekln edited #457
  • Apr 21 14:52
    guillaumekln edited #457
  • Apr 21 14:51
    guillaumekln labeled #457
  • Apr 21 14:51
    guillaumekln opened #457
  • Apr 21 10:54
    guillaumekln closed #456
  • Apr 21 10:54
    guillaumekln commented #456
  • Apr 21 02:58
    PJ-Finlay opened #456
  • Apr 19 11:51
    guillaumekln commented #455
  • Apr 19 11:42
    WilliamVJacob opened #455
  • Apr 16 09:43
    guillaumekln closed #454
  • Apr 16 08:35
    guillaumekln opened #454
  • Apr 15 09:36
    guillaumekln closed #453
  • Apr 15 09:16
    guillaumekln opened #453
  • Apr 14 14:37
    guillaumekln closed #452
  • Apr 14 14:22
    guillaumekln synchronize #452
Santosh Boina
@santoshboina

(engineNMT) C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github\tests\data\models\v2>docker run -it --rm -v cd/data --entrypoint python opennmt/ctranslate2:1.6.1-ubuntu16-gpu
Python 2.7.12 (default, Oct 8 2019, 14:14:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import ctranslate2
translator = ctranslate2.Translator("/data/ende_ctranslate2/", device="cpu")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: failed to load the model /data/ende_ctranslate2//model.bin
translator = ctranslate2.Translator("C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github\tests\data\models\v2\aren-transliteration-i16", device="
cpu")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: failed to load the model C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github ests\data\models 2ren-transliteration-i16/model.bin

This sis the code I have tried , but i am getting an error saying failed to load the model
Guillaume Klein
@guillaumekln
Within the Docker image, the path should be /data/<MODEL> where <MODEL> is a directory in C:\Users\q1027036\OneDrive - IQVIA\Documents\customEngine_Inference\github\tests\data\models\v2
Santosh Boina
@santoshboina

Thanks Guillaume. Stil no luck with model inference.I am trying the following code as you mentioned:
Step 1:
docker run -it --rm -v /data/customEngine_Inference/github/tests/data/models/v2/aren-transliteration-i16/ --entrypoint python opennmt/ctranslate2:latest-ubuntu18
Step2: Python instance opened where

import ctranslate2
translator = ctranslate2.Translator("/data/ende_ctranslate2/", device="cpu")
RuntimeError: failed to load the model /data/ende_ctranslate2//model.bin

Guillaume Klein
@guillaumekln
Are you familiar with Docker volume mounting? I'm not sure you are using it right.
VishalKakkar
@VishalKakkar
Hi @guillaumekln does it support custom transformer model having different no of layer, heads as well as embedding and model size?
Guillaume Klein
@guillaumekln
Hi, yes it does. If you use the conversion tool that is integrated in the training framework (e.g. release_model in OpenNMT-py), these parameters will be correctly taken into account.
VishalKakkar
@VishalKakkar
@guillaumekln Thanks.
Santosh Boina
@santoshboina
Hi @guillaumekln , how do I get tokens/sec details for each inference i make using ctranslate2?
I am using --log_throughput as argument while passing converted model and test data as volume mount to docker image available in documentation. I am getting inference but I do not see any log details of tokens/sec. Please help me.
docker run --rm --volume "//c/Users/santoshboina/Documents/ctranslate_github/CTranslate2/python/convrtd_en_de:/data" opennmt/ctranslate2:latest-ubuntu18 --model /data/ --src "/data/input_en.txt" --tgt "/data/output5.txt" --log_throughput
Guillaume Klein
@guillaumekln
Santosh Boina
@santoshboina
I am using vs code ide, I am getting console messages when i miss a model or other arguments , but not able to see anything for log throughput argument.
Guillaume Klein
@guillaumekln
You should check in your client or Docker and see if it is not discarding the standard error output. I just checked, and the value is correctly displayed in the console at the end of the translation.
Santosh Boina
@santoshboina
I am new to docker usage, not sure what is overlooked. I have tried couple of ways but no luck with log_throughput.
likerainsun
@likerainsun
Hello, I am trying to build the docker image or pull the image, run it. But, for both ways of doing that, it says "missing model" and the container does not start. Has anyone experienced the same issue with me?
Guillaume Klein
@guillaumekln
How did you run the image? You should pass a model as shown in this example: https://github.com/OpenNMT/CTranslate2#with-the-translation-client
likerainsun
@likerainsun
@guillaumekln Thanks for the tip. I think I didn't pass the model.
chengduo
@chengduoZH
Hi @guillaumekln, I am curious about how do you generate the vocabulary map to accelerate the translation program further.
Guillaume Klein
@guillaumekln
chengduo
@chengduoZH
Thank you.
alex-bustamante
@alex-bustamante
hi, I used to pull the docker image to use it with singularity and it worked just fine up until version 1.5.1. Starting 1.6.0, whenever I try to execute (--help as example below) I get the following output: singularity run ctranslate2_1.6.1-centos7-gpu.sif --help
/.singularity.d/runscript: line 38: set: --: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
Has anyone had this problem before?
Guillaume Klein
@guillaumekln
Hi, sorry we removed the translation client in CentOS 7 images starting from version 1.6.1. Could you use a Ubuntu-based image instead?
alex-bustamante
@alex-bustamante
I can't as the server runs CentOS 7, I'm unable to go for that option. I guess I'll keep using 1.5.1 then. Thanks
Guillaume Klein
@guillaumekln
I'm not familiar with Singularity, but if it just runs the Docker image then the OS does not matter.
alex-bustamante
@alex-bustamante
the host's kernel version does matter with singularity. I guess it's a very particular case
chengduo
@chengduoZH
Hi Guillaume, Have you tested how much the speed has increased after setting CT2_USE_EXPERIMENTAL_PACKED_GEMM=1?
And how can I get the 'weight_packed' when converting the model to ct2?
chengduo
@chengduoZH

And how can I get the 'weight_packed' when converting the model to ct2?

I see, just set CT2_USE_EXPERIMENTAL_PACKED_GEMM=1.

Guillaume Klein
@guillaumekln
Hi, it does not always increase performance so you should try with your own setup (and yes, it's a runtime flag). I find this to help for single core decoding of models that are not too big.
chengduo
@chengduoZH
I tested the CT2 on a 24 core machine and set CT2_USE_EXPERIMENTAL_PACKED_GEMM=1, the result shows that the latency seems to be reduced by 18%.
Guillaume Klein
@guillaumekln
@alex-bustamante The latest version of the CentOS 7 images (1.11.0) include again the translation client.
Hobson Mokaya
@Atuti
I have no NVidia gpu on my laptop, so I use the cpu for training a model which relativley slow. I bumped into CTranslate2 this morning and learned that it could hasten the training, but I am unable to install.
I have installed docker, and an ubuntu docker image, but could not be able to continue installing ctranslate2.
I have openNMT-py on windows. Any help please?
Guillaume Klein
@guillaumekln
CTranslate2 only supports inference, not training.
Hobson Mokaya
@Atuti
Thank you @guillaumekln.
Hobson Mokaya
@Atuti
I am new to onmt, I have so far learnt that only NVIdia gpus are recommended. On my laptop I have an intel HD graphics 620, is there any chance that I can use that for training?
@guillaumekln
Guillaume Klein
@guillaumekln
No, I don't think you can train on Intel Graphics.
chengduo
@chengduoZH
hi @guillaumekln, it seems that there only have the performance on the Intel CPU in the readme doc, do you make benchmarks on AMD CPU?
Guillaume Klein
@guillaumekln
Hi. I did not run a full benchmark on AMD yet.
angelicgoyal
@angelicgoyal
Hi @guillaumekln , I'm getting varying scores for same words in the test data during batch translation, trying to resolve where might the problem be. Where do you think the issue might be at the inference or training end?
Guillaume Klein
@guillaumekln
Hi. Can you give an example? I'm not sure what you are referring to. Consider posting on the forum if there are many details: https://forum.opennmt.net/
angelicgoyal
@angelicgoyal
Here's an example @guillaumekln :

source [[‘v’, 'm', 'o', 'n']]
translate_batch [[{‘tokens': ['व', 'म', 'ो', 'न'], 'score': -2.462678909301758}, {'tokens': ['व', 'म', 'न'], 'score': -3.114733934402466}, {'tokens': ['व', 'ी', 'म', 'न'], 'score': -3.5103278160095215}]]

source [[‘v’, 'm', 'o', 'n'], ['v', 'm', 'o', 'n']]
translate_batch[[{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.4626777172088623}, {'tokens': ['व', 'म', 'न'], 'score': -3.1147348880767822}, {'tokens': ['व', 'ी', 'म', 'न'], 'score': -3.510328769683838}], [{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.457219123840332}, {'tokens': ['व', '्', 'म', 'न'], 'score': -2.4626777172088623}, {'tokens': ['व'], 'score': -3.2032346725463867}]]

source [[‘v’, 'm', 'o', 'n'], ['v', 'm', 'o', 'n'], ['v', 'm', 'o', 'n']]
translate_batch[[{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.4626779556274414}, {'tokens': ['व', 'म', 'न'], 'score': -3.114734411239624}, {'tokens': ['व', 'ी', 'म', 'न'], 'score': -3.5103282928466797}], [{'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.457218885421753}, {'tokens': ['व', '्', 'म', 'न'], 'score': -2.4626779556274414}, {'tokens': ['व'], 'score': -3.2032341957092285}], [{'tokens': ['व', '्', 'म', 'न'], 'score': -2.457218885421753}, {'tokens': ['व', 'म', 'ो', 'न'], 'score': -2.7263221740722656}, {'tokens': ['व'], 'score': -3.3992576599121094}]]

for the same output tokens, the scores fluctuate based on input's repetition in the batch call
Guillaume Klein
@guillaumekln
Can you try setting the environment variable MKL_CBWR=AUTO,STRICT, for example when running Python:
MKL_CBWR=AUTO,STRICT python ...
Guillaume Klein
@guillaumekln
@angelicgoyal Did this help solving your issue?
angelicgoyal
@angelicgoyal
No @guillaumekln it didn't work
Guillaume Klein
@guillaumekln
Is your example running on CPU or GPU?
angelicgoyal
@angelicgoyal
On CPU
Guillaume Klein
@guillaumekln
Is it an Intel CPU? I suppose not otherwise I'm pretty sure the flag above would fix the small numerical differences.
angelicgoyal
@angelicgoyal
yes it's intel based CPU. These differences are introduced on adding length/coverage penalties.