Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 08:21
    Sanghyuk-Choi commented #1918
  • 08:20
    Sanghyuk-Choi commented #1918
  • May 13 15:45
    PosoSAgapo closed #2162
  • May 13 15:45
    PosoSAgapo closed #2163
  • May 13 13:01
    Ch3nYe closed #2148
  • May 12 06:22
    kagrze synchronize #2161
  • May 11 14:43
    PosoSAgapo edited #2163
  • May 11 10:11
    PosoSAgapo edited #2163
  • May 11 10:10
    PosoSAgapo edited #2163
  • May 11 10:04
    PosoSAgapo edited #2163
  • May 11 09:52
    PosoSAgapo opened #2163
  • May 10 12:38
    kagrze commented #2161
  • May 09 12:19
    kagrze commented #2161
  • May 09 09:58
    kagrze commented #2161
  • May 09 09:35
    kagrze commented #2161
  • May 08 12:25
    kagrze commented #2161
  • May 08 12:17
    kagrze commented #2161
  • May 08 12:09
    kagrze commented #2161
  • May 08 12:08
    kagrze commented #2161
  • May 08 12:07
    kagrze commented #2161
François Hernandez
@francoishernandez
This is probably what you're looking for: https://forum.opennmt.net/t/simple-opennmt-py-rest-server/1392
İshak DÖLEK
@ishakdolek
is it possible Char2Char translation with openNMT ?
François Hernandez
@francoishernandez
Not sure if you're referring to a specific model or architecture. If you just want to use a character level vocab rather than a word/subword level, you can probably just tokenize your data accordingly and run a classic pipeline.
İshak DÖLEK
@ishakdolek
@francoishernandez thank you.
Gaurav Ghati
@gauravghati
Hey all, I'm new to openNMT, and trying to run train the model by command given in README>me of data folder "python train.py -data data/data -save_model /n/rushlab/data/tmp -world_size 1 -gpu_ranks 0 -rnn_size 100 -word_vec_size 50 -layers 1 -train_steps 100 -optim adam -learning_rate 0.001", I'm getting acc 0, what should I do?
MinJune Kim
@qria
@qria
Hi, I'm a newbie trying to train a NMT model for Korean-English. I have a question about the amount of data it needs. Of course it won't be set in stone but I'm guessing you have a hunch of how much data it will need for it to be decent. I have 1.6M parallel sentence set. Will this be enough or do I have to scour the web for more?
Currently training with openNMT default setting (LSTM encoder/decoder w/ 500 hidden units) btw, probably will look into transformer models later
Vincent Nguyen
@vince62s
it's ok to start.
MinJune Kim
@qria
Thanks for the answer. I guess that means I'll need to focus on tuning the model because the result wasn't very impressive
cemyr
@cemyr_gitlab
Hi everyone, great to be here...
Ghost
@ghost~5ef9ef38d73408ce4fe8397f
Hi Geeks,
I'm going to preprocess data for handwritten Mathematcal expression recognization on images but i'm not understnading "--share_vocab, -share_vocab" this
please help me get idea about this argument.
Vincent Nguyen
@vince62s
This is only NMT type tasks. We share the source and target vocab. Not applicable to your task.
Isaac Ng
@isaacngcs
Hi, what arguments should I run onmt_preprocess with if I am training without parallel data?
Hobson Mokaya
@Atuti
Hi, how can I know that the training of the model is over. I have managed to preprocess the datasets and even run the train command, but I am not sure if the training is over. are there any files other than the '.vocab.pt', '.train.pt', and 'valid.pt', that I can check for to ascertain that the training is over.
François Hernandez
@francoishernandez
It should have saved some checkpoints using the base path you provided in -save_model. Also, you can check the logs of your training execution.
Hobson Mokaya
@Atuti
@francoishernandez thank you.
VedantParikh
@Vedant-06

Hi, I was using OpenNMT for English to Gujarati Translation. I used sentence piece transform for Gujarati Corpus. I trained the model for 70k sentences. But when I use translate.py for the output and printed out the verbose, I found that most of the subwords were being mapped to the <unk> tag. Eg.

"""
[2020-11-30 12:07:40,417 INFO]
SENT 1: ['but', 'some', 'really', 'good', 'work', 'is', 'being', 'done', 'around', 'us', ',', 'in', 'our', 'society', '.', 'and', 'all', 'this', 'has', 'been', 'possible', 'through', 'the', 'collective', 'efforts', 'of', 'a', '130', 'crore', 'countrymen', '.']
PRED 1: ▁છત <unk> <unk> ▁કરવા . . . . ▁સમગ્ર ▁ચ વ ર્ક ી સુર ક્ષ િત ▁અને ▁સમગ્ર ત ▁વધાર ો ▁છે ▁.
PRED SCORE: -30.2231
GOLD 1: <unk> <unk> <unk> <unk> <unk> <unk> <unk> આપણી <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> તે <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> .
GOLD SCORE: -674.3291

[2020-11-30 12:07:40,417 INFO]
SENT 2: ['this', 'can', 'pave', 'a', 'new', 'way', 'to', 'serve', 'the', 'purpose', 'of', 'ek', 'bharat', 'shresht', 'bharat', 'abhiyan', ',', 'ultimately', 'providing', 'a', 'great', 'boost', 'to', 'india’s', 'tourism', 'development', '.']
PRED 2: ▁સરપંચ ▁છું ▁ , ▁વિમુદ્રીકરણ ▁ <unk> ▁ , ▁ડેરી ▁ઉદ્યોગ નું ▁ઉદ્ઘાટન ▁કરવું ▁ , ▁ડેરી ઓ ▁ ,
PRED SCORE: -21.4394
GOLD 2: <unk> એક ભારત , શ્રેષ્ઠ ભારત <unk> <unk> પૂર્ણ <unk> <unk> માર્ગ <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> .
GOLD SCORE: -431.0326
"""

So is there any specific format, the input has to be given. Please guide me with this issue.
Thanks in advance.

Here GOLD Sentences are target gorund truth sentences.

VedantParikh
@Vedant-06
@francoishernandez
François Hernandez
@francoishernandez
Please do not multi-post. Discussion happening here: OpenNMT/OpenNMT-py#1954
Ze Xuan Ong
@ongzexuan
in the new data processing pipeline, what exactly does build-vocab do? Looking at the output, it seems like it maps vocab words to indices, but this mapping doesn't seem to be unique i.e. multiple words seem to share the same index
using the toy example in the quickstart, my example_vocab.src has the following snippet:
products 117 still 117 market 116 Hotel 116 see 115 great 114 today 114 through 114 against 112 rights 112 That 111 believe 111 therefore 110 under 110 case 109 same 109 But 109 offer 109
Ze Xuan Ong
@ongzexuan
ok never mind I should have noticed that the code wrote "Counter" and that these were counts and not indices
Megha Jain
@meghajain-1711
Hi everyone, I recently started exploring the OpenNMT-py library and was wondering if it supports splitting the dataset into train/valid/test sets or is it something we should preprocess earlier in our system ?
I understood we can specify it in the config file to be passed to build_vocab. Any help / link to appropriate doc is appreciated :)
François Hernandez
@francoishernandez
No it's not, you should do it before hand
Megha Jain
@meghajain-1711
Thanks @francoishernandez !
Megha Jain
@meghajain-1711
I had a few questions related to the REST Server as well. The server.py stated in the Documentation is showing as unavailable. Is there any upgrade in the server as well with the library upgrade ?
François Hernandez
@francoishernandez

I had a few questions related to the REST Server as well. The server.py stated in the Documentation is showing as unavailable. Is there any upgrade in the server as well with the library upgrade ?

I don't understand. Post an issue on the repo with exact command and error trace.

nashid
@nashid
preprocess.py does the tokenization step? OR the preprocessing steps lile splitting a given string by delimiters, multiple spaces, tabs are handled the preprocess.py?
François Hernandez
@francoishernandez
preprocess does not apply any tokenization, it just takes text data files and dump them in a more convenient format for training.
You need to apply tokenization before hand.
That being said, preprocess is no longer necessary since 2.0
nashid
@nashid
@francoishernandez ok. Can you please refer me to the last stable version with preprocess.py script?
Current "main" in the git repo does not have preprocess.py
nashid
@nashid
I was tryign to use https://github.com/OpenNMT/OpenNMT-py/tree/1.2.0 for last many hours

Note:

some features require Python 3.5 and after (eg: Distributed multigpu, entmax)
we currently only support PyTorch 1.4

So readme refers to "PyTorch 1.4" and setup.py is saying "torch>=1.6.0"
:-(
At the end, failing with " File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/site-packages/torch-1.8.1-py3.7-linux-x86_64.egg/torch/multiprocessing/reductions.py", line 312, in reduce_storage
metadata = storage._sharefilename()
RuntimeError: error executing torch_shm_manager at "/root/miniconda3/envs/open-nmt-env/lib/python3.7/site-packages/torch-1.8.1-py3.7-linux-x86_64.egg/torch/bin/torch_shm_manager" at /pytorch/torch/lib/libshm/core.cpp:99
"
nashid
@nashid
OpenNMT cannot find GPU resources. Any help is welcome.

I can see pytorch can find GPU resources.

>>> torch.cuda.is_available()
True
>>> torch.cuda.current_device()
0
>>> torch.cuda.device(0)
<torch.cuda.device object at 0x7f8c0a3cec50>
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'NVIDIA Tesla K80'
>>> torch.cuda.get_device_name(0)
'NVIDIA Tesla K80'

However, OpenNMT is not finding the GPU resource. What is going on here?
I am using OpenNMT from https://github.com/OpenNMT/OpenNMT-py/tree/1.2.0

Traceback (most recent call last):
  File "../../OpenNMT-py/train.py", line 6, in <module>
    main()
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 197, in main
    train(opt)
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 91, in train
    p.join()
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 181, in signal_handler
    raise Exception(msg)
Exception: 

-- Tracebacks above this line can probably
                 be ignored --

Traceback (most recent call last):
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 135, in run
    gpu_rank = onmt.utils.distributed.multi_init(opt, device_id)
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/utils/distributed.py", line 27, in multi_init
    world_size=dist_world_size, rank=opt.gpu_ranks[device_id])
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 510, in init_process_group
    timeout=timeout))
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 603, in _new_process_group_helper
    timeout)
RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!
François Hernandez
@francoishernandez
So readme refers to "PyTorch 1.4" and setup.py is saying "torch>=1.6.0"
Trust the latest information, which is setup.py -- the readme was probably not updated when upgrading for 1.6 support
For your GPU issue, I don't know, never seen it. Maybe create a detailed issue on github with your setup, commands, etc.
Peng Shi
@shipengAlan
Hi everyone. I try to do some experiments by customizing Transfomer. Does OpenNMT-py support this ? And how I can do that ? Thanks a lot
ANASUA BANERJEE
@anasua123:matrix.org
[m]
Hi everyone. I try to execute the preprocess part but it throws an error "No module named 'onmt.bin.preprocess'" and if i installed the omnt then it is showing another error . can anyone plz guide me how to resolve this one
François Hernandez
@francoishernandez
@anasua123:matrix.org the whole preprocess logic has been removed since 2.0. Check the docs for quickstart and some more comprehensive examples: https://opennmt.net/OpenNMT-py/quickstart.html
bcabgil
@bcabgil
Hello! Is it possible to build a text classification model from a pre trained bert sequence to sequence translation model using openNMT-py? Is there a way to get the outputs of a specific layer of the pre-trained model instead of the translated sequence?
Suzenne
@Suzenne
Why I got a very low bleu score (bleu1 is around 2.9 ) ,using an about 180 thousand en-zh corpus to train the model with the config is consist of quick-start and config-transformer-base-1GPU.yaml ? Is there something wrong ?
πόσο σ' αγαπώ.
@PosoSAgapo
Why the model behaves differently in training and tranlsation? Even translation on the training dataset cannot give the same performance when the model is training, like the model gets 70% on training, after training, using the model to translate the train set, the accuracy is just 30%.
Guillaume Klein
@guillaumekln