OpenNMT/openmt

Discussion about OpenNMT (deprecated, use OpenNMT-py instead)

chiting765

@chiting765

Also, -replace_unk just doesn't work for me and I don't know why

Konstantin Glushak

@gsoul

As for single quote backslash should also work. But if it doesn’t, please paste your JSON on some site like https://pastebin.com/ and give us a link to it. Thank you.

As for unk, what model did you train? What data did you use for it? And for how long did you train it, using what hardware?

Konstantin Glushak

@gsoul

@guillaumekln now I used only Lua version of tokenizer/learn_bpe + python “create_vocab” and I got this: https://drive.google.com/file/d/1Iwu0CvU48WzYfl1M6_LkKbtc8RvN-ymQ/view?usp=sharing

14323 tokens instead of 32k.

Guillaume Klein

@guillaumekln

Is this with the latest version of the Lua scripts? Also, could you try without changing the joiner marker? It should not be an issue but it wasn't properly tested I guess.

chiting765

@chiting765

@gsoul, my test JSON is really simple: curl -v -H "Content-Type: application/json" -X POST -d '[{ "src" : "Mary\'s company" }]' http://IP_address:7784/translator/translate. I cannot execute the command because it is trying to find the match of the third ' I think.

As for -replace_unk, does it matter what model, data and machine did I use?

Also, I am trying to use GRU instead of LSTM as the -rnn_type, however both the perplexity and the validation perplexity is really large after two epoch. I have more than 228000 sentences, and usually with LSTM, the ppl are pretty small after some iterations.

chiting765

@chiting765

To use GRU, is there anything I need to do beside change -rnn_type in the train.lua? Do I need to do anything special during preprocessing?

chiting765

@chiting765

Can GRU handle additional features like case and domain?

lpzreq

@lpzreq

Why tokenization with case_feature can influence to translate result?
For example: House-> House House, sonne -> House

lpzreq

@lpzreq

For example: House-> House, house -> House

Guillaume Klein

@guillaumekln

@chiting765 This has been covered many times on the forum and GitHub issues. You should use a smaller learning rate with GRUs. They support all features.

chiting765

@chiting765

@guillaumekln Thank you! I will try a smaller learning rate

chiting765

@chiting765

As for the -replace_unk problem, I didn't train any bpe_model, is it required to use the -replace_unk feature?

Guillaume Klein

@guillaumekln

chiting765

@chiting765

So what could be the reason that -replace_unk does not work for me?

Guillaume Klein

@guillaumekln

What do you mean by does not work?

chiting765

@chiting765

I trained a model and I translate a sentence with some fake words I created which is definitely OOV, the translation of the fake words are some random target words instead of the fake words themselves.

Guillaume Klein

@guillaumekln

That's not how -replace_unk works, see the option description. It only replaces generated target unknown words.

chiting765

@chiting765

It should replace the target unknown token with the source token that has the highest attention weight right? But I just got some random target token as the translation for my unknown source tokens.

For example, if I translate 'Here is my "alkhcxli" word' to Spanish, I get 'Aqui es mi palabra "rnnr5555"'

chiting765

@chiting765

I want to get 'Aqui es mi palabra "alkhcxli"' when I turn on -replace_unk

Guillaume Klein

@guillaumekln

What do you get without -replace_unk?

chiting765

@chiting765

I get the same translation with or without -replace_unk

Guillaume Klein

@guillaumekln

So it's working as expected. This option only replaces unknown words that are generated. Your issue is a training data issue.

Typically you want the model to learn to translate <unk> by <unk> and then -replace_unk can do its job at inference time.

chiting765

@chiting765

How to train the model to learn to translate <unk> by <unk>? I think the tokenizer separate <unk> into different tokens

Guillaume Klein

@guillaumekln

More like how to make source OOV also OOV in the target. But you can also use BPE and it will mitigate by a lot your issue.

chiting765

@chiting765

So after training my model using my training data, I need another set of data which contains OOV in both source and target file to continue to train the current model?

Is BPE model good for English to Spanish? I think it is probably good for German or Turkish, not sure about Spanish

Jean Senellart

@jsenellart

@chiting765 - yes BPE is excellent for English to Spanish, with this language pair, you should train a joint model.

regarding OOV issue - you just need to make sure that your vocabulary does not include all the vocabs of your training data which can happens when working with relatively small dataset

chiting765

@chiting765

@jsenellart Thanks!

Another question, when I train with Pyramidal deep bidirectional encoder -encoder_type pdbrnn, I got the following error:
[12/08/17 12:02:38 INFO] Preparing memory optimization...
/home/languageintell/torch/install/bin/luajit: bad argument #2 to '?' (out of range)
stack traceback:
[C]: at 0x7f9d0cce44b0
[C]: in function 'index'
./onmt/data/Batch.lua:181: in function 'addInputFeatures'
./onmt/data/Batch.lua:197: in function 'getSourceInput'
./onmt/modules/Encoder.lua:233: in function 'forward'
./onmt/modules/BiEncoder.lua:153: in function 'forward'
./onmt/modules/PDBiEncoder.lua:137: in function 'forward'
./onmt/Seq2Seq.lua:209: in function 'trainNetwork'
./onmt/utils/Memory.lua:39: in function 'optimize'
./onmt/train/Trainer.lua:137: in function 'init'
...anguageintell/torch/install/share/lua/5.1/torch/init.lua:91: in function 'new'
train.lua:282: in function 'main'
train.lua:288: in main chunk
[C]: in function 'dofile'
...tell/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

Did I miss anything or is it a bug?

Konstantin Glushak

@gsoul

now I used only Lua version of tokenizer/learn_bpe + python “create_vocab” and I got this: https://drive.google.com/file/d/1Iwu0CvU48WzYfl1M6_LkKbtc8RvN-ymQ/view?usp=sharing
14323 tokens instead of 32k.
@guillaumekln
Is this with the latest version of the Lua scripts? Also, could you try without changing the joiner marker? It should not be an issue but it wasn't properly tested I guess.

Sorry for the late reply. I used latest master version of Lua code and I just tried without the joiner marker, the result is exactly the same.

cat [pathto]/enfr/giga-fren.release2.token.en [pathto]/enfr/giga-fren.release2.token.fr | th tools/learn_bpe.lua -size 32000 -save_bpe giga_codes.txt

th tools/tokenize.lua -bpe_model giga_codes.txt -nparallel 6 -joiner_annotate -mode aggressive -segment_numbers < [pathto]/enfr/giga-fren.release2.token.shuf.en > [pathto]/enfr/giga-fren.release2.bpe.shuf.en

cat ./enfr/giga-fren.release2.token.shuf.en ./enfr/giga-fren.release2.token.shuf.fr > data/enfr/tmp.txt

python -m bin.build_vocab --save_vocab data/enfr/src-bpe.txt data/enfr/tmp.txt --size 32000

@guillaumekln please take a look, when you have a minute.

Konstantin Glushak

@gsoul

I’m thinking that maybe “-size 32000” for learn_bpe.lua is too much?

Because in comments it say that -size option is not a vocabulary size, but rather:

[[The number of merge operations to learn.]]

Manga

@sai10

translation process is not working properly
in this link discription is given https://imgur.com/a/JP8M62
my src sentences is that- https://drive.google.com/a/iiit-bh.ac.in/file/d/1zYBVXo19QphH5Ksdj3zvHE51oCOE9unt/view?usp=sharing
Training file- https://drive.google.com/a/iiit-bh.ac.in/file/d/1Kd8l3WLpgG7c5XoymzxQeLAmzD9252i2/view?usp=sharing

Konstantin Glushak

@gsoul

@guillaumekln when I increased “-size” parameter of learn_bpe to 64000, I finally could collect my vocabulary of 32k tokens. While there were ~48k unique tokens in the corpora.

lpzreq

@lpzreq

Anybody tested nvidia Volt with OpenNMT? Hou much it faster than GTX 1080 Ti?

Guillaume Klein

@guillaumekln

@gsoul Can you try without -nparallel? I already trained models on OpenNMT-tf with joiners and there were no issues. On a side note, you should pass -tok_mode aggressive -tok_segment_numbers to learn_bpe.lua for consistency.

Konstantin Glushak

@gsoul

@guillaumekln I will try, thank you. And if there are any recommendations, could you please advise on proper “-size” parameter for learn_bpe.lua?

Guillaume Klein

@guillaumekln

30000 is frequently used.

Konstantin Glushak

@gsoul

The issue is that when I use it for the giga-gren corpora, I can’t get a 32k vocabulary. Only about 14k. Though maybe that’s because I didn’t apply -tok_mode to learn_bpe, but did it for all the other steps. I’ll try it once more, thanks again!

Ben Peters

@bpopeters

Is there an easy way to extract feature embeddings? extract_embeddings.lua is only giving me the word embeddings.

Ratish Puduppully

@ratishsp

Hi, I am trying to understand the decoding process in decoder.lua. I want to know why we iterate in the reverse direction in backward method for t = batch.targetLength, 1, -1 do

Guillaume Klein

@guillaumekln

@bpopeters, in extract_embeddings.lua instead of catching torch.type(m) == "onmt.WordEmbedding" you could catch all torch.type(m) == "nn.LookupTable", dump m.weight and recognize the feature based on the vocabulary size.

@ratishsp, this is the idea of the backward pass: to walk the graph in the reverse order.

Vincent Nguyen

@vince62s

Did you guys read this https://arxiv.org/pdf/1712.05690.pdf

Where communities thrive

People

Repo info

Activity