OpenNMT/openmt

Discussion about OpenNMT (deprecated, use OpenNMT-py instead)

Konstantin Glushak

@gsoul

In general I follow this default mode:

tools/learn_bpe.lua -size 30000 -save_bpe codes < input_tokenized
tools/tokenize.lua -bpe_model codes < input_tokenized

http://opennmt.net/OpenNMT/tools/tokenization/#bpe

Guillaume Klein

@guillaumekln

Hello, use the -joiner_annotate flag.

Konstantin Glushak

@gsoul

Thanks, that’s mostly helped. It’s just now it looks like:
"A Republ￭ ican strategy to counter the re - election of Ob￭ ama”

whatever synbols I pass as “-bpe_EOT_marker”/“-bpe_BOT_marker"

Guillaume Klein

@guillaumekln

That's expected, ￭ is the symbol used by the Lua tokenizer (equivalent of @@).

Konstantin Glushak

@gsoul

Is there some parameter that allows to sets some ASCII symbol? I’m going to further use it with TF version...

Guillaume Klein

@guillaumekln

You can use the -joiner option but there is little reasons to change it.

Konstantin Glushak

@gsoul

It worked, thank you.

//The idea is that I do some manual checking of the results, and so strongly prefer human-readable symbols. It’s a matter of personal convenience rather than necessity.

Konstantin Glushak

@gsoul

@guillaumekln my process for the BPE was:

Learn BPE codes on joined corpus of En and Fr files
Apply those codes to both files through tokenize.lua
Build 2 vocabularies for En and Fr files with python script from OpenNMT-tf. I set vocabulary size to 32000

The issue I see is that vocabulary size I get is around 11k tokens and last ones are some random chinese characters. So I guess I have some mistake somewhere in the process. Maybe it’s the number or merges for learn_bpe.lua (30000). I don’t know. Could you please advise if you have insight any here?

Guillaume Klein

@guillaumekln

What tokenization did you apply when using learn_bpe.lua?

Konstantin Glushak

@gsoul

I fed the tokenised file, and I used Spacy to produce it. Manual check showed that tokenization looks good and shouldn’t be an issue, imho.

Previously I learnt t2t model with the same tokenisation, just without BPE part

Guillaume Klein

@guillaumekln

What is the reasoning behind using the Spacy tokenizer? I would simply do something like this:

tools/learn_bpe.lua -size 32000 -save_bpe codes -tok_mode aggressive -tok_segment_numbers < input_raw
tools/tokenize.lua -bpe_model codes -mode aggressive -segment_numbers -joiner_annotate < input_raw

Konstantin Glushak

@gsoul

I’ll try it, thanks.

Another question I have is if it’s correct to learn BPE on a joined (src + tgt) file, but then for NMT training create separate dictionary for src and tgt?

Guillaume Klein

@guillaumekln

You can generate a single vocabulary on the joined tokenized corpus.

chiting765

@chiting765

Hi~~ I have a question about the REST server, when I test it with the curl command, I include the source text in JSON. If my source text contains quotations marks, single or double, how should I escape them? Thanks!

Konstantin Glushak

@gsoul

Hey, did you try backslash? “\"

chiting765

@chiting765

\" seems to be OK, but bash does not like \'

How should I escape apostrophe?

chiting765

@chiting765

Also, -replace_unk just doesn't work for me and I don't know why

Konstantin Glushak

@gsoul

As for single quote backslash should also work. But if it doesn’t, please paste your JSON on some site like https://pastebin.com/ and give us a link to it. Thank you.

As for unk, what model did you train? What data did you use for it? And for how long did you train it, using what hardware?

Konstantin Glushak

@gsoul

@guillaumekln now I used only Lua version of tokenizer/learn_bpe + python “create_vocab” and I got this: https://drive.google.com/file/d/1Iwu0CvU48WzYfl1M6_LkKbtc8RvN-ymQ/view?usp=sharing

14323 tokens instead of 32k.

Guillaume Klein

@guillaumekln

Is this with the latest version of the Lua scripts? Also, could you try without changing the joiner marker? It should not be an issue but it wasn't properly tested I guess.

chiting765

@chiting765

@gsoul, my test JSON is really simple: curl -v -H "Content-Type: application/json" -X POST -d '[{ "src" : "Mary\'s company" }]' http://IP_address:7784/translator/translate. I cannot execute the command because it is trying to find the match of the third ' I think.

As for -replace_unk, does it matter what model, data and machine did I use?

Also, I am trying to use GRU instead of LSTM as the -rnn_type, however both the perplexity and the validation perplexity is really large after two epoch. I have more than 228000 sentences, and usually with LSTM, the ppl are pretty small after some iterations.

chiting765

@chiting765

To use GRU, is there anything I need to do beside change -rnn_type in the train.lua? Do I need to do anything special during preprocessing?

chiting765

@chiting765

Can GRU handle additional features like case and domain?

lpzreq

@lpzreq

Why tokenization with case_feature can influence to translate result?
For example: House-> House House, sonne -> House

lpzreq

@lpzreq

For example: House-> House, house -> House

Guillaume Klein

@guillaumekln

@chiting765 This has been covered many times on the forum and GitHub issues. You should use a smaller learning rate with GRUs. They support all features.

chiting765

@chiting765

@guillaumekln Thank you! I will try a smaller learning rate

chiting765

@chiting765

As for the -replace_unk problem, I didn't train any bpe_model, is it required to use the -replace_unk feature?

Guillaume Klein

@guillaumekln

chiting765

@chiting765

So what could be the reason that -replace_unk does not work for me?

Guillaume Klein

@guillaumekln

What do you mean by does not work?

chiting765

@chiting765

I trained a model and I translate a sentence with some fake words I created which is definitely OOV, the translation of the fake words are some random target words instead of the fake words themselves.

Guillaume Klein

@guillaumekln

That's not how -replace_unk works, see the option description. It only replaces generated target unknown words.

chiting765

@chiting765

It should replace the target unknown token with the source token that has the highest attention weight right? But I just got some random target token as the translation for my unknown source tokens.

For example, if I translate 'Here is my "alkhcxli" word' to Spanish, I get 'Aqui es mi palabra "rnnr5555"'

chiting765

@chiting765

I want to get 'Aqui es mi palabra "alkhcxli"' when I turn on -replace_unk

Guillaume Klein

@guillaumekln

What do you get without -replace_unk?

chiting765

@chiting765

I get the same translation with or without -replace_unk

Guillaume Klein

@guillaumekln

So it's working as expected. This option only replaces unknown words that are generated. Your issue is a training data issue.

Typically you want the model to learn to translate <unk> by <unk> and then -replace_unk can do its job at inference time.

chiting765

@chiting765

How to train the model to learn to translate <unk> by <unk>? I think the tokenizer separate <unk> into different tokens

Guillaume Klein

@guillaumekln

More like how to make source OOV also OOV in the target. But you can also use BPE and it will mitigate by a lot your issue.

chiting765

@chiting765

So after training my model using my training data, I need another set of data which contains OOV in both source and target file to continue to train the current model?

Is BPE model good for English to Spanish? I think it is probably good for German or Turkish, not sure about Spanish

Jean Senellart

@jsenellart

@chiting765 - yes BPE is excellent for English to Spanish, with this language pair, you should train a joint model.

Where communities thrive

People

Repo info

Activity