@guillaumekln hi there...!
In case if two language doesn't share the characters then how do we do the translation on a Name. Like "Rahul" (a male name) is english, should be "راهول" in Arabic, "राहुल" in Hindi.
So how actually
If there is any related reference paper you can cite that would be great also. And also does OpenNMT supports this?
In case if two language doesn't share the characters then how do we do the translation on a Name. Like "Rahul" (a male name) is english, should be "راهول" in Arabic, "राहुल" in Hindi.
So how actually
<unk> tag replace work in that case.If there is any related reference paper you can cite that would be great also. And also does OpenNMT supports this?
Hi! In OpenNMT-Lua, there is the
-phrase_table that can be used for this: for any <unk> token in the target, the corresponding source token is looked up in the table to find a translation. Other approaches include splitting names on characters and let the model learn the translation or, more commonly, replacing entities with placeholder tokens and have a separate processing software to replace these placeholders.
I don't see any -phrase_table tag for OpenNMT-py
I was thinking about doing some kind of Transliteration/Romanization of the target language to the source language.
what are the pre/post processing? Any resource out there?
Apart from that if I see(in the following link) google translate, see the red marked text here. Is this the pre/post processing steps here?
https://drive.google.com/file/d/1POkH3XE6yfCoNkJn4DyMRP6J11Ecyqjl/view?usp=sharing
Otherwise, this requires an external pre- and post- processing.what are the pre/post processing? Any resource out there?
Apart from that if I see(in the following link) google translate, see the red marked text here. Is this the pre/post processing steps here?
https://drive.google.com/file/d/1POkH3XE6yfCoNkJn4DyMRP6J11Ecyqjl/view?usp=sharing
@guillaumekln Thank you for your answer. I will think and work on your points. If I came to any specific problem then I will ask you again. Actually right now I'm exploring NMT. I wanted to see how to handle the different issues of NMT. This is a pedagogical exploration. Right now I don't have any specific workflow. My research goal is to do kind of Discourse exploration in NMT. I just started.
[02/08/19 09:43:54 INFO] Epoch 4 ; Iteration 98300/103388 ; Optim SGD LR 1.000000 ; Source tokens/s 8042 ; Perplexity 10.81
/root/torch/install/bin/luajit: bad argument #2 to '?' (out of range at /root/torch/pkg/torch/generic/Tensor.c:913)
stack traceback:
[C]: at 0x7f65146d1580
[C]: in function '__index'
./onmt/train/Trainer.lua:234: in function 'getBatchIdx'
./onmt/train/Trainer.lua:276: in function 'trainEpoch'
./onmt/train/Trainer.lua:484: in function 'train'
train.lua:333: in function 'main'
train.lua:338: in main chunk
[C]: in function 'dofile'
/root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
root@Ubuntu-1604-xenial-64-minimal:~/OpenNMT
/root/torch/install/bin/luajit: bad argument #2 to '?' (out of range at /root/torch/pkg/torch/generic/Tensor.c:913)
stack traceback:
[C]: at 0x7f65146d1580
[C]: in function '__index'
./onmt/train/Trainer.lua:234: in function 'getBatchIdx'
./onmt/train/Trainer.lua:276: in function 'trainEpoch'
./onmt/train/Trainer.lua:484: in function 'train'
train.lua:333: in function 'main'
train.lua:338: in main chunk
[C]: in function 'dofile'
/root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
root@Ubuntu-1604-xenial-64-minimal:~/OpenNMT
What can be do it?
@sergei-from-vironit most NMT models use byte pair encoding for to solve unknow words can you explain why it is useful?
_
_
I have multiple questions regarding copy mechanism implemented in Opennmt toolkit.
1) "Since we are using copy-attention [1] in the model, we need to preprocess the dataset such that source and target are aligned and use the same dictionary. " .This is a line from the readme . What does alignment between source and target words mean?
2) Has anyone tried copy mechanism with transformer architecture? If yes how was the performance?
1) "Since we are using copy-attention [1] in the model, we need to preprocess the dataset such that source and target are aligned and use the same dictionary. " .This is a line from the readme . What does alignment between source and target words mean?
2) Has anyone tried copy mechanism with transformer architecture? If yes how was the performance?
You probably want to post these questions in https://gitter.im/OpenNMT/OpenNMT-py or in the forum https://forum.opennmt.net/
I wonder maybe I should try a different BPE model other the aggressive one? I will try to update more often at https://gitter.im/OpenNMT/OpenNMT-py https://wordcounter.tools
