Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    James Hirschorn
    @quantitative-technologies
    Yes, that is true. But it was easy to figure out how to use TOKEN by looking at the dialog problem. I'm actually not sure what SUBWORD is, but for my problem with symbolic reasoning TOKEN was definitely what I needed.
    Bedapudi Praneeth
    @bedapudi6788
    Hi all, does anyone know if getting token level scores is possible in tensor2tensor.
    I am working on a machine translation task, where I want to mark the words with low confidence.
    James Hirschorn
    @quantitative-technologies

    When I train my transformer using t2t-trainer, I see the targets/accuracy go up to around 65% on tensorboard. But then when I try:

    t2t-decoder --data_dir=$DATA_DIR --problem=$PROBLEM --model=$MODEL --hparams_set=$HPARAMS --output_dir=$TRAIN_DIR --t2t_usr_dir='.'

    It looks like none of the OUTPUTs matches the corresponding TARGETs. Am I missing something obvious?

    Lukasz Kaiser
    @lukaszkaiser
    @bedapudi6788 : are you asking for sth like the TransformerScorer model? https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py#L1302
    @quantitative-technologies : Transformer is autoregressive, so at 65% accuracy it may not produce any fully identical sequence at decoding time; but normally it should give some reasonable output...?
    James Hirschorn
    @quantitative-technologies

    @lukaszkaiser Yes, the output is "reasonable" about 2/3 of the time. But then what does the 65% refer to?

    I am still learning about the transformer architecture so maybe it will become clear with further reading.

    Lukasz Kaiser
    @lukaszkaiser
    The 65% is per-symbol accuracy of eval; at eval, you do get the ground truth (so all correct symbols) upto time t, and you predict symbol at t+1; at infer, there are no "correct" symbols so you predict t+1 from what you generated upto time t -- so it's usually less accurate than eval
    Bedapudi Praneeth
    @bedapudi6788
    @lukaszkaiser Thank you! I think that's exactly what I need. I will try to figure out how to use that class.
    James Hirschorn
    @quantitative-technologies
    @lukaszkaiser Thanks! I should have guessed that. So I assume that targets/accuracy_per_sequence is the accuracy of predicting the entire target sequence. How about targets/accuracy_top5? My guess is that it is either the accuracy of getting one of the top 5 symbols correct, or perhaps the accuracy of getting all of the top 5 symbols correct. I will have to check the code...
    Lukasz Kaiser
    @lukaszkaiser
    You're right! top_5 is getting the correct one in the top5 (so it's always higher than accuracy)
    Giovanni-Alzetta
    @Giovanni-Alzetta
    Hi everyone! I have a question about the hparam "num_trainable_top_decoder_layers"; I have found it into the evolved transformer. I am expecting it to freeze a certain number of layers for transfer learning, but reading the code I cannot fully grasp what it does when it calls the function "tf.stop_gradient"...could someone please help me? Thank you!
    Anton Kretov
    @anton_kretov_twitter
    Hi everyone! What T2T model has the same architecture as trax Transformer model? Seems like they are all different basing on their parameters number.
    Lukasz Kaiser
    @lukaszkaiser
    @anton_kretov_twitter : as said above, it may be tricky to get that...
    @Giovanni-Alzetta : stop_gradient(x) is just x on the forward pass, but stops all gradient propagation through x on the backward pass... sorry if that's tricky, it is a bit...
    KosayJabre
    @KosayJabre
    Hi, anyone here?
    I'm trying to use a pretrained character level language model, but I keep getting the last character of my input as the output
    I've submitted an issue here tensorflow/tensor2tensor#1808
    Any help or tips would be appreciated
    Lukasz Kaiser
    @lukaszkaiser
    Hi @KosayJabre - I don't think I can help with that, as tensor2tensor is now in maintenance mode... we've switched to Trax, that's where we help and debug.
    Giovanni-Alzetta
    @Giovanni-Alzetta
    @lukaszkaiser Thank you for your answer! Indeed that's a bit tricky...
    Yongqiang Cheng
    @ForeverStrongCheng
    @lukaszkaiser
    Yongqiang Cheng
    @ForeverStrongCheng
    TIMIT dataset - audio_timit_characters_test - speech recognition
    for i in xrange(self._model_hparams.audio_compression):
    AttributeError: 'HParams' object has no attribute 'audio_compression'
    Yongqiang Cheng
    @ForeverStrongCheng
    tensorflow/tensor2tensor#1821 Any help or tips would be appreciated。
    Lukasz Kaiser
    @lukaszkaiser
    @ForeverStrongCheng : T2T is deprecated by now, we're using Trax and TFDS (tensorflow-datasets) for data. I think TFDS may offer more help with data.
    dinosaxon
    @dinosaxon
    Hi all, I have a question. On a 1-GPU machine, I have several mt models and I want to translate using the same model batches of files. I am using t2t-decoder but this way the model is loaded from scratch for each batch. Is there any way to keep the model loaded and translate each batch separately? I tried to merge the batches in one single file but the system crashes due to OoM so this is why I am splitting it into smaller files. I don't want to use the servering. Any help would be appreciated
    Giovanni-Alzetta
    @Giovanni-Alzetta
    I would be interested into training a transformer with multiple targets (from a string generate a list containing two different strings). I have seen this tensorflow/tensor2tensor#183 but the link is broken. Has anything changed in this direction?
    Jiajia Zhao
    @snowbabyjia
    would it be possible to acquire a docker image of the previously working version of SimPLe?
    dinosaxon
    @dinosaxon
    Hi, I am facing an issue training an Engine on an 8GPU google instance. However, when I train on a single GPU, the training starts with no issues. I also tried to use smaller batch size but no luck. Any suggestions? Here is the command I use
    t2t-trainer \
    --data_dir=$DATA_DIR \
    --problem=translate_enfr_wmt_small8k\
    --model=transformer \
    --hparams='max_length=100,eval_drop_long_sequences=True'\
    --hparams='batch_size=1024'\
    --worker_gpu=8 \
    --train_steps=350000 \
    --hparams_set=$HPARAMS \
    --eval_steps=5000 \
    --output_dir=$TRAIN_DIR \
    --schedule=continuous_train_and_eval
    I am getting an Out of Memory error or an allocation error. I am using Tesla K80 with cuda 10.1 and tensorflow 2
    it crashes even if I use batch size 400 but no with one GPU. I also tried to CUDA_VISIBLE_DEVICES
    Regalia
    @RegaliaXYZ
    @lukaszkaiser Hello, it seems like every PR on tensor2tensor fails during the Travis build with the same error:
    The command "./oss_scripts/oss_pip_install.sh" failed and exited with 1 during .
    Regalia
    @RegaliaXYZ
    pip install -q -e . Traceback (most recent call last): File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pkg_resources/__init__.py", line 567, in _build_master ws.require(__requires__) File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pkg_resources/__init__.py", line 884, in require needed = self.resolve(parse_requirements(requirements)) File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pkg_resources/__init__.py", line 775, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (tensorflow-probability 0.7.0 (/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages), Requirement.parse('tensorflow-probability==0.8'), {'kfac'})
    It seems requirements installation is failing during Travis build
    dinosaxon
    @dinosaxon
    Hi all, hi @lukaszkaiser! It seems that I am stuck and I cannot train an Engine with 8GPUs. Can you please help me? I can train with one GPU but not with more than two. Tried with sockeye to train with 8GPUs and it works well in the same installation. Any hep is appreciated
    Nishit Shah
    @nishitnshah

    Hi All, I am trying to run tensor2tensor in colab and I am getting following error. Any help is appreciated.
    '''
    ImportError Traceback (most recent call last)

    <ipython-input-2-00e41dd9537e> in <module>()
    9 import collections
    10
    ---> 11 from tensor2tensor import models
    12 from tensor2tensor import problems
    13 from tensor2tensor.layers import common_layers

    7 frames
    /usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/tf_compat.py in ensure_tf_install()
    57 if tf_version < min_tf_version:
    58 raise ImportError(
    ---> 59 "This version of TensorFlow Datasets requires TensorFlow "
    60 f"version >= {MIN_TF_VERSION}; Detected an installation of version "
    61 f"{tf.version}. Please upgrade TensorFlow to proceed."

    ImportError: This version of TensorFlow Datasets requires TensorFlow version >= 2.1.0; Detected an installation of version 1.15.2. Please upgrade TensorFlow to proceed.
    '''

    Aleksas Pielikis
    @aleksas
    I've strugled a lot with this also.
    choosing tf to be 1.x version doesn't restrict t2t dependencies to install later versions

    to work around it i had to uninstall rogue packages and ensure they are consistent with tf 1.5.2.
    but that's too much of a hassle. In the end i've stayed with tf 2.0 version but instead of installing t2t from pip directly i used my own fork. That was necessary because t2t in master branch is incommpatble with tf2. It shows errors like flags ar not part of tf and so on. The only changes that were necessary was to change import statement in tensor2tensor/bin/*.py from "import tensorflow as tf" to "import tensorflow.compat.v1 as tf"

    then you just install run "pip install ." from root repo dir.

    Nishit Shah
    @nishitnshah
    Its not working there for me.
    Aleksas Pielikis
    @aleksas

    that what i was telling about.

    see these lines:
    if 'google.colab' in sys.modules: # Colab-only TensorFlow version selector
    %tensorflow_version 1.x

    it instructs collab to switch to TF1.x (1.15.2) version

    Aleksas Pielikis
    @aleksas
    well... actually it's not all that goof with "fixed" tensor2tesor. training went fine but exporting or evaluating model caused new exceptions...
    apologies for confusion
    Lukasz Kaiser
    @lukaszkaiser
    Yeah, I'm sorry but you'll not be really happy with T2T and TF2. Give Trax a try (https://github.com/google/trax) - this is our new version that works with new TF and JAX.
    dinosaxon
    @dinosaxon
    Hi I am facing an issue since last year with T2T. I am able to train a system with 1GPU but not with 8! Can you please help me?
    here is my training settings CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
    t2t-trainer \
    --data_dir=$DATA_DIR \
    --problem=$PROBLEM \
    --model=$MODEL \
    --hparams='max_length=70,batch_size=1024,eval_drop_long_sequences=true'\
    --worker_gpu=8 \
    --train_steps=500000 \
    --hparams_set=$HPARAMS \
    --output_dir=$TRAIN_DIR \
    without --worker_gpu=8 the training starts but it uses 1GPU
    Alankar Shukla
    @alan-ai-learner
    how can we restart training from the last saved checkpoints in tensor2tensor?
    vasanth2
    @vasanth2:matrix.org
    [m]
    I suppose it would automatically start from the latest checkpoint in your checkpoint dir. Not completely sure though.