Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Alex Gomez
    @alejgomez
    Help appreciated, after few hours evaluation is stuck in the same state.
    State : Evaluation Pending ⌛
    @KarolisRam are you having the same issue?
    SP Mohanty
    @spMohanty
    @alejgomez : This should be resolved now !!
    And we added a separate cleanup job for stuck submissions
    Alex Gomez
    @alejgomez
    thank you! @spMohanty submission status now says failed. I guess I should try resubmitting.
    Alex Gomez
    @alejgomez
    Now that the submissions do submit succesfully and are evaluated, the result is always failed after evaluation. Running in debug mode does not show any logs.
    Any clue?
    image.png
    Alex Gomez
    @alejgomez
    Good Morning! Submissions are stuck again since yesterday.
    Alex Gomez
    @alejgomez
    OK tagging aicrowd-bot did the trick to start the evaluation
    is there a way to discard the other pending submissions ?
    Shivam Khandelwal
    @skbly7
    @alejgomez which submissions you want to discard? If they are still in pending queue, I can do it for you.
    Shivam Khandelwal
    @skbly7
    @here We noticed problem in our inter-pod communication queue, due to which multiple submissions were stuck in pending state. The issue has been fully resolved now and re-evaluation is triggered on all the effected submissions. Please let us know if you still face any issue.
    Karl
    @zook111
    anyone know how to fix this?
    docker run --env OTC_EVALUATION_ENABLED=true --network=host -it obstacle_tower_challenge:latest ./run.sh
    root
    INFO:mlagents_envs:Start training by pressing the Play button in the Unity Editor.
    INFO:mlagents_envs:
    'ObstacleTower-v1.3' started successfully!
    Unity Academy name: ObstacleTower-v1.3
    Number of Brains: 1
    Number of Training Brains : 1
    Reset Parameters :
    dense-reward -> 1.0
    tower-seed -> -1.0
    floor-number -> 0.0
    Unity brain name: LearningBrain
    Number of Visual Observations (per agent): 1
    Vector Observation space size (per agent): 7
    Number of stacked Vector Observation: 1
    Vector Action space type: discrete
    Vector Action space size (per agent): [3, 3, 2, 3]
    Vector Action descriptions: Movement Forward/Back, Camera, Jump, Movement Left/Right
    Traceback (most recent call last):
    File "run.py", line 28, in <module>
    env = ObstacleTowerEnv(args.environment_filename, docker_training=args.docker_training)
    File "/srv/conda/lib/python3.6/site-packages/obstacle_tower_env.py", line 59, in init
    str(self.ALLOWED_VERSIONS)
    obstacle_tower_env.UnityGymException: Invalid Obstacle Tower version. Your build is v1.3 but only the following versions are compatible with this gym: ['1', '1.1', '1.2']
    tduval
    @tduval
    @zook111 update your obstacle-tower-env with the latest version. It is because you are using ObstacleTower v1.3 binary but your obstacle-tower-env is not updated and support only 1.2
    Karl
    @zook111
    @tduval am confused I keep reinstalling latest env (see top of output)
    Installing collected packages: obstacle-tower-env
    Found existing installation: obstacle-tower-env 1.3
    Uninstalling obstacle-tower-env-1.3:
    Successfully uninstalled obstacle-tower-env-1.3
    Running setup.py develop for obstacle-tower-env
    Successfully installed obstacle-tower-env
    (otc-env) karl@DESKTOP-2MBV500:~/obstacle-tower-env$ docker run --env OTC_EVALUATION_ENABLED=true --network=host -it obstacle_tower_challenge:latest ./run.sh
    root
    INFO:mlagents_envs:Start training by pressing the Play button in the Unity Editor.
    INFO:mlagents_envs:
    'ObstacleTower-v1.3' started successfully!
    Unity Academy name: ObstacleTower-v1.3
    Number of Brains: 1
    Number of Training Brains : 1
    Reset Parameters :
    dense-reward -> 1.0
    tower-seed -> -1.0
    floor-number -> 0.0
    Unity brain name: LearningBrain
    Number of Visual Observations (per agent): 1
    Vector Observation space size (per agent): 7
    Number of stacked Vector Observation: 1
    Vector Action space type: discrete
    Vector Action space size (per agent): [3, 3, 2, 3]
    Vector Action descriptions: Movement Forward/Back, Camera, Jump, Movement Left/Right
    Traceback (most recent call last):
    File "run.py", line 28, in <module>
    env = ObstacleTowerEnv(args.environment_filename, docker_training=args.docker_training)
    File "/srv/conda/lib/python3.6/site-packages/obstacle_tower_env.py", line 59, in init
    str(self.ALLOWED_VERSIONS)
    obstacle_tower_env.UnityGymException: Invalid Obstacle Tower version. Your build is v1.3 but only the following versions are compatible with this gym: ['1', '1.1', '1.2']
    Alex Gomez
    @alejgomez
    @zook111 did you clone the latest version of the repo ? https://github.com/Unity-Technologies/obstacle-tower-env
    apparently the obstacle tower is now v1.3
    so that means you need to update both the ObstacleTower application, and the repo if you want to use the latest.
    I had this issue as my docker environment was continuously dowloading the v1.2 of the repo, while I already had the v1.3 of the app
    Karl
    @zook111
    @alejgomez yes cloned latest repo. I think there must be an issue with the docker image maybe?? Did you fix the docker issue?
    Alex Gomez
    @alejgomez
    @zook111 it depends how you install dependencies in your docker image. my "requirements.txt" had "git+git://github.com/Unity-Technologies/obstacle-tower-env@v1.2"
    i deleted @v1.2 and voila
    Arthur Juliani
    @awjuliani
    Hi all. Thanks for pointing this out. Last week we released v1.3 of the binary and gym interface which fixed a couple of issues participants were having. We didn't yet update the challenge repo to target this latest version. I will update it today.
    Just pushed a fix to master branch of obstacle-tower-challenge
    Nguyen Huynh
    @huynhnguyen
    This message was deleted
    I got this error on evaluation but it quite difficult to trace without log returned. Please help on it?
    sorry I try to show my screen picture but fail to load
    Karl
    @zook111
    @awjuliani many thanks.
    Nguyen Huynh
    @huynhnguyen
    I think I miss the debug configure setting, sorry for the previous message. Thanks.
    KarolisRam
    @KarolisRam
    what the... I haven't trained my agent on puzzle rooms at all (floors 10+) and it somehow got a score of 25 on one of the runs (14 floors probably)?! It gets to floor 10 from time to time, but not often. I hope I made a mistake in reward calculation... otherwise it might have become self aware D:
    Arthur Juliani
    @awjuliani
    Hi @KarolisRam Could you possibly record a video of the agent? It might have discovered a bug that allows it to skip floors :)
    KarolisRam
    @KarolisRam
    I'll try. Sadly I wasn't setting/saving random seeds, I will try to do a sweep of seeds and figure out what's happening.
    KarolisRam
    @KarolisRam
    hmm even with a set seed the agent starting position seems to be a bit random (the agent is sometimes a bit further forward). Is this intended?
    parilo
    @parilo
    SP Mohanty
    @spMohanty
    I will be offline soon, but managed to catch this one fast enough
    but please do try running your agents locally before submitting
    parilo
    @parilo
    I tried and it worked well
    SP Mohanty
    @spMohanty
    Many of the errors that we see are usually missing dependencies, etc
    parilo
    @parilo
    Thank you!
    SP Mohanty
    @spMohanty
    parilo
    @parilo
    Maybe I need to rebuild docker image, I use ObstacleEnv 1.2 - is it a problem?
    SP Mohanty
    @spMohanty
    Shouldnt be !! But now that I look closely to your error
    yes, its unusual why it has an issue with finding mlagents
    :/
    well the image is rebuilt on every evaluation