Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    SP Mohanty
    @spMohanty
    @mengdong : We are looking into this with @harperj and @awjuliani
    Arthur Juliani
    @awjuliani

    Dear Obstacle Tower Challenge participant(s),

    We have received an amazing response to the challenge these past two months, with over 200 teams signed up and the top participants reaching an average of over nine floors solved by their agents!

    *Due to some initial technical challenges related to automatic evaluation we have decided to extend the contest dates to ensure that all participants can make the best submissions possible. The new deadline for submission to Round 1 is April 30th. We have also extended the dates for Round 2, which will now be open for submissions from May 15th to July 15th. This will give everyone an additional month in each round to experiment with new ideas for training their agents. Please see the official rules here for more details: https://gitlab.aicrowd.com/unity/obstacle-tower-challenge-resources/blob/master/Rules.md

    In summary, the new challenge dates will be:

    Round 1: Feb 11 - Apr 30

    Round 2: May 15 - Jul 15

    Important: Please also note that you’ll need to agree to the updated terms with the new dates via the AICrowd webpage by clicking the challenge “participate” button again before making any new submissions to the challenge.

    DougM
    @mengdong
    @spMohanty @awjuliani I found out the Frame Per Second I got on a single node with multiple ObtacleTowerEnv is just too low, comparing with some other environment like gym atari or dmlab
    48 envs in parallel running on virutal GL, I am only getting ~1.5k fps
    SP Mohanty
    @spMohanty
    The platform is undergoing some maintenance at the moment, and some services might be unavaible.
    We will post an update here when that is resolved.
    SP Mohanty
    @spMohanty
    The platform is back online now
    The submissions which werre affected, will be requeued automatically
    Arthur Juliani
    @awjuliani

    Hi Obstacle Tower Challenge contestants,

    Per our contest rules, only the final submission made by each team will be accepted for evaluation at the end of each round. In the past, the leaderboard had incorrectly reflected the best submission. As of this morning, April 11, 2019, we have addressed this, and the leaderboard rankings now reflect the last submission (not the best). You may have noticed your ranking go up or down.

    Please be sure that your final submission before the end of each round reflects your best work, and be sure to re-submit that work if necessary.

    Thank you,
    Obstacle Tower Team

    KarolisRam
    @KarolisRam
    I guess the evaluation server is down - "Pod Scheduling Time 🕐 : 10005 secs"
    what was the main reason for this change to leaderboard?
    Arthur Juliani
    @awjuliani
    @KarolisRam It was changed to align with the written rules for the contest.
    Alex Gomez
    @alejgomez
    Hello is it normal to wait over one hour in evaluation?
    State : Evaluation Pending ⌛
    Total Execution Time ⌛ : 4148s
    Pod State : Pending
    Pod Scheduling Time 🕐 : 3950 secs
    Alex Gomez
    @alejgomez
    Help appreciated, after few hours evaluation is stuck in the same state.
    State : Evaluation Pending ⌛
    @KarolisRam are you having the same issue?
    SP Mohanty
    @spMohanty
    @alejgomez : This should be resolved now !!
    And we added a separate cleanup job for stuck submissions
    Alex Gomez
    @alejgomez
    thank you! @spMohanty submission status now says failed. I guess I should try resubmitting.
    Alex Gomez
    @alejgomez
    Now that the submissions do submit succesfully and are evaluated, the result is always failed after evaluation. Running in debug mode does not show any logs.
    Any clue?
    image.png
    Alex Gomez
    @alejgomez
    Good Morning! Submissions are stuck again since yesterday.
    Alex Gomez
    @alejgomez
    OK tagging aicrowd-bot did the trick to start the evaluation
    is there a way to discard the other pending submissions ?
    Shivam Khandelwal
    @skbly7
    @alejgomez which submissions you want to discard? If they are still in pending queue, I can do it for you.
    Shivam Khandelwal
    @skbly7
    @here We noticed problem in our inter-pod communication queue, due to which multiple submissions were stuck in pending state. The issue has been fully resolved now and re-evaluation is triggered on all the effected submissions. Please let us know if you still face any issue.
    Karl
    @zook111
    anyone know how to fix this?
    docker run --env OTC_EVALUATION_ENABLED=true --network=host -it obstacle_tower_challenge:latest ./run.sh
    root
    INFO:mlagents_envs:Start training by pressing the Play button in the Unity Editor.
    INFO:mlagents_envs:
    'ObstacleTower-v1.3' started successfully!
    Unity Academy name: ObstacleTower-v1.3
    Number of Brains: 1
    Number of Training Brains : 1
    Reset Parameters :
    dense-reward -> 1.0
    tower-seed -> -1.0
    floor-number -> 0.0
    Unity brain name: LearningBrain
    Number of Visual Observations (per agent): 1
    Vector Observation space size (per agent): 7
    Number of stacked Vector Observation: 1
    Vector Action space type: discrete
    Vector Action space size (per agent): [3, 3, 2, 3]
    Vector Action descriptions: Movement Forward/Back, Camera, Jump, Movement Left/Right
    Traceback (most recent call last):
    File "run.py", line 28, in <module>
    env = ObstacleTowerEnv(args.environment_filename, docker_training=args.docker_training)
    File "/srv/conda/lib/python3.6/site-packages/obstacle_tower_env.py", line 59, in init
    str(self.ALLOWED_VERSIONS)
    obstacle_tower_env.UnityGymException: Invalid Obstacle Tower version. Your build is v1.3 but only the following versions are compatible with this gym: ['1', '1.1', '1.2']
    tduval
    @tduval
    @zook111 update your obstacle-tower-env with the latest version. It is because you are using ObstacleTower v1.3 binary but your obstacle-tower-env is not updated and support only 1.2
    Karl
    @zook111
    @tduval am confused I keep reinstalling latest env (see top of output)
    Installing collected packages: obstacle-tower-env
    Found existing installation: obstacle-tower-env 1.3
    Uninstalling obstacle-tower-env-1.3:
    Successfully uninstalled obstacle-tower-env-1.3
    Running setup.py develop for obstacle-tower-env
    Successfully installed obstacle-tower-env
    (otc-env) karl@DESKTOP-2MBV500:~/obstacle-tower-env$ docker run --env OTC_EVALUATION_ENABLED=true --network=host -it obstacle_tower_challenge:latest ./run.sh
    root
    INFO:mlagents_envs:Start training by pressing the Play button in the Unity Editor.
    INFO:mlagents_envs:
    'ObstacleTower-v1.3' started successfully!
    Unity Academy name: ObstacleTower-v1.3
    Number of Brains: 1
    Number of Training Brains : 1
    Reset Parameters :
    dense-reward -> 1.0
    tower-seed -> -1.0
    floor-number -> 0.0
    Unity brain name: LearningBrain
    Number of Visual Observations (per agent): 1
    Vector Observation space size (per agent): 7
    Number of stacked Vector Observation: 1
    Vector Action space type: discrete
    Vector Action space size (per agent): [3, 3, 2, 3]
    Vector Action descriptions: Movement Forward/Back, Camera, Jump, Movement Left/Right
    Traceback (most recent call last):
    File "run.py", line 28, in <module>
    env = ObstacleTowerEnv(args.environment_filename, docker_training=args.docker_training)
    File "/srv/conda/lib/python3.6/site-packages/obstacle_tower_env.py", line 59, in init
    str(self.ALLOWED_VERSIONS)
    obstacle_tower_env.UnityGymException: Invalid Obstacle Tower version. Your build is v1.3 but only the following versions are compatible with this gym: ['1', '1.1', '1.2']
    Alex Gomez
    @alejgomez
    @zook111 did you clone the latest version of the repo ? https://github.com/Unity-Technologies/obstacle-tower-env
    apparently the obstacle tower is now v1.3
    so that means you need to update both the ObstacleTower application, and the repo if you want to use the latest.
    I had this issue as my docker environment was continuously dowloading the v1.2 of the repo, while I already had the v1.3 of the app
    Karl
    @zook111
    @alejgomez yes cloned latest repo. I think there must be an issue with the docker image maybe?? Did you fix the docker issue?
    Alex Gomez
    @alejgomez
    @zook111 it depends how you install dependencies in your docker image. my "requirements.txt" had "git+git://github.com/Unity-Technologies/obstacle-tower-env@v1.2"
    i deleted @v1.2 and voila
    Arthur Juliani
    @awjuliani
    Hi all. Thanks for pointing this out. Last week we released v1.3 of the binary and gym interface which fixed a couple of issues participants were having. We didn't yet update the challenge repo to target this latest version. I will update it today.
    Just pushed a fix to master branch of obstacle-tower-challenge
    Nguyen Huynh
    @huynhnguyen
    This message was deleted
    I got this error on evaluation but it quite difficult to trace without log returned. Please help on it?
    sorry I try to show my screen picture but fail to load
    Karl
    @zook111
    @awjuliani many thanks.
    Nguyen Huynh
    @huynhnguyen
    I think I miss the debug configure setting, sorry for the previous message. Thanks.
    KarolisRam
    @KarolisRam
    what the... I haven't trained my agent on puzzle rooms at all (floors 10+) and it somehow got a score of 25 on one of the runs (14 floors probably)?! It gets to floor 10 from time to time, but not often. I hope I made a mistake in reward calculation... otherwise it might have become self aware D:
    Arthur Juliani
    @awjuliani
    Hi @KarolisRam Could you possibly record a video of the agent? It might have discovered a bug that allows it to skip floors :)
    KarolisRam
    @KarolisRam
    I'll try. Sadly I wasn't setting/saving random seeds, I will try to do a sweep of seeds and figure out what's happening.
    KarolisRam
    @KarolisRam
    hmm even with a set seed the agent starting position seems to be a bit random (the agent is sometimes a bit further forward). Is this intended?