Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Michael Krax
    @michaelkrax
    @Miffyli it might be interesting to build a third track for the 2019 competition in a 1v1 format on 10 known maps. this could facilitate the creation of new strategies more focused on longer term planning, understanding the map, keeping track of where an opponent is or might be, item control, etc. and therefore result in more "human like" behavior. that being said: if i as a human could 360 spin at will in a FPS game with perfect precision, recognize everything in a split second and without getting dizzy i probably would.
    Anssi
    @Miffyli
    @michaelkrax Yeah something more strategy-oriented would be nice: Maybe we could mimic Counter-Strike's objective game or CTF, and have similar setup to what Deepmind just released? That would be super cool ^^
    n+e
    @Trinkle23897

    @mwydmuch I'm confused about track1's evaluation maps. Our group have tested on both test_wad and childs_play_wad, uniform sampled 100 maps, and the result shows that the passing rate is nearly 100%. However, on this platform, the result seems very bad, 40%~60%. So I want to confirm that:

    1. which configuration do you use for the evaluation, difficulty0 or difficult1 or higher? In issue https://gitlab.crowdai.org/TSAIL/track1/issues/41, you claimed that the testing difficult level will be increased to difficult1.
    2. If the difficulty1 configuration is adopted, is this config the same as the configuration published on github (https://github.com/mwydmuch/PyOblige/blob/master/pyoblige/wad_configs.py)?
    3. Do you randomly sample these maps uniformly, or selected some more difficult maps on difficulty1 in humans' perspective to make those maps more challengeable and do a comprehensive test?

    Thank you very much!

    Shiyu Huang
    @huangshiyu13
    @spMohanty @mwydmuch I find that the result of generated video is different from the showed log. I think there is something wrong with your evaluation system.
    For more details, plz refer to this issue: https://gitlab.crowdai.org/Albert/track2_2018/issues/55
    image.png
    I guess the video is generated from map07. You can see the result is as above
    image.png
    As you can see, only Jurgen-Schmidhuber and Yann-LeCunn show the right frag counts.
    Michael Krax
    @michaelkrax
    I started a thread for participant biographies. Would be great if everyone could add a little detail to see who is competing https://www.crowdai.org/topics/participant-bio-thread/discussion
    n+e
    @Trinkle23897
    @mwydmuch Which submission will be selected to do the final evaluation? We have submitted several versions which got mean 1.0. Will you grade on the last submission?
    Marek Wydmuch
    @mwydmuch
    @Trinkle23897 according to the rules we've set, we will grade only the last submission.
    @michaelkrax nice idea :)
    Marek Wydmuch
    @mwydmuch
    @Trinkle23897 we've decided that we will re-evaluate all the submission that scored a tie (on the same set of maps as used for leaderboard) and we will take the best one for the final evaluation.
    Islandman93
    @Islandman93
    @mwydmuch any news on the results of the competition?
    Marek Wydmuch
    @mwydmuch
    @Islandman93 I've just published the results here http://vizdoom.cs.put.edu.pl/competition-cig-2018/competition-results. Unfortunately, I won't manage to publish the movies and detailed statistics today, I will later this week. Also in few minutes I will publish on crowdAI and our google group some comment/summary of the competition.
    Anssi
    @Miffyli
    And Marvin takes the spot! I really like the improvements over last year (and especially over e.g. IntelAct). Good stuff!
    Will be interesting to see how all three competitors this year managed to get 2x better score than last year's top1
    Track1 is going to need more love next time though ^^
    @mwydmuch Was the evaluation painless this time?
    Marek Wydmuch
    @mwydmuch
    We had to move final evaluation to machines used during previous editions and we've encountered some minor problems, but we've managed to solve them :) We aim to make a final evaluation deathmatch evaluation on crowdAI in some tournament-fashion in the future.
    Anssi
    @Miffyli
    Alright! Also Track1 scores seem to "flipped" in final eval vs. crowdai eval system. Bit of overfitting going on there. I guess for next time you will have different online evaluation for track1?
    Marek Wydmuch
    @mwydmuch
    Yeah, so, other submission didn't manage to finish even one level in timeline so that's why we discarded them. I have a lot of doubts about the decision we've made when preparing this track, also it should get more of our attention. It turned out to be much more difficult then we thought, so I think that we definitely should modify the evaluation procedure. Rising number of maps to much higher number (eg. 100) would give us some statistical significance.
    Anssi
    @Miffyli
    Oh okey ^^. Indeed it was quite difficult. That's why it would be cool to know what e.g. DoomNet used, if that is not too nosey thing to ask @akolishchak (congraz!)
    Michael Krax
    @michaelkrax
    Congrats to Ben/Marvin
    Funny to see TSAIL and CVFighter having the exact same score in the last five maps.
    Marek Wydmuch
    @mwydmuch
    Ohh, that's the mistake
    Fixed, sorry for that @michaelkrax, the total scores were correct, I somehow messed up the scores when I was formatting this table.
    Andrey Kolishchak
    @akolishchak
    @Miffyli I'll release the code at https://github.com/akolishchak/doom-net-pytorch this weekend or so.
    Andrey Kolishchak
    @akolishchak
    I tried many things and that was the simplest one - PPO with distance from exit as a reward. I have more performing agent in work now, in case second round materializes for track1.
    Michael Krax
    @michaelkrax

    Many thanks to everyone involved in organizing the competition. This was my first AI competition and it was a whole lot of fun! Looking forward to round 2 of the multiplayer track.
    Special thanks to @spMohanty for being super responsive to questions about Crowd AI and the evaluation process. I am not sure when you ever sleep!

    Congratulations to the winners of both tracks!

    n+e
    @Trinkle23897
    @mwydmuch will you release the testing maps both evaluating on both leaderboard and final? I want to re-evaluate locally. Thanks!
    Anssi
    @Miffyli
    @akolishchak Oo nice! The good ol' K.I.S.S in use ^^
    Marek Wydmuch
    @mwydmuch
    @Trinkle23897 preparing new maps takes some time, so we maybe we will want to keep them also for round 2. But at some point of the time, we will release everything for sure.
    n+e
    @Trinkle23897
    @mwydmuch ok, thanks! But how about track1?
    Marek Wydmuch
    @mwydmuch
    @Trinkle23897 when it comes to track 1, we can easily generate a new set, so we can publish them right now :)
    n+e
    @Trinkle23897
    yeah, that's good
    songshshshsh
    @songshshshsh
    @mwydmuch I want to confirm that you are using the right version of track 1 submission since we never used AWM as the name of track 1 bot
    Marek Wydmuch
    @mwydmuch
    Sure, lets then discuss together with @Trinkle23897
    SP Mohanty
    @spMohanty
    Thanks @/all for all your submissions. It was indeed fun organising this.
    And I must remind you that the round-2 will start accepting submissions for the multiplayer track until November, and the winner there will be extended a travel grant.
    So the broad idea to have this round is to iron out all the issues we have with the whole evaluation setup, collect more feedback from you guys, address them, and apply with the VizDoom challenge for NIPS next year. So any feedback you already have are very welcome, and looking forward to your participation in the second round too.
    Michael Krax
    @michaelkrax
    @spMohanty
    Marek Wydmuch
    @mwydmuch
    @michaelkrax Thank you for your feedback, it's very helpful!
    After yesterday/today's discussion with TSAIL team about results and performance differences in the final evaluation, we've decided to re-evaluate the final results of track 1. I explain the detail here: https://groups.google.com/forum/#!topic/vizdoom/UQXrW6rJupM
    Marek Wydmuch
    @mwydmuch
    I hope you will understand our decision, TSAIL team prepared really good agent and I think this final way we used to compere the agents was the most fair
    Andrey Kolishchak
    @akolishchak
    @mwydmuch re-evaluating mistakenly taken submission is fine but changing the maps to favor hard-coded behavior of one of the agent is not a good move...
    benbellheron
    @benbellheron
    @spMohanty any news on when the round 2 evaluation will open?
    SP Mohanty
    @spMohanty
    @benbellheron : there wasn't much activity in this challenge, and we were actually considering if it's worth it to launch a new round
    If there's enough interest among participants, we can still launch a new round, but I think that might be unlikely. We would rather gather all the forces, and try for a NIPS challenge next year.
    Michael Krax
    @michaelkrax
    @spMohanty is there going to be a video of the first round? i would participate in a round two but without seeing where the current bots succeeded/failed it's hard to improve the bots. For example i have no idea why the bots in the final evaluation of the multiplayer track were doing so much different than they were in the qualifier (the qualifier was close but the final evaluation was not). I assume TSAIL and i overfit for the qualifier maps or so while Ben was more generic... but it's hard to tell without replays or videos.
    SP Mohanty
    @spMohanty
    @michaelkrax : @mihahauke should have the videos, so I would let him answer that. But I doubt we will have time to hold a second round properly. But we will definitely have a new version of the challenge next year. Hopefully at NIPS