Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Paulo de Freitas Coleta Neto
    @paulodfreitas
    hi guys, first congrats for the great work
    i'm a computer science student from Brazil and i'm trying to work with Dota for a data mining project. Early this year a colleague sent @howardchung a e-mail asking about using Yasp data
    so you told us to grab then with ?json=1, that works fine
    currently to avoid 500 errors, that happen when we request a "invalid match id" or simply a match_id that you didn't grab yet
    Paulo de Freitas Coleta Neto
    @paulodfreitas
    i'm using steam api to retrieve matches, and thats also good because I can filter some types of matches
    to retrieve matches ids
    but i'm still get to many 500 errors, that brings a useless data traffic to your servers,
    are there any way that i can make just the useful requests?
    thanks
    Paulo de Freitas Coleta Neto
    @paulodfreitas
    i'm also already filtering matches by lobby_type
    Howard Chung
    @howardchung
    Unfortunately not since we dont know what matches you want and we don't have every match. You could try running the parser/retriever yourself to parse just the replays you are interested in
    Aleksandar Bukurecki
    @natoshi
    Anyone have details on how to get the tower log? I am interested in when each tower was destroyed
    Howard Chung
    @howardchung
    there's a chat event that has "tower destroyed" but doesn't say which tower
    Aleksandar Bukurecki
    @natoshi
    ah, thats annoying
    Matt Doak
    @sklink
    skill.js can't seem to keep up with scanner.js. I put skill.js in cluster mode and gave it 3 processes but I'm not sure that helped anything. Have you just been accepting that some matches won't be marked with skill or do you have a solution for this?
    I see mention of a TODO to cluster spawn a separate worker for each skill. That's something I may be able to do and create a PR for if you can give me a little direction--I assume each process needs to receive it's own config to know which skill to process for.
    Howard Chung
    @howardchung
    We aren't really seeing skill loss in production so I never implemented it.
    really it just needs a single env var passed in (like SKILL)
    tarriel
    @tarriel
    So I just did a clean install to get yasp running, I had to install nf manually, postgress 9.5 doesn't install, but I found 9.3 (I think there was another library I had to find but can't remember...) The win % on the overview and peers are 0%, the parser doesn't seem to start thought it is getting the URL, I think you must have changed something here when you introduced the proxies so it is probably my .env file apart from that it is awesome
    Howard Chung
    @howardchung
    9.5 isn't released yet so we use 9.4 in prod.
    if it is getting the url it is probably a problem with parser, not env file
    i dont know if you managed to do maven build (java)
    tarriel
    @tarriel
    when I go java -jar dirname "$0"/target/stats-0.1.0.jar < 1730707255.dem it gives me a lot of text that looks like a valid replay parse
    Howard Chung
    @howardchung
    I forgot to put parser.js in the Procfile.dev
    pull and try again
    we are moving away from workParser/workServer pair soon I think
    tarriel
    @tarriel
    Thanks, getting 404 errors now, so the oldest replays have expired I guess.
    still getting %0 win rate on the overview and peers page.
    Howard Chung
    @howardchung
    Yes Valve deletes replays older than 7 days so you need to have them already if you want to parse them (and put them somewhere network accessible and edit the code to not look for URL)
    I don't know about the other problem though
    tarriel
    @tarriel
    perhaps I shouldn't feed as much...
    tarriel
    @tarriel
    do you have any code to put the data files you uploaded back into the database?
    Howard Chung
    @howardchung
    you can look at queries.js for sample insert queries. But you can use any language/db you want
    Derek Li
    @likered
    This message was deleted
    This message was deleted
    Evaldo Bratti
    @evaldobratti

    Hi there
    I have already talked to you about a functionality that I wish to develop on yasp. I have already implemented this and it's working on dotaparty.me, but I want to learn some new technologies.
    I have a question about the fullhistory.js
    It starts to collect my match ids, and when I think it should start to download them, it only download one match each time

    23:47:57 fullhistory.1 | 712 matches found, 4 already in db, 708 to add
    23:47:57 fullhistory.1 | 2016-02-02T01:47:57.437Z - info: getData: http://api.steampowered.com/IDOTA2Match_570/GetMatchDetails/V001/?key=<ommited>&match_id=<ommited>
    23:47:59 fullhistory.1 | 2016-02-02T01:47:59.369Z - info: getData: http://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001/?key=<ommited>&account_id=<ommited>&matches_requeste...

    and then it starts to collects all matches ids again
    is it correct or am I missing something?

    Howard Chung
    @howardchung
    if you have a feature idea, please open an issue for it on github!
    so how it works is that it scans all the heroes and builds a set of all the match ids it found
    then it removes the ids that are already found in the db for that user
    then it does a getmatchdetails on each remaining ID
    a fh request is queued each time a user signs in
    so if you signed in multiple times we might be processing them in parallel
    you can use the status page to see how many fh are in queue
    generally it's idempotent so you end up with the same result but you might do some extra api reqs if you're doing the the same fh request in parallel
    Matt Doak
    @sklink

    Hey YASP!

    I'm having some troubles with the Docker setup. Would you be able to give me a hand?

    scanner-0 connecting redis://127.0.0.1:6379/0
    scanner-0 rebuilding sets
    scanner-0 [SCANNER] listening at http://:::5500
    scanner-0 saving sets to redis
    scanner-0 set build complete
    scanner-0 2016-05-05T05:28:38.151Z - info: getData: http://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001/?key=152916B4010FD803493134788B1C0FE3
    scanner-0 { [Error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.] command: 'SET', code: 'MISCONF' }

    I've run into this twice, once it stopped at ~150k matches and the 2nd time it stopped at ~125k matches. I have 300GB of space, so there shouldn't be any issues for a while. I've confirmed this by checking up the disk usage and it's sitting at about 2%. From what I've read it's either a permissions issue or an issue with the redis volume. Have you seen this error before? Do you have any suggestions on how to dig deeper?

    Howard Chung
    @howardchung
    that means you are out of RAM
    either you need to disable stop-writes-on-bgsave-error (google it) or free up some RAM
    also I recommend discord, we don't really use this anymore :)
    Matt Doak
    @sklink
    Thanks, I just jumped over to that. Really appreciate the help!
    Jason Sun
    @sunapi386
    What's going on with this gitter channel, do we have another one for YASP? Looks like the last message was May 05 2016.