These are chat archives for arenanet/api-cdi

19th
Mar 2017
other requests probably dropped because the endpoints started timing out, I'm guessing
not quite sure
darthmaim
@darthmaim
Mar 19 2017 00:00 UTC
thats probably when the errors started and some bad crawler started retrying items over and over
while(true) {
  try {
    loadItemsFromApi(); break;
  } catch() {}
}
¯\_(ツ)_/¯
lol
this is also interesting
download (1).png
Archomeda
@Archomeda
Mar 19 2017 00:05 UTC
sooo
the amount of requests went up and up and up and up and up........ during downtime?
darthmaim
@darthmaim
Mar 19 2017 00:05 UTC
more requests while the API is down than when its up
haha yeah
darthmaim
@darthmaim
Mar 19 2017 00:06 UTC
well your average response time went way down :P
lmao glass half full right there
kinda tempted to turn everything except /v2/items back on
Archomeda
@Archomeda
Mar 19 2017 00:07 UTC
lol
but that's not really fixing the problem
well we had a good long run without needing to add rate limiting
Archomeda
@Archomeda
Mar 19 2017 00:07 UTC
post a reddit thread, "That person who hits /v2/items in a loop... yeah, fuck you"
darthmaim
@darthmaim
Mar 19 2017 00:07 UTC
:D
nah it's cool
i shoulda added rate limiting a long time ago i think
darthmaim
@darthmaim
Mar 19 2017 00:09 UTC
go for it
Archomeda
@Archomeda
Mar 19 2017 00:09 UTC
in the future, might be worth to also add support to listen to a specific client headers and log those, eg User-Agent?
darthmaim
@darthmaim
Mar 19 2017 00:09 UTC
^
we don't really have a good way to store that right now
like the request volume is too high to make it easy
Archomeda
@Archomeda
Mar 19 2017 00:10 UTC
so as an opt-in for developers, you can yell/contact a developer here whenever something goes wrong
hmm :(
definitely would be a nice feature to have
i think it'd have to be a bespoke thing though
darthmaim
@darthmaim
Mar 19 2017 00:11 UTC
limit everyone to 1request/minute unless they are in this chat room
the API has it's own separate little cluster so it's not infeasible
Eearslya Sleiarion
@Eearslya
Mar 19 2017 00:11 UTC
^
lol
Eearslya Sleiarion
@Eearslya
Mar 19 2017 00:11 UTC
Also dammit, I just started writing actual API scraping code for GW2Oracle today
what bad timing
X-Github-Token: <github oauth key>
Archomeda
@Archomeda
Mar 19 2017 00:11 UTC
and throw a 402 for requests without a user agent set :P
oh well, maybe in the very far future, have a application key on request for higher rate limiting :P
Eearslya Sleiarion
@Eearslya
Mar 19 2017 00:13 UTC
authorized user-agent keys for the backend scrapers
darthmaim
@darthmaim
Mar 19 2017 00:13 UTC
I guess you could limit on api keys already
Archomeda
@Archomeda
Mar 19 2017 00:13 UTC
but that's authenticated only
darthmaim
@darthmaim
Mar 19 2017 00:13 UTC
you can add that to every request if you want higher limit
Archomeda
@Archomeda
Mar 19 2017 00:14 UTC
not for unauthenticated requests
hm
don't think you want to mix those 2 kind of keys really
Eearslya Sleiarion
@Eearslya
Mar 19 2017 00:14 UTC
but then JS-based programs give out the website owner's tokeninfo and account at the least
Archomeda
@Archomeda
Mar 19 2017 00:15 UTC
then you would ask the user for an api key
it's something github does as well i think?
darthmaim
@darthmaim
Mar 19 2017 00:15 UTC
yes
thats why you have to authorize waffle.io for example
Archomeda
@Archomeda
Mar 19 2017 00:16 UTC
then there should also be a dummy api key with no account access
in normal use cases, you won't hit the rate limit anyway
so 99% of the users that use websites with api capabilities won't have to provide their api key
i suppose it's possible
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 00:53 UTC
What I really want is some kind of 'Hey, obviously new coder at <region/site> come here. Get help. Don't make use all give up our nice things?!"
Darqam
@Darqam
Mar 19 2017 00:55 UTC
why not just return that as an error?
"error: Yo you! yes you! come to gitter so we can explain why what you're doing is not OK"
Darrian
@rikkuness
Mar 19 2017 00:56 UTC
just throw the classic HTTP 418 ;D
on a legit note tho I have used 429 in the past for rate limiters
idk how many apps code for it
Twitter use 429 though and then set some headers to let you know when you can request again and stuff
hmm, this is going to be interesting
our rate limiting backend doesn't provide an interface for determining how many requests a user has remaining
really wanted to have X-Remaining-Requests or something ala twitter's response headers
Archomeda
@Archomeda
Mar 19 2017 01:36 UTC
that sucks :(
also when you make a request that gets rate limited, you're still charged for it
Archomeda
@Archomeda
Mar 19 2017 01:36 UTC
so you end up with a negative balance?
yep!
Archomeda
@Archomeda
Mar 19 2017 01:37 UTC
...
it caps out eventually
Archomeda
@Archomeda
Mar 19 2017 01:37 UTC
well, it will punish people that don't listen to the rate limit...
but not being able to see how many requests you've left is quite punishing this way
looking at twitter's docs, I can definitely add in X-Rate-Limit-Limit since that's just a static number
and it's on the application's side to track things
also I can make the rate limits fairly high probably
Archomeda
@Archomeda
Mar 19 2017 01:38 UTC
hmm, any chances you'll be able to implement it eventually?
given that the API is turned off, even a garbage solution is better than the current state
yeah
I need to run it by some other people
Archomeda
@Archomeda
Mar 19 2017 01:39 UTC
meh, it will work with this solution for now :smile:
and then wait a month or two for the change to propagate through the release process
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 01:39 UTC
I just can't imagine why that'd be the endpoint someone would hammer =/
Eearslya Sleiarion
@Eearslya
Mar 19 2017 01:39 UTC
So wait, does the 'balance' just reset to full after the timeout period, or does it add to your pool?
maybe they're fetching each item individually for each language :/
@Eearslya it's a running sum, so it's constantly filling back up
Eearslya Sleiarion
@Eearslya
Mar 19 2017 01:40 UTC
Okay yeah, definitely gonna need to write backoff routines, then
we don't have X-Rate-Limit-Reset which is kind of nice
Eearslya Sleiarion
@Eearslya
Mar 19 2017 01:40 UTC
lest the crawler just stay infinitely negative XD
Archomeda
@Archomeda
Mar 19 2017 01:41 UTC
i still vote to post on reddit, 'You! Yeah you! The one that hammers /v2/items' :P
yeah that's the downside though
@Archomeda eh i'm not gonna call people out
you shouldn't be able to take down an API via misuse
Archomeda
@Archomeda
Mar 19 2017 01:41 UTC
i know you won't, i won't either; i just find it funny to think of these things :D
ugh i need to send emails to people
you know what would be great fun
if the rate limiting component fell over
i probably should stand up a separate one instead of using the one that's shared with authentication attempts and such
Archomeda
@Archomeda
Mar 19 2017 01:42 UTC
(side note, we're running a poll in our guild of which raid boss we should have the first golden statue... i've thought of alternative names... so yeah, hence why i'm thinking about this stuff)
Replace moving now for hammering /v2/items
Archomeda
@Archomeda
Mar 19 2017 01:44 UTC
like for example...
Vale Guardian, red yellow green red blue blue blue
Slothasor, the lazy bastard
Xera, weeeeeee
anyway i'm not really feeling a live deploy over a weekend
so APIs gonna be down until monday
rip
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 01:55 UTC
Yeah. Don't live-deploy.
darthmaim
@darthmaim
Mar 19 2017 01:55 UTC
don't worry :heart:
Darqam
@Darqam
Mar 19 2017 01:55 UTC
:fire:
but yeah, take your time
rate-limiting is coming though
i don't see a way around it this time ;__;
Darqam
@Darqam
Mar 19 2017 01:55 UTC
feel better and take your time
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 01:55 UTC
I'd turn everything but v2/items back on if I could trust whoever that was not to try something else...
yeahhhh
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 01:55 UTC
My trust levels are low this week.
Darqam
@Darqam
Mar 19 2017 01:56 UTC
sucks you have to put effort and time into doing that :(
the changelist is already done lol
just needs some testing and stuff
@queicherius how many requests/minute does gw2e need to function?
I'm toying with the idea of 600 requests/minute (10 requests/second)
Archomeda
@Archomeda
Mar 19 2017 01:59 UTC
is that per endpoint or globally btw?
globally
Archomeda
@Archomeda
Mar 19 2017 02:00 UTC
alright
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 02:00 UTC
Are you thinking per source IP? Most of his are probably authenticated.
it's per-source IP
Archomeda
@Archomeda
Mar 19 2017 02:00 UTC
@queicherius also has server stuff running for statistics
Our rate-limiting bit doesn't support anything except per-IP limits
so adding in per-account limits gets kinda hairy
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 02:01 UTC
=/
Archomeda
@Archomeda
Mar 19 2017 02:01 UTC
how many accounts does gw2e have nowadays? can't seem to find it anymore
Darqam
@Darqam
Mar 19 2017 02:01 UTC
Lye, will it be written as 600/min or 10/second?
nvm, if I could read properly I wouldn't have had to ask
it'll be tracked as 600/min, so you can burst up to 600 requests.
i'm a little bit concerned about how this will affect TP sites, orz
Darqam
@Darqam
Mar 19 2017 02:04 UTC
well worst case you implement it, and someone comes complaining about it, then you negotiate
Archomeda
@Archomeda
Mar 19 2017 02:05 UTC
maybe post this on the forums as a heads-up?
you might get responses from other people that use the API but don't use gitter
yeah I'm totally going to post on the forums
I want to work out an initial set of numbers that make sense though
Archomeda
@Archomeda
Mar 19 2017 02:06 UTC
that's fair
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:07 UTC
Didn't he once say he only did 1/s on the backend?
Archomeda
@Archomeda
Mar 19 2017 02:07 UTC
what endpoints are most likely requested the most per IP basis... hmm
commerce and account stuff, most likely
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:07 UTC
wvw
Archomeda
@Archomeda
Mar 19 2017 02:07 UTC
i doubt it's wvw
darthmaim
@darthmaim
Mar 19 2017 02:08 UTC
you can get all wvw data with one request
Darqam
@Darqam
Mar 19 2017 02:08 UTC
I think most wvw ones have a self-imposed limit of updating every 5 seconds
Archomeda
@Archomeda
Mar 19 2017 02:08 UTC
but yeah, i agree with tp and account things
wvw certainly has the highest volume of all the endpoints, but it's also (probably) from the most unique IPs
Archomeda
@Archomeda
Mar 19 2017 02:08 UTC
commerce is also bulk-expanded right?
yeah it is
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:09 UTC
Hmm. I'd wager /v2/account/characters, /v2/account/bank, and /v2/account/materials then
Archomeda
@Archomeda
Mar 19 2017 02:09 UTC
shouldn't be that much of an issue for commerce then, if it's coded right
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:09 UTC
/v2/account/characters in particular, because you need one request per character
Darqam
@Darqam
Mar 19 2017 02:09 UTC
"if it's coded right"
Jonathan Andrist
@rwfrk_twitter
Mar 19 2017 02:10 UTC
Not-coded right is how we got here in the first place.
yeah applications that aren't are just gonna break
nothing i can do about that really
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:10 UTC
"git gud"
anyway, I'll post on the forums tomorrow
darthmaim
@darthmaim
Mar 19 2017 02:10 UTC
you can get all of commerce in 300 requests or so -> every 30 seconds
sent out some internal emails and want to give people a chance to reply
Archomeda
@Archomeda
Mar 19 2017 02:10 UTC
true, but this will force them to change their code, which is something we want really
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:11 UTC
hold their sites hostage until they fix their code XD
darthmaim
@darthmaim
Mar 19 2017 02:12 UTC
was the initial api crash caused by too many requests?
that's what it looks like, yeah
Archomeda
@Archomeda
Mar 19 2017 02:13 UTC
hmmmm... @lye, what do you think of having bulk-expanded account endpoints? bulk-expanded as in, multiple accounts? :P
@Archomeda ehhhhhhhh
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:13 UTC
?access_tokens
Archomeda
@Archomeda
Mar 19 2017 02:13 UTC
probably have to support POST then for multiple keys
seems more effort than it's worth
Archomeda
@Archomeda
Mar 19 2017 02:14 UTC
it does make things way more complicated
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:14 UTC
Something I actually do think would help sites like GW2E is a /v2/account/characters/inventories to get all character inventories at once; I think it'd help all of those account-value crawling and searching
Archomeda
@Archomeda
Mar 19 2017 02:14 UTC
i suppose you'll have to see what endpoint has the most requests per IP first
hah we'll see where the complaints come from, at least
darthmaim
@darthmaim
Mar 19 2017 02:15 UTC
:P
@Eearslya doesn't /v2/account/characters?ids=all include all inventories already?
Eearslya Sleiarion
@Eearslya
Mar 19 2017 02:17 UTC
...Probably.
derp.
darthmaim
@darthmaim
Mar 19 2017 02:18 UTC
Can't check right now, but according to the wiki it contains bags and equipment
Archomeda
@Archomeda
Mar 19 2017 02:27 UTC
i really have a slight urge to help improving the repo
noticed a few inconsistencies and a few wrong references
wtf
why does https://api.guildwars2.com/ return ["matches", "objectives", "upgrades"]
i am so confused
Archomeda
@Archomeda
Mar 19 2017 02:38 UTC
it doesn't for me
for me it's an empty array
darthmaim
@darthmaim
Mar 19 2017 02:38 UTC
[]
that's what it should be
apparently i have simply gone insane
and/or there is a ghost in my vpn connection
darthmaim
@darthmaim
Mar 19 2017 02:39 UTC
:ghost:
3spooky5me
Archomeda
@Archomeda
Mar 19 2017 02:53 UTC
free karma for the grabs people :smile:
slightly more on topic, it's well formulated and i don't think it's missing anything that's discussed here?
Darqam
@Darqam
Mar 19 2017 03:01 UTC
Well there's no finger pointing to whoever did the 15k spike... but I guess it's ok if that omited :(
darthmaim
@darthmaim
Mar 19 2017 03:02 UTC
:+1:
Darqam
@Darqam
Mar 19 2017 03:02 UTC
.... is that meant to be a finger pointing to me, or an actual thumbs up?
darthmaim
@darthmaim
Mar 19 2017 03:03 UTC
Thumbs up for the forum post
Was just bad timing just as a sent your message :P
:point_up:
:point_up_2:
Those are fingers pointing up
Darqam
@Darqam
Mar 19 2017 03:05 UTC
fair enough
Pat Cavit
@tivac
Mar 19 2017 05:33 UTC
Huh I literally never thought we'd see the day where we needed to rate limit APIs
Glad they're popular I suppose!
The real shame is that this is probably caused by one person doing something stupid
making it more complicated/annoying for everyone else
Darqam
@Darqam
Mar 19 2017 05:35 UTC
:/
I'll be honest I'm glad that's not me
There's always next time!
Never stop believing!
Eearslya Sleiarion
@Eearslya
Mar 19 2017 07:35 UTC
follow your dreams and one day you, too can take down an API
David Reeß
@queicherius
Mar 19 2017 09:17 UTC
@lye Well, getting the data for a single account is around 30 requests, so with a ratelimit of 600 requests per minute I can only get 20 accounts each minute, which means crawling 200.000 accounts would take nearly 7 days...
And that only takes into account the account statistics, not the item/achievement/skins etc that I pull, not sure how many requests that is
darthmaim
@darthmaim
Mar 19 2017 09:20 UTC
:/
David Reeß
@queicherius
Mar 19 2017 09:23 UTC
So I would need somewhere around 4,5k + probably another 500 for all the other small stuff a minute or start using load of different vps's for getting different ips or only crawl each account once every week :(
Werdes
@werdes
Mar 19 2017 09:33 UTC
Is there ratelimit whitelisting in your component? :sweat_smile:
David Reeß
@queicherius
Mar 19 2017 09:40 UTC
Actually, I forgot guilds in that calculation, so that's another 5*7 requests per account on top
So that'd be around 10k / minute required
Alternatively, with an endpoint that dumps all of the account data and does not require ~70 requests per account, 600 would be perfectly fine, and I could probably even do with lower than that
darthmaim
@darthmaim
Mar 19 2017 09:43 UTC
/v2/gw2efficiency
Michael Dougall
@madou
Mar 19 2017 10:17 UTC
Lmao
David Reeß
@queicherius
Mar 19 2017 11:44 UTC
Yet another option would be ratelimiting based on the endpoint, not globally. That'd block the people that don't know about bulk expansion and spam /items or /commerce, but not spread out requests
ChieftainAlex
@ChieftainAlex
Mar 19 2017 11:58 UTC
@lye I suppose each wiki request would trigger from the IP of whoever views the page, and afaik they're all using bulk expanded stuff, but is there a way to figure out how many times your api receives a query with "wiki=1" in the request url?
darthmaim
@darthmaim
Mar 19 2017 12:00 UTC
Not nearly enough to run into the 600/minute limit
Archomeda
@Archomeda
Mar 19 2017 16:01 UTC
@lye mentioned that currently the backend doesn't support endpoint-based ratelimiting, so that's gonna be a pain for now :(
ehm... @lye, is it possible to have a manual IP whitelist for the time being? i think it would benefit @queicherius in that regard
windwarrior
@windwarrior
Mar 19 2017 17:26 UTC
well, what is the n for which the rate limiter catches people listing all items per lang one by one, but not to negatively impact gw2e
and sadly there is no per-access-token ratelimit
Eearslya Sleiarion
@Eearslya
Mar 19 2017 17:56 UTC
@ChieftainAlex not retroactively; I'd have to add tracking for it.
I can manually whitelist gw2e but it's a solution i'm not really happy with
Adding a separate endpoint to dump all account info seems like a better solution
darthmaim
@darthmaim
Mar 19 2017 19:26 UTC
:+1:
@queicherius are your requests sent from gw2e.com's IP address? If not, PM me the IP address you're using to make requests.
An IP rate-limiter is trivial to work around by just using more VPS's so I feel much less bad about whitelisting on-demand.
Also if I have a hardcoded whitelist I can flag requests as "from gw2e" for graphing purposes :wink:
If we're gonna take that route I'm probably going to make a sticky about it so people know the rules can be bent a bit
Not that the gitter channel is necessarily a secret cabal
I think we need matching funny hats for that
David Reeß
@queicherius
Mar 19 2017 19:39 UTC
Yeah, backend requests are send from a bunch of different IPs that belong to me. I can PM them to you later. And I agree with the workaround, whitelisting would be nice instead. :)
(Frontend requests come from the user's IP, but I'm probably gonna start setting a header for tracking, just in case)
David Reeß
@queicherius
Mar 19 2017 20:09 UTC
And where do I go to get funny hats
Pat Cavit
@tivac
Mar 19 2017 20:16 UTC
I think having a whitelist is fine, we do need to call out that it exists and define the process for getting added to it though
ChieftainAlex
@ChieftainAlex
Mar 19 2017 20:28 UTC
can anyone recommend a slightly more visual traceroute than windows command line?
David Reeß
@queicherius
Mar 19 2017 20:30 UTC
Also @lye for the question you asked in the forum - /v2/build (which I check for uptime) is flickering up and down as well
I've no idea why /v2/build would be sporadically up
I need to add an explicit off switch to the APIs
uhh
@tivac the end of that solid red bar is when I changed from "false" to "arena"
Archomeda
@Archomeda
Mar 19 2017 20:53 UTC
that's an amazing graph
it really makes me want to have graphs like https://status.github.com/
hah yeah.
someone mentioned on the forums having an official status page for the API
Pat Cavit
@tivac
Mar 19 2017 20:57 UTC
doing that correctly would be a PITA
since it would need to be on someone else's infrastructure
@tivac what if we had a Js2ApiStatus which grabbed metrics every 5 minutes
Pat Cavit
@tivac
Mar 19 2017 20:57 UTC
that would by definition still be on our infra
oh, I mean there's that (correct) approach too
Pat Cavit
@tivac
Mar 19 2017 20:58 UTC
how would it grab metrics?
probably asking spawn and doing aggregation itself
Pat Cavit
@tivac
Mar 19 2017 20:58 UTC
if we were going to do the easy status thing we could put it on 2api
yeah lol fair
lye @lye registers "isthegw2apiup.com"
<h1>no</h1>
I don't see how requests were making it past
man it's going to be such a !!fun!! week
David Reeß
@queicherius
Mar 19 2017 21:20 UTC
i was thinking about making a status page for the api + efficiency
the one on the forum was down last time I checked :(
darthmaim
@darthmaim
Mar 19 2017 21:21 UTC
I almost started working on one yesterday
David Reeß
@queicherius
Mar 19 2017 21:21 UTC
atm i only use uptimerobot
(and newrelic for the servers)
there's always https://www.statuspage.io/ but they are really expensive :(
darthmaim
@darthmaim
Mar 19 2017 22:06 UTC
there is https://cachethq.io/ as open source alternative
you have to host it your self though
but shouldn't be to bad to write your own, just a cronjob that pings a bunch of urls and throws the response times and status codes into a db
and a simple fronted showing that data
David Reeß
@queicherius
Mar 19 2017 22:09 UTC
uuuh, that looks pretty good
at the time I checked I only found one running on github issues, but that one looks way better
Darrian
@rikkuness
Mar 19 2017 22:10 UTC
nodejs script and influxdb ;D influx all the things