These are chat archives for arenanet/api-cdi

19th
Feb 2016
smiley
@codemasher
Feb 19 2016 00:46
anyone alive? :o
Pat Cavit
@tivac
Feb 19 2016 00:46
no
darthmaim
@darthmaim
Feb 19 2016 00:46
never
smiley
@codemasher
Feb 19 2016 00:46
i made more thing! burn the API!
codemasher/php-curl@2023ad0
why aren't you using PDO?
smiley
@codemasher
Feb 19 2016 00:47
because mysqli seems to be slightly faster
but i have a PDO driver included if you favor that one ;)
heh fair.
also for some reason my gitter notifications stopped working :|
smiley
@codemasher
Feb 19 2016 00:49
these are some kind of mystery anyay
windwarrior
@windwarrior
Feb 19 2016 01:05
was trying to kill sabetha, but she has a flamethrower... its just not fair
smiley
@codemasher
Feb 19 2016 01:05
turns out the PDO driver is pretty picky SQLSTATE[42S02]: Base table or view not found
smiley
@codemasher
Feb 19 2016 01:19
for you @lye codemasher/php-curl@47b15dc ;)
smiley
@codemasher
Feb 19 2016 01:26
ha! i broke it and i won't do anything about it because i go to bed now. gn8!
night :>
Steven Liekens
@StevenLiekens
Feb 19 2016 10:28
it's almost noon here
how come there are no timezones in GW?
flat earth, anyone?
(only just now noticed that 9 hours passed)
Simon Amor
@simonamor
Feb 19 2016 12:13
maybe the world is so much larger than the small area we see on zooming out, and as a result, it's all in the same timezone?
smiley
@codemasher
Feb 19 2016 15:27
yup, ArenaNet Bureau Time (ABT) http://www.guildwiki.de/wiki/ArenaNet-Zeit
hmm, seems like i don't get much below 8 minutes with my updater
but then again: i wonder if you guys do even use some kind of throttling or limited concurrent requests...
so i can imagine ~30s only with all requests shot out at once and not even checking the responses
darthmaim
@darthmaim
Feb 19 2016 15:33
nope, me and @Eearslya parsed the responses and got below 30s
smiley
@codemasher
Feb 19 2016 15:33
but did you also drop them to DB? :D
darthmaim
@darthmaim
Feb 19 2016 15:34
nope, but @Eearslya did
didn't make any difference
smiley
@codemasher
Feb 19 2016 15:34
i benchmarked my DB driver yesterday and it takes ~30-45s to create those ~55k lines from the /v2/items fetch
darthmaim
@darthmaim
Feb 19 2016 15:35
huhh
smiley
@codemasher
Feb 19 2016 15:35
a drop of a single response of ~200 items takes up to 0.2s
darthmaim
@darthmaim
Feb 19 2016 15:35
shouldn't take that long
insert them all at once
still going to be .2s
...
smiley
@codemasher
Feb 19 2016 15:52
[2016-02-19T15:51:23+00:00][  518.75219s] [1330  ][en] 0e854acae9eb7a0986670889c4b03744 updated
[2016-02-19T15:51:23+00:00][  518.75445s] MultiRequest::fetch() finished
php7, win, cli (not phpstorm cli), no xdebug
smiley
@codemasher
Feb 19 2016 16:02
but then again: what's 7 min for that amount of data? i mean, we don't do this permanently anyway XD
getting closer
const CONCURRENT = 1000;
const CHUNK_SIZE = 50;
smiley
@codemasher
Feb 19 2016 16:09
i think i've just self-ddos'd our company's interwebs
Eearslya Sleiarion
@Eearslya
Feb 19 2016 16:16
My fetcher could fetch, parse, and commit all items for 1 language into an empty table in 19s
Could probably lower that further if I did that all simultaneously
smiley
@codemasher
Feb 19 2016 16:40
one language doesn't count :D
David Reeß
@queicherius
Feb 19 2016 16:41
I take ~30 secs for all items in all languages, and that seems to be capped by API response speed (on a 8gb memory / 3ghz / 200 mbit vps)
And it takes the same amound of time with 2gb memory 3ish ghz and 10mbit (sad times uni internet)
smiley
@codemasher
Feb 19 2016 16:43
i doubt that you would hit the AWS bandwidth cap - there's also stuff like response time/ping etc which all influences performance;)
David Reeß
@queicherius
Feb 19 2016 16:44
I never said bandwidth cap, I said response speed. :D
smiley
@codemasher
Feb 19 2016 16:46
but it's in fact the parallel execution of requests.
so i won't hop on the performance train and have the only updater which will work once API request limits are a thing :D
David Reeß
@queicherius
Feb 19 2016 16:48
That said, speed really doesnt matter for crawling all items. You're not gonna do that a lot anyway :D
Eearslya Sleiarion
@Eearslya
Feb 19 2016 16:48
I'm
...Thanks phone.
smiley
@codemasher
Feb 19 2016 16:49
i know that for sure, thats why i didn't care much about it
Eearslya Sleiarion
@Eearslya
Feb 19 2016 16:49
I'm tempted now to make an API fetcher as fast as I can..
smiley
@codemasher
Feb 19 2016 16:49
it's just, now that i've basically rewritten the updater, i'd like to get best performance for sure ;)
David Reeß
@queicherius
Feb 19 2016 16:51
And I am very curious how API limits will turn out. IP based limits would be not fun for stuff like tunneling user requests through a server... :c
Pat Cavit
@tivac
Feb 19 2016 18:03
our APIs shouldn't really need tunnelling through your server, generally
David Reeß
@queicherius
Feb 19 2016 18:08
  • Proxying character API requests because the users want to share their characters without sharing API keys
  • Querying all account endpoints for the overview (currently that's multiple requests on the user side for me, but I want to do that on the server and just spit back the results at the user, since all the calculations and data fetching is not light on network)
So, only authenticated APIs I guess.
Also, heads up there might be some downtime today while I perform some shenanigans
smiley
@codemasher
Feb 19 2016 19:12
great, tell me when, so i can test how the failed requests return to the stack :D
okay it should start catching fire shortly