These are chat archives for arenanet/api-cdi

21st
Mar 2017
yeah i need to figure out what the story is gonna be for getting rate limit exemptions
figure it's going to be (1) have one or more static IPs, (2) an email address, (3) hopes and prayers
i'm really hype about having per-site graphs of api usage
but i'll worry about that when i get the ratelimiting bit deployed (hopefully tomorrow)
Eearslya Sleiarion
@Eearslya
Mar 21 2017 03:18
3) participation in the api-cdi gitter so we have a quick line of contact
and/or a guage on programming skill
Archomeda
@Archomeda
Mar 21 2017 03:38
4) your soul
Jonathan Andrist
@rwfrk_twitter
Mar 21 2017 04:02
Absolutely a fan of 3. Gotta have super quick point of contact somehow for whitelist "Something broke man"
Jonathan Andrist
@rwfrk_twitter
Mar 21 2017 04:44
Maybe require an in-game primary account name as another point of contact
Edgar Doiron
@Coffee4cr
Mar 21 2017 12:59
5) Profit!
Werdes
@werdes
Mar 21 2017 16:03
i doubt those are logged?
Eearslya Sleiarion
@Eearslya
Mar 21 2017 16:13
No, nothing's logged right now, just the amount of requests. But I think he was saying that whitelisted sites should have to use User-Agents
smiley
@codemasher
Mar 21 2017 16:30

yea, re this

figure it's going to be (1) have one or more static IPs, (2) an email address, (3) hopes and prayers

@Eearslya "they". thank you <3
Eearslya Sleiarion
@Eearslya
Mar 21 2017 16:31
wat
smiley
@codemasher
Mar 21 2017 16:32
user the pronouns they/them when you refer to me ;)
Eearslya Sleiarion
@Eearslya
Mar 21 2017 16:34
well alrighty then
Jonathan Andrist
@rwfrk_twitter
Mar 21 2017 16:47
Aww. Lawton are you that Gold Raider? sadface No one deserves siege on their body.
yeah i am
i throw siege on their bodies
it's all fun
lol
i'm laughing more than I should on this
thanks gitter for the HUGE embed
Pat Cavit
@tivac
Mar 21 2017 17:37
ahahahaha that's so fucking stupid
I love it
Werdes
@werdes
Mar 21 2017 18:27
rumors say it swims once in the water.
now that's the stuff i'd pay for, if i actually was into minis
Edgar Doiron
@Coffee4cr
Mar 21 2017 18:28
well you can be into "this" mini
Nabrok
@Nabrok
Mar 21 2017 18:31
I'd have a mini out much more often if they weren't seen by enemy players in wvw
hint hint
Edgar Doiron
@Coffee4cr
Mar 21 2017 18:37
who would suspect a fish though
Jonathan Andrist
@rwfrk_twitter
Mar 21 2017 18:49
Hey, at least minis respect the stealthed state of their owner now.
Nabrok
@Nabrok
Mar 21 2017 18:50
Wait, they do? When did that happen?
Pat Cavit
@tivac
Mar 21 2017 18:50
holy shit it totally does start swimming in water
darthmaim
@darthmaim
Mar 21 2017 18:52

Dear API Devs,

On Behalf of the World Vs World Community we are requesting the WvW part of the API back. The health of the community depends on these services. I understand that it is broken or parts of it are broken. However it is imperative that sites such as mos.millenium.org/na/matchups/ and https://wvwintel.com/ are working even if nothing else works. Please hurry with up with it we need it.

some people...
where's that from? lol
darthmaim
@darthmaim
Mar 21 2017 18:52
forum
orz
Archomeda
@Archomeda
Mar 21 2017 18:55
Lol
darthmaim
@darthmaim
Mar 21 2017 18:55
I would wait to turn on the wvw endpoints for one more day after you get the api running again...
Eearslya Sleiarion
@Eearslya
Mar 21 2017 18:56
"it is imperative" why
press B
Nabrok
@Nabrok
Mar 21 2017 18:56
I missed the bit where the wvw community chose him as spokesperson.
Darqam
@Darqam
Mar 21 2017 19:01
heh " I understand that it is broken or parts of it are broken." Obviously not....
"Please re-enable a broken thing without fixing it"
Archomeda
@Archomeda
Mar 21 2017 19:02
What was the API license again (or w/e it's called)? Something about no guarantee?
Might want to remind I'm about that :smile:
Him* thanks phone
Darqam
@Darqam
Mar 21 2017 19:03
I feel like answering...
I really do...
darthmaim
@darthmaim
Mar 21 2017 19:06
SLA
(Service Level Agreements)
Eearslya Sleiarion
@Eearslya
Mar 21 2017 19:07
Yeah, the API was specifically made with no SLA
Archomeda
@Archomeda
Mar 21 2017 19:08
Yeah
Megan Walker
@lambada
Mar 21 2017 19:12
Something Something cloud doesn't go down something ;)
i'd rather it go down than be charged $100k/hour for "extra capacity" :P
Eearslya Sleiarion
@Eearslya
Mar 21 2017 19:13
wait what
what charges $100k/h
hiring a fleet of cloud-capturing helicopters
Megan Walker
@lambada
Mar 21 2017 19:14
:joy:
btw. any chance on GeoJSON once the API is back up and running?
Archomeda
@Archomeda
Mar 21 2017 19:20
I think improving the ratelimit system wille more priority atm
Will have* thanks again phone
more like going home and crying myself to sleep is more of a priority
darthmaim
@darthmaim
Mar 21 2017 19:21
<3
Archomeda
@Archomeda
Mar 21 2017 19:21
Aw :worried:
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:21
"in alignment with other ArenaNet official policy on WvW, we've decided to switch off the WvW endpoint permanently"
Archomeda
@Archomeda
Mar 21 2017 19:21
We still love you lye <3
Darqam
@Darqam
Mar 21 2017 19:21
But Smiley, he is the voice of WvW community, he is of course serious
David Reeß
@queicherius
Mar 21 2017 19:21
You got this lye. <3
Archomeda
@Archomeda
Mar 21 2017 19:22
Also lol @codemasher
smiley
@codemasher
Mar 21 2017 19:24
"on behalf of the wvw noobs who aren't able to follow the zerg"
just noticed i still have this in my forum signature http://gw2wbot.darthmaim.de/smiley/
is this still the current database, @darthmaim ?
darthmaim
@darthmaim
Mar 21 2017 19:29
uhhm
yup
contains the latest items of gw2treasures
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:29
was the day the api tanked the same as when gw2armory released the embed?
Dominic
@DomGrieco
Mar 21 2017 19:29
no
smiley
@codemasher
Mar 21 2017 19:30
ha, nice - didn't expect this script working for that long.. .:D
Archomeda
@Archomeda
Mar 21 2017 19:30
Doubt it, @madou knows the exact date probably
@ChieftainAlex lol
Dominic
@DomGrieco
Mar 21 2017 19:30
embeds were released over 2 weeks ago
I was under the impression that the embed used gw2a's internal API
Dominic
@DomGrieco
Mar 21 2017 19:30
it does
since it's probably faster than using the gw2 api lol
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:31
i thought that until none of the data popped up on qtfy skills when the api went down..
but I'm a noob so I got it wrong /giveup
Dominic
@DomGrieco
Mar 21 2017 19:31
I'm pretty sure thats because he just did updates
darthmaim
@darthmaim
Mar 21 2017 19:34

well, I'm getting this:

GET https://api.guildwars2.com/v2/specializations?ids=36,11,18&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/specializations?ids=4,11,18&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/specializations?ids=36,51,18&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/skills?ids=30189,14410,14404,14405,14407,30343&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/skills?ids=30189,14403,30074,14405,14407,30343&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/skills?ids=29941&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/skills?ids=30189,14410,30074,14404,30343&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/skills?ids=30189,14404,14405,14407,30343&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/specializations?ids=36,11,18&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/skills?ids=29941&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/specializations?ids=4,11,18&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=46766,46770,24562,24639&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=46762,24615,24554&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=48129,48077,48073,48074,48130,48072,24714&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=48129,48077,48073,48074,48130,48072,24836&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67302,67298,37075,37086,39308,37049&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=39233,39232,37075,37086,39308,37049&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67531,68633&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=70838,76271,24562,24605&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=70504,24562,24605&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=74412,67455,76377,76776,77143,72548,24845&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67531,12467&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67906,77926,77413,77958,77380,80434&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=48917,12464&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=74412,67455,76377,76776,77143,72548,71425&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=70504,24560,24605&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=70838,76271,24560,24605&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=46762,36053,24615&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=46762,24868,24615&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=46762,24658,24615&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=79722,71659,75427,76150,70596&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=48129,48077,48073,48074,48130,48072,24836&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67302,67298,37075,37086,39308,37049&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67531,68633&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=67906,77926,77413,77958,77380,80434&lang=en 404 (Not Found)
GET https://api.guildwars2.com/v2/items?ids=48917,12464&lang=en 404 (Not Found)

just loading https://qtfy.eu/build/warrior

smiley
@codemasher
Mar 21 2017 19:35
caching & databases are for nubs :D
Eearslya Sleiarion
@Eearslya
Mar 21 2017 19:35
There's a lot of duplicate IDs in those, too
smiley
@codemasher
Mar 21 2017 19:36
math hard
Dominic
@DomGrieco
Mar 21 2017 19:37
thats because of the way the embeds are used on the website
multiples that is
he might of changed something recently because I was pretty sure it was from his internal api
Archomeda
@Archomeda
Mar 21 2017 19:41
Ah, yeah this is a downside of having embeds
Loading things separately
darthmaim
@darthmaim
Mar 21 2017 19:43
those embeds are just one js file
not iframes or something like that
and you can even queue requests from different iframes if you really wanted to
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:44
can't they just make it load the js after loading every individual frame?
darthmaim
@darthmaim
Mar 21 2017 19:44
there are no frames
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:44
element with data attributes*
Dominic
@DomGrieco
Mar 21 2017 19:45
the js is loaded in the header
Archomeda
@Archomeda
Mar 21 2017 19:48
Oh
I was under the impression it was iframes
darthmaim
@darthmaim
Mar 21 2017 19:48
nope, its somewhere in the 342kb of js loaded from gw2armory
Archomeda
@Archomeda
Mar 21 2017 19:50
Oh wow
That's quite large
Darrian
@rikkuness
Mar 21 2017 19:50
das what she said
ohhhhhhh
darthmaim
@darthmaim
Mar 21 2017 19:51
342kb of js just for tooltips
the whole buildeditor we are working on is just 100 or so when build for production
Archomeda
@Archomeda
Mar 21 2017 19:53
Then it is possible to reduce the amount of requests
darthmaim
@darthmaim
Mar 21 2017 19:53
indeed
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:54
should be easy enough to combine all of the above requests darthmaim quoted in 3 queries.
darthmaim
@darthmaim
Mar 21 2017 19:54
thats the js that loads the js
blob
ChieftainAlex
@ChieftainAlex
Mar 21 2017 19:55
lol 4 scripts even better
Archomeda
@Archomeda
Mar 21 2017 19:56
Ouch
darthmaim
@darthmaim
Mar 21 2017 19:56
also 150kb webfonts and 300kb tooltip background
gw2armory loads a total of 781kb
Archomeda
@Archomeda
Mar 21 2017 19:57
Well, does at least 304 work?
darthmaim
@darthmaim
Mar 21 2017 19:58
yep
Archomeda
@Archomeda
Mar 21 2017 19:58
Ah good :)
darthmaim
@darthmaim
Mar 21 2017 19:59
but still
Aralicia
@Aralicia
Mar 21 2017 19:59
Hi all ! Tbh, what's bothers me the most with gw2armory is the 865Ko png that display a bunch of fake clouds, with a meh resolution.
darthmaim
@darthmaim
Mar 21 2017 19:59
blob
https://qtfy.eu/build/mesmer/ makes 408 requests with a total of 3.1MB
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:00
how popular is qtfy?
darthmaim
@darthmaim
Mar 21 2017 20:00
no idea
Archomeda
@Archomeda
Mar 21 2017 20:02
Quite popular under raiders
Do they have cloudflare set up?
Nabrok
@Nabrok
Mar 21 2017 20:02
Yeah, it's kind of the go-to site for raid builds.
Dominic
@DomGrieco
Mar 21 2017 20:03

Sessions
130,172

Users
36,597

Pageviews
1,247,971

Yes I have cloudflare set up
That's analytics since release
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:03
36597*408?
oh right
Dominic
@DomGrieco
Mar 21 2017 20:03
release was 15 days ago?
I think
John Carroll
@johncarroll
Mar 21 2017 20:04
they broke their 25gb per month limit on enjin by a lot
Dominic
@DomGrieco
Mar 21 2017 20:04
march 6th
Aralicia
@Aralicia
Mar 21 2017 20:04
blob
Archomeda
@Archomeda
Mar 21 2017 20:04
Wow
darthmaim
@darthmaim
Mar 21 2017 20:04
that site is a performance nitghmare
Aralicia
@Aralicia
Mar 21 2017 20:04
Not API-related, but the site could gain from not loading ALL the youtube embed at load.
Dominic
@DomGrieco
Mar 21 2017 20:05
it doesnt lol
its loading a tiny screenshot for the lazy loaded video
darthmaim
@darthmaim
Mar 21 2017 20:05
hqdefault is 1080p (or was it 720p?)
Dominic
@DomGrieco
Mar 21 2017 20:05
if all those videos weren't lazy loaded
the page would be atleast 30mb load
Archomeda
@Archomeda
Mar 21 2017 20:05
YouTube embeds are sometimes slow to load as well, at least that was because of AdBlock and Firefox, a couple of years back
Dominic
@DomGrieco
Mar 21 2017 20:06
do a gtmetrix @Aralicia
Archomeda
@Archomeda
Mar 21 2017 20:06
At least I don't use that anymore
Dominic
@DomGrieco
Mar 21 2017 20:06
those images are nothing on the load
damn wiki images are what kill load times and even those are lazy loaded, most of them
Archomeda
@Archomeda
Mar 21 2017 20:07
Haha, Mediawiki is just slow to begin with
darthmaim
@darthmaim
Mar 21 2017 20:07
well you have to start somewhere, that page is horribly slow and loads way to much
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:07
uh why would you pull images off the wiki?
Dominic
@DomGrieco
Mar 21 2017 20:07
Ive told the qt people to stop using wiki images
I just made a template for them to fill with builds
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:10
everything else is forgivable, but there's a font called "raleway" whose spelling irks me
darthmaim
@darthmaim
Mar 21 2017 20:10
but 1,247,971 pageviews in 15 days, making 50 api requests (48/sec) each which could be combined to <5... (even when only 50% of the viewed pages are build pages). I'd say we found the reason the API is crashing
Archomeda
@Archomeda
Mar 21 2017 20:10
Wait, they use wiki images?
Aralicia
@Aralicia
Mar 21 2017 20:10
@DomGrieco my bad. Since hqdefault.jpg are used by the embed themselves, I assumed the embed where loaded. (funny sidenote, I took the time to gtmetrix, and got a nice "write failed : Broken pipe" error)
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:10
I can't see any wiki image requests in the console
Dominic
@DomGrieco
Mar 21 2017 20:12
wiki images are in cc breakdown, lazy loaded
darthmaim
@darthmaim
Mar 21 2017 20:12
1247971*50/15/24/60/60 = 48.147 API requests per second
Aralicia
@Aralicia
Mar 21 2017 20:12
@darthmaim good point. That would be about 175k call/hour on average.
Dominic
@DomGrieco
Mar 21 2017 20:13
cloudflare flex ssl has been causing a lot of issues on the site that I've been messing with might be why gt just failed
most of the website traffic was the first 3 days of release
I dont recall the api crashing
more then 30% of those analytics are from initial release
windwarrior
@windwarrior
Mar 21 2017 20:15
it was /v2/items/ that blew up
wasnt it
Jonathan Andrist
@rwfrk_twitter
Mar 21 2017 20:15
Yes
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:15
pity it was v2/items/, otherwise we could blame the official wvw representative
windwarrior
@windwarrior
Mar 21 2017 20:15
well qtfy has some items on the site
Archomeda
@Archomeda
Mar 21 2017 20:15
Lmao
windwarrior
@windwarrior
Mar 21 2017 20:15
but I can hardly imagine that being too much
darthmaim
@darthmaim
Mar 21 2017 20:16
https://qtfy.eu/build/mesmer/ is making 33 requests to /v2/items
Dominic
@DomGrieco
Mar 21 2017 20:16
your using the most populated page lol
Archomeda
@Archomeda
Mar 21 2017 20:16
Nah, the API blew up because of something else
Dominic
@DomGrieco
Mar 21 2017 20:16
mesmer is like 2x the content of all the other builds
mesmer and druid
Archomeda
@Archomeda
Mar 21 2017 20:16
Qtfy was online for a while already
When it blew up
Aralicia
@Aralicia
Mar 21 2017 20:17
Well, at least we're pretty sure it's not Jormag either this time.
darthmaim
@darthmaim
Mar 21 2017 20:17
warrior is still making 25 requests to items, 24 too many
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:17
..probably one guy wondering why his IE5 isn't displaying qtfy just hitting reload when everything breaks.
Archomeda
@Archomeda
Mar 21 2017 20:17
Well, I can't disagree with the amount of requests per page. It can be made less
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:17
anyway regardless of what tipped the API over the edge, rate limiting is a good idea
windwarrior
@windwarrior
Mar 21 2017 20:18
yeah but those will be cached as well
Archomeda
@Archomeda
Mar 21 2017 20:18
But I very much doubt it was the main cause
John Carroll
@johncarroll
Mar 21 2017 20:18
site doesn't even have full minstrels chrono. don't know why anyone bothers
windwarrior
@windwarrior
Mar 21 2017 20:18
dont think qtfy is putting much strain on the API
Jonathan Andrist
@rwfrk_twitter
Mar 21 2017 20:18
Qtfy wasnt it
Nothing new there.
Dominic
@DomGrieco
Mar 21 2017 20:18
metabattle guy was working on implementing embeds
windwarrior
@windwarrior
Mar 21 2017 20:18
also, minstrel is situational
Dominic
@DomGrieco
Mar 21 2017 20:18
but I dont think he did anything past testing yet
John Carroll
@johncarroll
Mar 21 2017 20:19
minstrels is good for everything
windwarrior
@windwarrior
Mar 21 2017 20:19
not for a speedclearing guild like qT
lets try to remember that
domi illu on golem is 13k dps, minstrel like 4 at most
and if they can get away with not playing any strong healers
they will
John Carroll
@johncarroll
Mar 21 2017 20:20
lets try to remember this is an api chat :D
Dominic
@DomGrieco
Mar 21 2017 20:21
that qt site was literally a rushed project that gives anyone the ability to update the site while spending the least amount of money, cms site
nothing fancy
Archomeda
@Archomeda
Mar 21 2017 20:22
Yeah that's fair
Doesn't mean that there isn't any room for improvement :D
Dominic
@DomGrieco
Mar 21 2017 20:24
oh yeah things will get fine tuned I just barely have time to spend on it :(
Archomeda
@Archomeda
Mar 21 2017 20:24
But yeah, moving away from enjin on such a short notice, I understand that b)
Dominic
@DomGrieco
Mar 21 2017 20:25
I have paid clients that consume most of my time
Archomeda
@Archomeda
Mar 21 2017 20:25
;) * thanks again 3x phone
Ohh I see
Dominic
@DomGrieco
Mar 21 2017 20:25
either way back to api, its supposed to be using gw2a internal api
which is cached on a 7 day cycle
something just not working as intended there
@madou will check it out
Archomeda
@Archomeda
Mar 21 2017 20:29
:+1:
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:36

so sending ajax identifier headers is just as simple as adding

$.ajaxSetup({    
    headers: { 'X-please-don-t-ban-me': 'signed chieftain alex' }    
});

before $.getJSON or $.ajax?

pfft markdown.
@ChieftainAlex except that request would no longer qualify as a "simple request" for CORS purposes and would trigger the browser to submit an OPTIONS preflight
Which we don't support, unfortunately.
ChieftainAlex
@ChieftainAlex
Mar 21 2017 20:50
welp. nevermind that then.
Good luck with fixing the API and let us know what you want us to change (please let it not be the wiki that broke the API...)
pretty sure it's not the wiki
Darqam
@Darqam
Mar 21 2017 20:51
inb4 some bot mass refreshed a page on the wiki to crash everything
wiki would have fallen over first
Darqam
@Darqam
Mar 21 2017 20:52
fair
API isn't up so I can't find interwikis for the new fish mini, nor it's mini-id :/
on behalf of all wiki editors out there, Lye please re-enable minis and items endpoints. we NEED them, and should be prioritized
.....
plz no ban
Archomeda
@Archomeda
Mar 21 2017 21:01
Rofl
windwarrior
@windwarrior
Mar 21 2017 21:06
well at least it is now clear how useful the API is
Eearslya Sleiarion
@Eearslya
Mar 21 2017 21:07
Don't know how good we have it until it's gone
Pat Cavit
@tivac
Mar 21 2017 21:11
holy shit some of these waterfalls you were linking
I would straight-up burn the site down before it shipped like that :fire:
darthmaim
@darthmaim
Mar 21 2017 21:11
indeed
Eearslya Sleiarion
@Eearslya
Mar 21 2017 21:11
Waterfalls? The slew of requests?
except, you know, for website network requests
blob
darthmaim
@darthmaim
Mar 21 2017 21:26
The entire wiki page is smaller than just the js for tooltips
Michael Dougall
@madou
Mar 21 2017 21:42
Hey boyz - ok lets talk embeds and api requests, api first.
Archomeda
@Archomeda
Mar 21 2017 21:43
:wave:
Michael Dougall
@madou
Mar 21 2017 21:45
api requests - yeah they aren't being proxied through my servers. basic idea is i didnt want to pay for it. so - to get around that when an api request is made to gw2api the data is saved to local storage for 7 days. if the apis are working just load the site and then refresh, you won't see another api call unless you clear local storage.
the api requests do get batched, but only on a per embed. i need more dev time to batch it for all embeds (and probably implement lazy loading, qtfy has a lot of embeds that aren't even seen by the user hidden behind multiple tabs)
per embed meaning i can have one that loads like 10 different skills or items, all of those will be batched up and sent as one request (based on resource, obviously).
I really want to steal @queicherius's gw2api.com domain and just have it throw everything into PostgreSQL JSONB fields with extra indexes
Archomeda
@Archomeda
Mar 21 2017 21:49
hmm, so it's only because of the 404's that we see so many requests currently?
Michael Dougall
@madou
Mar 21 2017 21:49
yep
David Reeß
@queicherius
Mar 21 2017 21:49
if ya want the domain, let me know. it's up for grabs
Archomeda
@Archomeda
Mar 21 2017 21:49
fair
Michael Dougall
@madou
Mar 21 2017 21:49
funnily enough if you're a regular to qtfy you probably aren't affected by the outage (unless your cache was cleared)
Archomeda
@Archomeda
Mar 21 2017 21:49
will it get autocleared after 7 days even if the api isn't live?
Michael Dougall
@madou
Mar 21 2017 21:49
YEP. probably should do something about that...
Archomeda
@Archomeda
Mar 21 2017 21:49
or spitting out errors?
ouch :D
@lye for what purpose? backup?
Michael Dougall
@madou
Mar 21 2017 21:50
lets then talk about bundle size and assets being crazy. yeah - i know. theres a lot of shit being downloaded by the embeds that isn't needed. it takes dev time to make it better, and we all know we're just doing this for fun :) these things, they take time!
the embed system was made fast af, and just reuses everything from the armory
@Archomeda searching items by name :|
Michael Dougall
@madou
Mar 21 2017 21:51
only the thin layer to bootstrap the embeds was fresh
Archomeda
@Archomeda
Mar 21 2017 21:51
oh, haha
Archomeda
@Archomeda
Mar 21 2017 21:52
@madou yeah, everyone is doing it in their spare time, good luck :)
@lye, i think i have some code laying around for searching by name
from a long time ago
but it uses mongodb
and something with full text search i think
ah yeah, found it
    /**
     * Gets the item.
     * @param  {Array}    term     The search term.
     * @param  {Array}    params   The additional parameters from the request.
     * @param  {Function} callback The callback that will be called when the action has been executed.
     */
    getItem: function(term, params, callback) {
        var resultLimit = params.limit !== undefined ? parseInt(params.limit) : 10;
        if (resultLimit <= 0 || resultLimit > 50) {
            callback(new restify.BadRequestError('limit out of range. Use limit values 1 - 50'));
        }
        db.items.find({ '$text': { '$search' : term } }, { match_score: { '$meta': 'textScore' }, _id: 0 }).limit(resultLimit).sort({ match_score: { '$meta': 'textScore' }, name: 1 }, callback);
    }
where db is a mongojs object
urgh, this code is so old
i had it running on a free openshift instance
Archomeda
@Archomeda
Mar 21 2017 21:58
but yeah, mongodb and this text search feature is quite cool
Darqam
@Darqam
Mar 21 2017 22:03
speaking of databases, I will probably be rebuilding a good chunk of my website, and that will include storage method. Any particular database I should look at to store stuff like /v2/items, achievs, colors (no auth stuff)? I'm aware of mysql and mongodb, just wondering if I would be recommended a specific one over others
Archomeda
@Archomeda
Mar 21 2017 22:03
postgresql is good as well; i would probably prefer that over mysql
mongodb just stores things differently
if you want to have strict schemas and tables, try sql databases
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:05
I'm using postgres right now
Archomeda
@Archomeda
Mar 21 2017 22:05
mongodb just has documents and stores everything in json
Michael Dougall
@madou
Mar 21 2017 22:05
@darthmaim see above for some of my comments ;)
darthmaim
@darthmaim
Mar 21 2017 22:05
saw it
Michael Dougall
@madou
Mar 21 2017 22:05
should i get the fire pit out
Archomeda
@Archomeda
Mar 21 2017 22:06
without having strict schemas; so it can be useful for things like /v2/items
darthmaim
@darthmaim
Mar 21 2017 22:06
:fire: :D
should be all fine when they are cached for now
batching all embeds is nice to have for later
Michael Dougall
@madou
Mar 21 2017 22:06
things will be a bit better when i do this too: madou/armory-react#225
Archomeda
@Archomeda
Mar 21 2017 22:06
but otherwise @Darqam i don't know; it's just what you prefer i suppose
Darqam
@Darqam
Mar 21 2017 22:07
hrm.... I'll have a look at those. I haven't looked much into the differences for now. So was mainly looking for a starting point
Archomeda
@Archomeda
Mar 21 2017 22:07
currently i like working with mongodb for my discord bot :)
but it doesn't help that i'm rewriting stuff again
Michael Dougall
@madou
Mar 21 2017 22:08
cool, cya guys im off :shipit:
Archomeda
@Archomeda
Mar 21 2017 22:10
cya
Darqam
@Darqam
Mar 21 2017 22:12
yeah, well my current way of storing stuff for my copy of the api is, imo, the worst way I could have done it, so I'm looking for something that won't be hell to work with. I might look at mongo, and postgres later this week to see exactly how they work
Archomeda
@Archomeda
Mar 21 2017 22:13
with mongodb you just copy the json into it really, and it allows you to search with it :P
Darqam
@Darqam
Mar 21 2017 22:13
ugh wait, I might have no choice depending on the host capabilities
Archomeda
@Archomeda
Mar 21 2017 22:13
though, postgresql allows you to search in json as well nowadays
Darqam
@Darqam
Mar 21 2017 22:13
so does mongodb just make a json file per item then?
so you'd end up with thousands of files?
darthmaim
@darthmaim
Mar 21 2017 22:14
no
Archomeda
@Archomeda
Mar 21 2017 22:14
it's bundled
Darqam
@Darqam
Mar 21 2017 22:14
ok, good
Darqam
@Darqam
Mar 21 2017 22:15
aaaaand mongo is not supported unless I pay a premium. I've got to stop dealing with Hostgator....
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:16
Hostgator? Eesh. I'd recommend a simple VPS that you can install whatever you want on.
Darqam
@Darqam
Mar 21 2017 22:17
yeah, well I inherited that through me working on another website, and it's been a pain in the ass from day1
David Reeß
@queicherius
Mar 21 2017 22:17
mongodb is pretty good with documents with multiple structures (e.g. items, which are different based on what they are)
Darqam
@Darqam
Mar 21 2017 22:17
some day I'll change it... some day
David Reeß
@queicherius
Mar 21 2017 22:19
I personally dont touch any hoster run by EIG
Darqam
@Darqam
Mar 21 2017 22:19
I will keep mongodb in mind though, thanks
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:21
I've actually got a monster of a dedicated server to myself; 48GB RAM running VMs
Oh, wait, 64GB
Darqam
@Darqam
Mar 21 2017 22:22
Once I finally move to my own place I'll build my own machine for it, since the scope of the project is so small in usage it shouldn't be a problem. But hell if I'm going to run stuff of my laptop which is my only computer atm >.>
why use mongodb when pgsql has JSONB columns
darthmaim
@darthmaim
Mar 21 2017 22:28
mysql has json columns too if you need to run it on a webspace that doesn't support and other db
CREATE TABLE items (
  id INTEGER PRIMARY KEY,
  lang CHAR(2),
  blob JSONB
);
CREATE INDEX ON items USING GIN (to_tsvector('english', blob->>'name')) WHERE lang = 'en';
@darthmaim yeah but then you're using mysql
might as well be back on mongo at that point
darthmaim
@darthmaim
Mar 21 2017 22:30
if you need to run it on a webspace that doesn't support and other db
man mysql's json operators are almost the same as pgsql but look slightly different
that's so annoying
pgsql> SELECT blob->>'name';
mysql> SELECT JSON_UNQUOTE(blob->>'$.name');
darthmaim
@darthmaim
Mar 21 2017 22:33
:/
Archomeda
@Archomeda
Mar 21 2017 22:33
they have to be unique
:P
the unique snowflake
Does MySQL support computed indexes?
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:35
@lye why must you make me question my database structure
    create_table :items, { id: false } do |t|
      t.primary_key :id, :null => false
      t.string :name, :null => false
      t.string :icon, :null => false
      t.string :description
      t.string :item_type, :null => false
      t.string :rarity, :null => false
      t.integer :level, :default => 0
      t.integer :vendor_value, :default => 0
      t.timestamps
    end
I mean sure you can do that
but that's a lot of work
I have a schema somewhere that fully normalizes /v2/items (or at least, the output from 2 years ago)
darthmaim
@darthmaim
Mar 21 2017 22:36
no computed indexes in mysql :(
it was 26 tables? or something?
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:36
I was gonna do 1 main table and 1 for each type
@darthmaim apparently a new version has "generated columns" which you can index over
darthmaim
@darthmaim
Mar 21 2017 22:36
but you can create a view with the computed column and index that
5.7.x i think
does the computed column have to be materialized? or can be it virtual
darthmaim
@darthmaim
Mar 21 2017 22:36
I have no idea
Archomeda
@Archomeda
Mar 21 2017 22:37
@lye, get gw2api.com, make it your hobby-project to extend the API, and open source it on github :smile:
but i already have a hobby project
Archomeda
@Archomeda
Mar 21 2017 22:37
and pay the jackpot for hosting when it gets too popular
darthmaim
@darthmaim
Mar 21 2017 22:37
just one?
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:37
I'm gonna make a gw2oracle API too
it's a big one
Archomeda
@Archomeda
Mar 21 2017 22:38
related?
to gw2? no
i'll probably never get it to a workable state either
ChieftainAlex
@ChieftainAlex
Mar 21 2017 22:39
@Darqam german translation is best translation: Mini Krytanischer schlapper Fisch
darthmaim
@darthmaim
Mar 21 2017 22:40
ughh those german translations...
Archomeda
@Archomeda
Mar 21 2017 22:40
:|
man the forums
Archomeda
@Archomeda
Mar 21 2017 22:41
@Eearslya every time you mention gw2oracle, i keep thinking about something that can handle NLP and give me a nice result for every request i do
darthmaim
@darthmaim
Mar 21 2017 22:41
just don't look at them until the api is back up
right
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:41
NLP?
natural language processing
Archomeda
@Archomeda
Mar 21 2017 22:41
^
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:41
I'm not good at naming projects ;;
I don't really like the name either tbh
rename it to bookah.com
darthmaim
@darthmaim
Mar 21 2017 22:42
1380:05:17:32
uhm what?
uhhhh not my fault
Archomeda
@Archomeda
Mar 21 2017 22:42
like an oracle, "if i want to buy 42 stacks of item x, will i have enough gold leftover to buy 10 stacks of item y" for example :P
@Archomeda you should submit a gw2 integration to wolfram alpha
Archomeda
@Archomeda
Mar 21 2017 22:43
or "will i have enough gold to buy a legendary at the end of this month"
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:43
asuramasterrace.net
Archomeda
@Archomeda
Mar 21 2017 22:43
although that will be tricky
@lye lol
i don't even want to look at how NLP works
i know the concept, but man...
I think google opensourced a toolkit awhile back
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:44
I had the idea for gw2watchdog.net at one time
Archomeda
@Archomeda
Mar 21 2017 22:45
i mean, even google has an answer for this query: "who was the american president in 1988"
ehh
and it even autocorrected my spelling error lol
"present" lmao
risky click of the day
Darqam
@Darqam
Mar 21 2017 22:50
"Because as far as I can find on any site (wiki,forums,all wvw sites) they all say [victory points] not exposed." grumble grumble don't say wiki doesn't show it if you haven't looked
I don't update that for nothing
darthmaim
@darthmaim
Mar 21 2017 22:50
:P
Archomeda
@Archomeda
Mar 21 2017 22:51
hahaa
one day i wanted to update something in the API namespace
but i couldn't because i didn't have permissions :'(
Darqam
@Darqam
Mar 21 2017 22:52
ask for perms, vent hands it out like candy
"seeing that I'm not an active contributor here" it was literally my first contribution
creating a new talk page
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:54
I had fun updating the API namespace until job
Archomeda
@Archomeda
Mar 21 2017 22:54
bleh
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:54
now @Darqam is doing a good job about it
Darqam
@Darqam
Mar 21 2017 22:54
<3
that main API page though, needs to be manually maintained for each new endpoint. It is ugly and I hate it
/v2/wvw/matches.documentation
Darqam
@Darqam
Mar 21 2017 22:57
alomst tried to go there >.>
but then remembered :fire:
:goat:
Archomeda
@Archomeda
Mar 21 2017 22:58
outputs: This endpoint will be unavailable until further notice, while the rest of the API is online.
Archomeda @Archomeda is not sorry about referring to that forum thread
Eearslya Sleiarion
@Eearslya
Mar 21 2017 22:59
Blame the forum.
Darqam
@Darqam
Mar 21 2017 22:59
yes please
just a heads up -- in like half an hour I'm going to try turning the API on with a super low rate limit
Archomeda
@Archomeda
Mar 21 2017 23:00
woooo
good luck!
darthmaim
@darthmaim
Mar 21 2017 23:00
:+1:
I want to flood the system with errors basically
Just to make sure everything works
Darqam
@Darqam
Mar 21 2017 23:00
lol
good luck
Archomeda
@Archomeda
Mar 21 2017 23:01
my guild discord will probably get spammed with new guild logs... heh
darthmaim
@darthmaim
Mar 21 2017 23:01
I hope you have a graph setup with all requests that get rate limited
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:01
Super low?
like 60/minute
Archomeda
@Archomeda
Mar 21 2017 23:01
oh, @queicherius which reminds me, i haven't seen support in your library for the ?since param on /v2/guilds/:id/logs
that might be too high
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:02
time for me to quickly grab some example items so I can write tests
Archomeda
@Archomeda
Mar 21 2017 23:02
does it exist? or do i have to make a PR at some point?
Darqam
@Darqam
Mar 21 2017 23:02
Lye, 1/min, do it :D
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:02
1 lucky person on gw2e gets to see something per minute
Archomeda
@Archomeda
Mar 21 2017 23:02
what error will the API spit out when rate limited?
HTTP 429
Darqam
@Darqam
Mar 21 2017 23:03
api would be on fire if I let you in
{ "error" : "too many requests" } I think
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:03
rate limit exceeded
Archomeda
@Archomeda
Mar 21 2017 23:03
cool
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:03
I thought that's what you said before
i don't remember
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:03
we're about to find out
{"text":"too many requests"}
Darqam
@Darqam
Mar 21 2017 23:03
I like the json error format regardless
the 429 is what you should be looking for TBH
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:03
true
David Reeß
@queicherius
Mar 21 2017 23:04
@Archomeda Feel free to make an issue / PR, probably missed that :)
darthmaim
@darthmaim
Mar 21 2017 23:04
make a new issue in my php wrapper too :P
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:05
I was gonna make a ruby wrapper but I gave up
Archomeda
@Archomeda
Mar 21 2017 23:07
ehh, i'm not touching PHP :P
i'll create an issue for you then xD
<?php echo "is fun!"
Darqam
@Darqam
Mar 21 2017 23:10
I like php, it feels like I'm permanently stumbling in the dark and yet getting the results I want >.>
Archomeda
@Archomeda
Mar 21 2017 23:10
@darthmaim GW2Treasures/gw2api right?
darthmaim
@darthmaim
Mar 21 2017 23:10
:+1:
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:10
Maybe I should just name my site GW2Stuff
Darqam
@Darqam
Mar 21 2017 23:12
a few things called gw2stuff already :p
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:12
dangit
darthmaim
@darthmaim
Mar 21 2017 23:13
GET YOUR GUILD WARS 2 STUFF EASY
NO MORE LONG FARMING! ALWAYS CHEAPEST PRICE
welp, that name is probably banned in a few chats
Eearslya Sleiarion
@Eearslya
Mar 21 2017 23:14
I didn't even google it XD
Looks like I'm still stuck with this name for now
ChieftainAlex
@ChieftainAlex
Mar 21 2017 23:15
wiki won't be hitting you with TP requests when you boot up.
Archomeda
@Archomeda
Mar 21 2017 23:17
actually @queicherius, does the library support custom parameters at all?
there's also the world (or something) parameter for /v2/wvw/matches (i think?)
David Reeß
@queicherius
Mar 21 2017 23:17
Yeah, like here.
Archomeda
@Archomeda
Mar 21 2017 23:18
ah, thanks
i knew i saw it somewhere
David Reeß
@queicherius
Mar 21 2017 23:18
I probably missed nearly all the custom ones
Archomeda
@Archomeda
Mar 21 2017 23:19
honestly, the guild log, wvw match and the commerce ones are the only ones that i know of
darthmaim
@darthmaim
Mar 21 2017 23:20
recipe and guild search endpoints
Archomeda
@Archomeda
Mar 21 2017 23:21
ah, guild search by name as well yeah; i see it in the source
Darqam
@Darqam
Mar 21 2017 23:22
https://status.discordapp.com/incidents/dj3l6lw926kl if you guys want to read about that discord crash yesterday
tl;dr distributed systems are hard :crying_cat_face:
wow this sounds familiar
Once the event queues on these session nodes reached a limit, they ran out of available memory and crashed.
Michael Dougall
@madou
Mar 21 2017 23:29
Good luck @lye !!
Darqam
@Darqam
Mar 21 2017 23:29
confirmed, gw2 api runs on discord
These patches introduce limitations on connections and message sending
Archomeda
@Archomeda
Mar 21 2017 23:31
@queicherius running npm test still doesn't work :(
Error: Cannot find module '******\gw2api-client\tests\cache'
on windows
David Reeß
@queicherius
Mar 21 2017 23:32
That's weird as hell
Also @Darqam thanks, good read
Archomeda
@Archomeda
Mar 21 2017 23:32
let me try bash in windows
stack trace is quite long btw
npm installs dev dependencies by default right?
i thought you had to npm install --dev
darthmaim
@darthmaim
Mar 21 2017 23:36
nope, it installs always installs dev dependencies of the root package
Archomeda
@Archomeda
Mar 21 2017 23:37
well, it does run in bash on windows
but 93 failed tests... wtf
darthmaim
@darthmaim
Mar 21 2017 23:38
I had 41 or 49 failed tests in @queicherius gw2api-client
no idea why it wasn't always the same number
Archomeda
@Archomeda
Mar 21 2017 23:38
well, my added test passes, so there's that
David Reeß
@queicherius
Mar 21 2017 23:38
You need redis running for all tests to pass, since the cache depends on it
darthmaim
@darthmaim
Mar 21 2017 23:38
run the linter too
Archomeda
@Archomeda
Mar 21 2017 23:38
oh
darthmaim
@darthmaim
Mar 21 2017 23:38
because the linter doesn't run when you have failing tests
bumping my "30 minutes" back another hour
perforce died
darthmaim
@darthmaim
Mar 21 2017 23:40
rip
Archomeda
@Archomeda
Mar 21 2017 23:42
:(
does anyone know if the sub endpoints of /v2/wvw/matches also support ?world?
Darqam
@Darqam
Mar 21 2017 23:43
I'd hope Lye knows
@Archomeda they should yeah.
darthmaim
@darthmaim
Mar 21 2017 23:43
they do
Archomeda
@Archomeda
Mar 21 2017 23:44
oh wait, i see examples at the bottom of the wiki page
thanks <3