These are chat archives for fanout/pushpin

20th
Apr 2017
Christian Engel
@Paratron
Apr 20 2017 06:30
@dotort window.onbeforeunload = function(){ websocket.close(); } did the trick for me. I know one shouldnt rely on onbeforeunload but it looks like its working cross browser and its better than nothing :)
Francis Eytan Dortort
@dortort
Apr 20 2017 13:09
@jkarneges I get CLOSE when issuing a close command via wscat2. Just no DISCONNECT event when closing the connection (^C)
[DEBUG] 2017-04-20 12:55:00.588 m2: IN pushpin-m2-7999 2 / 546:{"PATH":"/","sec-websocket-extensions":"permessage-deflate; client_max_window_bits","FLAGS":"0x88","connection":"Upgrade","sec-websocket-version":"13","DOWNLOAD_CREDITS":"102400","sec-websocket-key":"MTMtMTQ5MjY5Mjg4MDM5MA==","host":"localhost:7999","upgrade":"websocket","authorization":"Basic MDRmNGI3M2UtZmU5Ny00YjMxLWJhYmItMWEyZjBlYWY1MmYxOmMzYWM1MmNiZGRkNzk1NDAxY2JiMmY2YjdlN2U2MjgzYzE3Y2ZmOGQxNmRhMDM2OWJkOWQwMmVjNTAxZDk0YmE=","METHOD":"WEBSOCKET","VERSION":"HTTP/1.1","URI":"/","PATTERN":"/","URL_SCHEME":"http","REMOTE_ADDR":"172.17.0.1"},2:?,
[DEBUG] 2017-04-20 12:55:00.588 zws: OUT instance=pushpin-proxy_11 T81:2:id,19:pushpin-m2-7999_2_0,4:type,5:close,4:from,7:m2zws_9,4:code,1:0#3:seq,1:2#}
[DEBUG] 2017-04-20 12:55:00.590 m2: IN pushpin-m2-7999 2 @* 17:{"METHOD":"JSON"},21:{"type":"disconnect"},
[DEBUG] 2017-04-20 12:55:00.590 m2: pushpin-m2-7999 id=2 disconnected
[DEBUG] 2017-04-20 12:55:00.590 zws: OUT instance=pushpin-proxy_11 T71:2:id,19:pushpin-m2-7999_2_0,4:type,6:cancel,4:from,7:m2zws_9,3:seq,1:3#}
Justin Karneges
@jkarneges
Apr 20 2017 18:29
@dortort hmm, for me, ^C still sends CLOSE. but if I kill the wscat process, then I get DISCONNECTED. what's the log output when you ^C?
Francis Eytan Dortort
@dortort
Apr 20 2017 19:15
the 5 lines above are what's written to m2adapter.log after hitting ^C
Justin Karneges
@jkarneges
Apr 20 2017 19:27
ah, yes. wscat sends a close message when ^C is hit, but then it disconnects and exits before waiting for the response. I remember this now and we did have a bug handling this. it's fixed in the develop branch but we haven't made a new release yet
so this should be fixed in 1.16
Francis Eytan Dortort
@dortort
Apr 20 2017 19:40

Great!
Regarding the issue of the service going down, I have the following at the end of a log

[INFO] 2017-04-19 07:57:06.911 [m2a] IN pushpin-m2-7999 id=68255 GET http://172.31.30.41/
[INFO] 2017-04-19 07:57:06.913 [zurl] IN id=ade8a529-6259-4c5e-bb99-dcb5068a7bc8, GET https://api.---.com/
[INFO] 2017-04-19 07:57:06.923 [zurl] OUT ERR id=ade8a529-6259-4c5e-bb99-dcb5068a7bc8 condition=undefined-condition
[INFO] 2017-04-19 07:57:06.923 [proxy] GET http://172.31.30.41/ -> api.---.com:443 code=502 32
[INFO] 2017-04-19 07:57:06.923 [m2a] OUT pushpin-m2-7999 id=68255 code=502 32
[INFO] 2017-04-19 07:57:14.028 [m2a] IN pushpin-m2-7999 id=68256 GET http://172.31.30.41/
[INFO] 2017-04-19 07:57:14.030 [zurl] IN id=6166e855-13f3-4b69-ad4d-68b526228ed1, GET https://api.---.com/
[WARN] 2017-04-19 07:57:14.043 [zurl] SSL: no alternative certificate subject name matches target host name 'api.---.com'
[INFO] 2017-04-19 07:57:14.054 [zurl] OUT ERR id=6166e855-13f3-4b69-ad4d-68b526228ed1 condition=tls-error
[INFO] 2017-04-19 07:57:14.055 [m2a] OUT pushpin-m2-7999 id=68256 code=502 32
[INFO] 2017-04-19 07:57:14.055 [proxy] GET http://172.31.30.41/ -> api.---.com:443 code=502 32
[INFO] 2017-04-19 07:57:21.927 [m2a] IN pushpin-m2-7999 id=68257 GET http://172.31.30.41/
[INFO] 2017-04-19 07:57:21.943 [zurl] IN id=a1577620-75d8-4ab4-9f3a-b55a29b1e746, GET https://api.---.com/
[WARN] 2017-04-19 07:57:21.957 [zurl] SSL: no alternative certificate subject name matches target host name 'api.---.com'
[INFO] 2017-04-19 07:57:21.968 [zurl] OUT ERR id=a1577620-75d8-4ab4-9f3a-b55a29b1e746 condition=tls-error
[INFO] 2017-04-19 07:57:21.969 [m2a] OUT pushpin-m2-7999 id=68257 code=502 32
[INFO] 2017-04-19 07:57:21.969 [proxy] GET http://172.31.30.41/ -> api.---.com:443 code=502 32

Does this mean anything to you?

Justin Karneges
@jkarneges
Apr 20 2017 20:01
looks like zurl's giving tls-error and undefined-condition
tls error looks straightforward. cert doesn't match host. are you saying that the cert worked for awhile, and then it stops working without having changed it?
Francis Eytan Dortort
@dortort
Apr 20 2017 20:24
that's correct. The cert worked for a while, nothing ever changed regarding the cert. It's a cert issued using AWS Certificate Manager, attached to an ELB.
Justin Karneges
@jkarneges
Apr 20 2017 20:24
and if you restart pushpin it starts working again?
Francis Eytan Dortort
@dortort
Apr 20 2017 20:29
haven't tried yet. I just terminated the instance & let a new one take it's place (using auto-scaling).
Justin Karneges
@jkarneges
Apr 20 2017 20:31
ah okay, same difference. what version of zurl?
and libcurl
Francis Eytan Dortort
@dortort
Apr 20 2017 20:34
whatever's included in the docker image fanout/pushpin:latest. Is the info in one of the log files?
Justin Karneges
@jkarneges
Apr 20 2017 20:36
won't be in a log file, but you could run "curl --version" on the instance
Francis Eytan Dortort
@dortort
Apr 20 2017 20:53
libcurl3/now 7.47.0-1ubuntu2.2 amd64 [installed,local]
libcurl3-gnutls/now 7.47.0-1ubuntu2.1 amd64 [installed,local]
python3-pycurl/now 7.43.0-1ubuntu1 amd64 [installed,local]
Justin Karneges
@jkarneges
Apr 20 2017 21:00
how long does it take to start failing?
Francis Eytan Dortort
@dortort
Apr 20 2017 21:04
5, 1 & 3 days, the last 3 times it happened
no traffic, still just developing
Justin Karneges
@jkarneges
Apr 20 2017 21:14
try running with --verbose until it happens again. the zurl log may tell us more
is there a chance the IP address of the backend changed?
well hmm the IP address is in the log. I was thinking of a case where there's a persistent connection open do a domain, and the domain changes where it pointed. but that's probably not the case here
Francis Eytan Dortort
@dortort
Apr 20 2017 21:20
it's possible, AWS ELBs don't have static ips
I'll add the verbose flag