fanout/pushpin

Reverse proxy for realtime web services

ccbeloy

@ccbeloy

@jkarneges - we did a bit of benchmarking on our setup for 800 concurrent users and 20 messages per second. We are getting below warn message:

[WARN] 2020-10-21 18:20:10.369 [handler] exceeded publish hwm (25000), dropping message

ccbeloy

@ccbeloy

Is there any config we need to be aware to avoid losing events?

Justin Karneges

@jkarneges

see message_rate and message_hwm in pushpin.conf

default rate is 2500. if your benchmark is broadcasting, then probably you're exceeding that rate, causing messages to get queued until there's no room in the queue

to avoid losing events, try not to send faster than message_rate

MauDagos

@MauDagos

Hi everyone :wave: I'm seeing that, when I connect to my pushpin server, if I lose the connection to the server (i.e. a network cut), Pushpin takes around 15 minutes to see that the connection has gone. At which it logs in the error_XXXX.log file:

Mon, 26 Oct 2020 14:33:28 GMT [ERROR] (src/io.c:777: errno: Transport endpoint is not connected) Write error when sending all.
Mon, 26 Oct 2020 14:33:28 GMT [ERROR] (src/connection.c:1230: errno: None) Error delivering to MSG listener on FD -1, closing them.

Is there some parameter for configuring the time Pushpin holds on to this connection?

Justin Karneges

@jkarneges

hi @MauDagos , Pushpin times out the connection based on the default TCP settings, which is usually in the ballpark of 15-20 mins or so. you could tune your kernel tcp or keepalive settings to reduce this

if you're trying to do presence, I'd recommend doing that with heartbeats instead of messing with tcp settings though

MauDagos

@MauDagos

Thanks @jkarneges ! I'll investigate what you've written.

MauDagos

@MauDagos

@jkarneges could you elaborate more on the heartbeats recommendation? Our connected clients just receive data, they don't send anything back.

Justin Karneges

@jkarneges

@MauDagos for example, the clients could send something to the server on an interval. this would be a better way to track which users are active/present, rather than trying to determine that from the existence of TCP connections

MauDagos

@MauDagos

We don't find this solution practical for our use case, so we'll investigate changing the TCP settings. Thanks for the all replies @jkarneges !

Justin Karneges

@jkarneges

Pushpin 1.31.0 has been released, with Condure (new connection manager) enabled by default.

NIRANJAN

@niranjan-kmit

Hi @jkarneges Some reason I am unable to open web socket connection. this is the logs I found in the my cluser. [ERR] 2020-11-16 06:20:59.632 [m2 http:7999] (src/register.c:157: errno: Resource temporarily unavailable) Invalid FD given for ping: -1

Justin Karneges

@jkarneges

@niranjan-kmit is there any other log message, such as from pushpin-proxy?

NIRANJAN

@niranjan-kmit

Hi @jkarneges here is the log messages from [INFO] 2020-11-18 12:29:08.958 [m2a] OUT pushpin-m2-7999 id=89678 code=200 61
[ERR] 2020-11-18 12:29:09.624 [m2 http:7999] (src/register.c:157: errno: Resource temporarily unavailable) Invalid FD given for ping: -1
[INFO] 2020-11-18 12:29:10.225 [m2 http:7999] 95:7:default,13:10.155.143.53,5:62308#10:1605702550#3:GET,15:/v1/healthcheck,8:HTTP/1.1,3:200#1:0#]
[INFO] 2020-11-18 12:29:10.225 [m2a] IN pushpin-m2-7999 id=89679 GET http://qa-us-v2.realtime.platts.com/v1/healthcheck

Justin Karneges

@jkarneges

that looks like a healthcheck rather than a websocket request. unless you're using websockets to do a health check?

NIRANJAN

@niranjan-kmit

Hi @jkarneges we are using pushpin proxy as gateway and java backend service using for establishing web socket connection. Custom healthcheck endpoint implement in java backend when accessed through pushpin is also working fine, but some reason connection not established.we are getting the "Resource not temporarly not avaible for FD given for ping "-1"

Justin Karneges

@jkarneges

@niranjan-kmit do you see a log line from "[m2a]" showing the WebSocket URL requested?

also I assume you are not using SSL?

NIRANJAN

@niranjan-kmit

@jkarneges while making requesting we are using wss protocol but internally it converting the ws request to http using overhttp .

Justin Karneges

@jkarneges

@niranjan-kmit what part of your stack is handling SSL termination though? is pushpin handling it, or something in front of pushpin?

ccbeloy

@ccbeloy

Hi @jkarneges I work with @niranjan-kmit. To answer your question, pushpin is fronted by AWS ALB which is doing the SSL termination and forwarding the request to pushpin on http or ws. What could be the possible issue here? This was previously working and just started getting this issue last week.

Justin Karneges

@jkarneges

Hi @ccbeloy, if there is no m2a log line associated with the websocket connection attempt, then the connection must be failing very early. maybe a misconfiguration was introduced with ALB or something external to pushpin?

you could try connecting a websocket directly to pushpin to see if that still works

Erick

@ericklind

I've got a basic node express server working with pushpin using websockets over http. I'm using serve-grip & grip and I really like how things are progressing. The question that I have is how do we setup ping/pong heartbeats to check for disconnected clients? I have scoured the docs and while I found reference to being able to send ping/pong, I have found nothing showing how this might be accomplished. It has also been mentioned in this room, but again with no explanation as to how. Can you point me to any examples? Thanks.

Justin Karneges

@jkarneges

hi @ericklind, are you saying you want to send pings from the server to the client?

Erick

@ericklind

Yes.

Justin Karneges

@jkarneges

@ericklind https://pushpin.org/docs/advanced/#keep-alives

Justin Karneges

@jkarneges

for example, to configure pushpin to send a PING frame whenever there is 30 seconds of inactivity, you could use this control message: c:{"type": "keep-alive", "message-type": "ping", "content": "", "timeout": 30}

to send a control message using WebSocketContext from serve-grip, do it like ws.sendControl({"type": "keep-alive", "message-type": "ping", "content": "", "timeout": 30})

Erick

@ericklind

Thanks. Can I recommend adding that to the JS docs? Small things like that are useful. I read up on the keep-alives last night, but then reading further it was a bit confusing and seemed like that was not the way to go. As someone new in this area, there's terminology and concepts that are not self explanatory, so examples are pretty key.

Justin Karneges

@jkarneges

@ericklind I agree on that :) thanks for the feedback

Erick

@ericklind

@jkarneges So when that kicks off a ping, are we supposed to handle the pong, or does pushpin handle it and automatically disconnect the websocket if it's not received?

Justin Karneges

@jkarneges

@ericklind pushpin just forwards the pong to the backend. what is your goal? often it's fine for the backend to not care about whether connections exist or not, in which case dead connections can be cleaned up by the OS when tcp times out

Erick

@ericklind

The client socket is returning analytics (ip, location, etc) on the init. After that it’s just sending back info like if the page is currently active or not. That info is like a ping/pong but I’m not sure what I want the interval to be so I was also checking out the idea of a heartbeat. It may be redundant in this case, but i want to understamd how it all works.

So in this case, do I just close() or disconnect(), or both?

Justin Karneges

@jkarneges

@ericklind unless you have a specific need to track the presence of connections, I wouldn't worry about processing pongs or explicitly trying to close dead connections or anything

on the server side I'd simply ack closes (if ws.recv() returns null, call ws.close()) and ignore disconnects

Justin Karneges

@jkarneges

heartbeats from server->client can be useful even if the server isn't tracking connections though. for example, the server could send a message every 30 seconds, and if the client receives nothing in the last 60 seconds then the client can assume its connection is dead. in this case, it may be better for the server to use a TEXT frame for the heartbeat (instead of PING) since a reply would not be required

Erick

@ericklind

What is the difference, and how would that be done? Would I hold onto each wsContext and call wsContext.send(?)? Or is there a collection on the req.grip that has all the connections? (Docs are kind of lacking on this area)

Justin Karneges

@jkarneges

@ericklind definitely don't hold on to WebSocketContext. that's a temporary object which only lives as long as the current http request being handled by the backend. you can use a keep-alive control message to tell pushpin to send TEXT frames for you

ws.sendControl({"type": "keep-alive", "content": "still alive", "timeout": 30})

ccbeloy

@ccbeloy

@jkarneges - just to update from our side with regards to 502 ALB issue. It just start working, we did not do any change. So I guess its a misconfiguration issue with ALB. Thanks.

ccbeloy

@ccbeloy

Hi @jkarneges is there a limit on how many channels a single pushpin instance could handle? Are channels expensive in Pushpin?
Also found this in the pushpin config "connection_subscription_max=20". Is this the max number of channels per client connection? Can this be increased in a much larger number? And will be the drawback?
Thanks.

Justin Karneges

@jkarneges

@ccbeloy channels are not too expensive, but they have an expense. tens of thousands of channels in one instance should be fine. feel free to increase connection_subscription_max, just be mindful of the potential amplification that could cause

ccbeloy

@ccbeloy

Thanks @jkarneges

Jimmy Marquez

@jimmysp_gitlab

Hi @jkarneges
I am trying to send messages to pushping using 0mq but I am unable to do it, I am using the following code:

ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket publisher = context.socket(ZMQ.PUB);
publisher.connect("tcp://" + ip + ":5562");
publisher.send("J{\"id\": \"an-item-id\", \"channel\": \"test\", \"formats\": {\"http-stream\": {\"content\": \"a chunk of data\n\"}}}",0);
publisher.close();
context.term();

I am also using the following versions of 0MQ,

zeromq-4.1.4
zeromq/jzmq

The code does not display an error but messages are not appearing in pushpin logs.

Any suggestion related to this? might I be missing something?

Justin Karneges

@jkarneges

hi @jimmysp_gitlab , you need to send a multipart message, where the first part is the channel, and the second part is the payload

some other things to consider. there must be at least one subscriber, otherwise your publisher won't actually send anything to pushpin (and nothing will get logged). and it can take a moment for subscription information to be sent to the publisher. I suggest adding a 500ms sleep between the connect() and send()

Where communities thrive

People

Repo info

Activity