Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 07 21:07

    jkarneges on develop

    ensure keep-alives are sent in … (compare)

  • Nov 06 2020 21:45

    jkarneges on master

    handler: make http publish maxi… Merge pull request #47691 from … Merge remote-tracking branch 'o… and 7 more (compare)

  • Nov 06 2020 21:44

    jkarneges on v1.31.0

    (compare)

  • Nov 06 2020 21:44

    jkarneges on develop

    update version (compare)

  • Nov 02 2020 23:59

    jkarneges on develop

    use condure by default (compare)

  • Nov 02 2020 22:32

    jkarneges on develop

    runner: support for ssl ports w… (compare)

  • Oct 09 2020 04:53

    jkarneges on develop

    ensure queueable events are onl… (compare)

  • Oct 02 2020 05:12

    jkarneges on develop

    avoid adding extra minutes to r… (compare)

  • Sep 24 2020 23:12

    jkarneges on develop

    handler: split out wssession make it possible to refresh web… (compare)

  • Sep 10 2020 21:47

    jkarneges on develop

    readme nits, remove mailing list Merge remote-tracking branch 'o… (compare)

  • Sep 10 2020 21:46

    jkarneges on master

    readme nits, remove mailing list (compare)

  • Aug 13 2020 18:17

    jkarneges on develop

    handler: make http publish maxi… Merge pull request #47691 from … (compare)

  • Jul 29 2020 18:41

    jkarneges on v1.30.0

    (compare)

  • Jul 29 2020 18:40

    jkarneges on master

    limit body bytes to internal bu… runner: support condure service use client_maxconn and condure_… and 1 more (compare)

  • Jul 29 2020 18:40

    jkarneges on develop

    update version (compare)

  • Jul 28 2020 02:16

    jkarneges on develop

    use client_maxconn and condure_… (compare)

  • Jul 28 2020 02:12

    jkarneges on develop

    runner: support condure service (compare)

  • Jul 22 2020 01:42

    jkarneges on develop

    limit body bytes to internal bu… (compare)

  • Jul 15 2020 18:45

    jkarneges on v1.29.0

    (compare)

  • Jul 15 2020 18:45

    jkarneges on master

    fix crash when parsing accept h… provide credits on websocket ac… fix crash when response hold ti… and 10 more (compare)

ccbeloy
@ccbeloy

@jkarneges - we did a bit of benchmarking on our setup for 800 concurrent users and 20 messages per second. We are getting below warn message:

[WARN] 2020-10-21 18:20:10.369 [handler] exceeded publish hwm (25000), dropping message

ccbeloy
@ccbeloy
Is there any config we need to be aware to avoid losing events?
Justin Karneges
@jkarneges
see message_rate and message_hwm in pushpin.conf
default rate is 2500. if your benchmark is broadcasting, then probably you're exceeding that rate, causing messages to get queued until there's no room in the queue
to avoid losing events, try not to send faster than message_rate
MauDagos
@MauDagos

Hi everyone :wave: I'm seeing that, when I connect to my pushpin server, if I lose the connection to the server (i.e. a network cut), Pushpin takes around 15 minutes to see that the connection has gone. At which it logs in the error_XXXX.log file:

Mon, 26 Oct 2020 14:33:28 GMT [ERROR] (src/io.c:777: errno: Transport endpoint is not connected) Write error when sending all.
Mon, 26 Oct 2020 14:33:28 GMT [ERROR] (src/connection.c:1230: errno: None) Error delivering to MSG listener on FD -1, closing them.

Is there some parameter for configuring the time Pushpin holds on to this connection?

Justin Karneges
@jkarneges
hi @MauDagos , Pushpin times out the connection based on the default TCP settings, which is usually in the ballpark of 15-20 mins or so. you could tune your kernel tcp or keepalive settings to reduce this
if you're trying to do presence, I'd recommend doing that with heartbeats instead of messing with tcp settings though
MauDagos
@MauDagos
Thanks @jkarneges ! I'll investigate what you've written.
MauDagos
@MauDagos
@jkarneges could you elaborate more on the heartbeats recommendation? Our connected clients just receive data, they don't send anything back.
Justin Karneges
@jkarneges
@MauDagos for example, the clients could send something to the server on an interval. this would be a better way to track which users are active/present, rather than trying to determine that from the existence of TCP connections
MauDagos
@MauDagos
We don't find this solution practical for our use case, so we'll investigate changing the TCP settings. Thanks for the all replies @jkarneges !
Justin Karneges
@jkarneges
Pushpin 1.31.0 has been released, with Condure (new connection manager) enabled by default.
NIRANJAN
@niranjan-kmit
Hi @jkarneges Some reason I am unable to open web socket connection. this is the logs I found in the my cluser. [ERR] 2020-11-16 06:20:59.632 [m2 http:7999] (src/register.c:157: errno: Resource temporarily unavailable) Invalid FD given for ping: -1
Justin Karneges
@jkarneges
@niranjan-kmit is there any other log message, such as from pushpin-proxy?
NIRANJAN
@niranjan-kmit
Hi @jkarneges here is the log messages from [INFO] 2020-11-18 12:29:08.958 [m2a] OUT pushpin-m2-7999 id=89678 code=200 61
[ERR] 2020-11-18 12:29:09.624 [m2 http:7999] (src/register.c:157: errno: Resource temporarily unavailable) Invalid FD given for ping: -1
[INFO] 2020-11-18 12:29:10.225 [m2 http:7999] 95:7:default,13:10.155.143.53,5:62308#10:1605702550#3:GET,15:/v1/healthcheck,8:HTTP/1.1,3:200#1:0#]
[INFO] 2020-11-18 12:29:10.225 [m2a] IN pushpin-m2-7999 id=89679 GET http://qa-us-v2.realtime.platts.com/v1/healthcheck
Justin Karneges
@jkarneges
that looks like a healthcheck rather than a websocket request. unless you're using websockets to do a health check?
NIRANJAN
@niranjan-kmit
Hi @jkarneges we are using pushpin proxy as gateway and java backend service using for establishing web socket connection. Custom healthcheck endpoint implement in java backend when accessed through pushpin is also working fine, but some reason connection not established.we are getting the "Resource not temporarly not avaible for FD given for ping "-1"
Justin Karneges
@jkarneges
@niranjan-kmit do you see a log line from "[m2a]" showing the WebSocket URL requested?
also I assume you are not using SSL?
NIRANJAN
@niranjan-kmit
@jkarneges while making requesting we are using wss protocol but internally it converting the ws request to http using overhttp .
Justin Karneges
@jkarneges
@niranjan-kmit what part of your stack is handling SSL termination though? is pushpin handling it, or something in front of pushpin?
ccbeloy
@ccbeloy
Hi @jkarneges I work with @niranjan-kmit. To answer your question, pushpin is fronted by AWS ALB which is doing the SSL termination and forwarding the request to pushpin on http or ws. What could be the possible issue here? This was previously working and just started getting this issue last week.
Justin Karneges
@jkarneges
Hi @ccbeloy, if there is no m2a log line associated with the websocket connection attempt, then the connection must be failing very early. maybe a misconfiguration was introduced with ALB or something external to pushpin?
you could try connecting a websocket directly to pushpin to see if that still works
Erick
@ericklind
I've got a basic node express server working with pushpin using websockets over http. I'm using serve-grip & grip and I really like how things are progressing. The question that I have is how do we setup ping/pong heartbeats to check for disconnected clients? I have scoured the docs and while I found reference to being able to send ping/pong, I have found nothing showing how this might be accomplished. It has also been mentioned in this room, but again with no explanation as to how. Can you point me to any examples? Thanks.
Justin Karneges
@jkarneges
hi @ericklind, are you saying you want to send pings from the server to the client?
Erick
@ericklind
Yes.
Justin Karneges
@jkarneges
Justin Karneges
@jkarneges
for example, to configure pushpin to send a PING frame whenever there is 30 seconds of inactivity, you could use this control message: c:{"type": "keep-alive", "message-type": "ping", "content": "", "timeout": 30}
to send a control message using WebSocketContext from serve-grip, do it like ws.sendControl({"type": "keep-alive", "message-type": "ping", "content": "", "timeout": 30})
Erick
@ericklind
Thanks. Can I recommend adding that to the JS docs? Small things like that are useful. I read up on the keep-alives last night, but then reading further it was a bit confusing and seemed like that was not the way to go. As someone new in this area, there's terminology and concepts that are not self explanatory, so examples are pretty key.
Justin Karneges
@jkarneges
@ericklind I agree on that :) thanks for the feedback
Erick
@ericklind
@jkarneges So when that kicks off a ping, are we supposed to handle the pong, or does pushpin handle it and automatically disconnect the websocket if it's not received?
Justin Karneges
@jkarneges
@ericklind pushpin just forwards the pong to the backend. what is your goal? often it's fine for the backend to not care about whether connections exist or not, in which case dead connections can be cleaned up by the OS when tcp times out
Erick
@ericklind
The client socket is returning analytics (ip, location, etc) on the init. After that it’s just sending back info like if the page is currently active or not. That info is like a ping/pong but I’m not sure what I want the interval to be so I was also checking out the idea of a heartbeat. It may be redundant in this case, but i want to understamd how it all works.
So in this case, do I just close() or disconnect(), or both?
Justin Karneges
@jkarneges
@ericklind unless you have a specific need to track the presence of connections, I wouldn't worry about processing pongs or explicitly trying to close dead connections or anything
on the server side I'd simply ack closes (if ws.recv() returns null, call ws.close()) and ignore disconnects
Justin Karneges
@jkarneges
heartbeats from server->client can be useful even if the server isn't tracking connections though. for example, the server could send a message every 30 seconds, and if the client receives nothing in the last 60 seconds then the client can assume its connection is dead. in this case, it may be better for the server to use a TEXT frame for the heartbeat (instead of PING) since a reply would not be required
Erick
@ericklind
What is the difference, and how would that be done? Would I hold onto each wsContext and call wsContext.send(?)? Or is there a collection on the req.grip that has all the connections? (Docs are kind of lacking on this area)
Justin Karneges
@jkarneges
@ericklind definitely don't hold on to WebSocketContext. that's a temporary object which only lives as long as the current http request being handled by the backend. you can use a keep-alive control message to tell pushpin to send TEXT frames for you
ws.sendControl({"type": "keep-alive", "content": "still alive", "timeout": 30})
ccbeloy
@ccbeloy
@jkarneges - just to update from our side with regards to 502 ALB issue. It just start working, we did not do any change. So I guess its a misconfiguration issue with ALB. Thanks.
ccbeloy
@ccbeloy
Hi @jkarneges is there a limit on how many channels a single pushpin instance could handle? Are channels expensive in Pushpin?
Also found this in the pushpin config "connection_subscription_max=20". Is this the max number of channels per client connection? Can this be increased in a much larger number? And will be the drawback?
Thanks.
Justin Karneges
@jkarneges
@ccbeloy channels are not too expensive, but they have an expense. tens of thousands of channels in one instance should be fine. feel free to increase connection_subscription_max, just be mindful of the potential amplification that could cause
ccbeloy
@ccbeloy
Thanks @jkarneges
Jimmy Marquez
@jimmysp_gitlab

Hi @jkarneges
I am trying to send messages to pushping using 0mq but I am unable to do it, I am using the following code:

ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket publisher = context.socket(ZMQ.PUB);
publisher.connect("tcp://" + ip + ":5562");
publisher.send("J{\"id\": \"an-item-id\", \"channel\": \"test\", \"formats\": {\"http-stream\": {\"content\": \"a chunk of data\n\"}}}",0);
publisher.close();
context.term();

I am also using the following versions of 0MQ,

zeromq-4.1.4
zeromq/jzmq

The code does not display an error but messages are not appearing in pushpin logs.

Any suggestion related to this? might I be missing something?

Justin Karneges
@jkarneges
hi @jimmysp_gitlab , you need to send a multipart message, where the first part is the channel, and the second part is the payload
some other things to consider. there must be at least one subscriber, otherwise your publisher won't actually send anything to pushpin (and nothing will get logged). and it can take a moment for subscription information to be sent to the publisher. I suggest adding a 500ms sleep between the connect() and send()