These are chat archives for fanout/pushpin

5th
Jul 2018
iahoorai
@iahoorai
Jul 05 2018 04:45
completely new to RealTime applications . how do i handle http stream on client side if I want to for example update an existing element on a page ? , do I have to use SSE or is there a way to have javascript call back that updates the page upon receiving more data
Justin Karneges
@jkarneges
Jul 05 2018 17:29
@iahoorai Pushpin is used for pushing data (via a variety of data transports) to receiving clients, which could be browsers. in the case of browsers you'd need javascript code to handle the received data
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:30
@jkarneges about production tips, have you some one? and other little question, I can just scale pushpin horizontally without issues at moment that I whant publish a message to a channel is connection is handle by pushpin-1 and control message is get by pushpin-2
Justin Karneges
@jkarneges
Jul 05 2018 17:31
what you do with the received data is up to you. if you want to easily update existing elements on a page when data arrives, you might want to use a UI framework of some kind like angular or react. but this is very outside the scope of pushpin
@jorgecuesta if you have multiple pushpin instances, you need to make sure you publish data to the instances that have listeners. an easy way to handle this is to simply publish to all instances. a more advanced way is to track subscriptions per-instance and publish only to certain instances
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:36
On incoming request to target came some kind of information that let me know which pushpin instance is that handle current connection id?
because if that came I could just relate connection-id/source to publish message to that source server
Justin Karneges
@jkarneges
Jul 05 2018 17:39
it's better to track subscriptions rather than connections. there are two ways to do this: 1) monitor the stats socket for subscription events and store the state somewhere, 2) use a zmq PUB socket for publishing instead of HTTP. the PUB socket automatically figures everything out
I recommend publishing via zmq PUB rather than monitoring the stats socket though. it's way easier
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:41
is is some doc about your recommendation?
I’m using pubcontrol from you too to use http, so I need change to any other library maybe?
Justin Karneges
@jkarneges
Jul 05 2018 17:43
there isn't much docs about publishing via PUB, but there's some info here https://pushpin.org/docs/usage/#publishing
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:44
but anyway if I have 2 pods of pushpin
should I publish using PUB to both?
Justin Karneges
@jkarneges
Jul 05 2018 17:45
@jorgecuesta what language are you using pubcontrol?
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:45
nodejs
Justin Karneges
@jkarneges
Jul 05 2018 17:46
yes you still connect to both, but the local zmq library will track subscriptions from both instances and it will only actually send data if they are listening
for now there's only a python example, but maybe you can use as reference: https://github.com/fanout/pushpin/blob/master/tools/publish2.py
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:48
that local zmq library if I start two pushpin on same container? because I’m talking about scaling it with multi containers (like replicas)
is not a problem python I know it too :)
Justin Karneges
@jkarneges
Jul 05 2018 17:49
the zmq library of the publisher, in memory
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:50
ok so you idea is connect zmq library (in memory) to each pushpin instance , so it will handle to which one it should send my message based on right subscription, that is?
Justin Karneges
@jkarneges
Jul 05 2018 17:51
yes
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:52
Ok so the unique troubble is know each pushpin instance
Justin Karneges
@jkarneges
Jul 05 2018 17:52
yes each publisher will need to call connect() for each instance
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 17:53
yes maybe I should does another service in the middle to skip my main service need to connect each pushpin instance
because if I want add one more pod is easy, but anyone whould publish to it if they does not know about it and his connection info
Justin Karneges
@jkarneges
Jul 05 2018 17:59
right. and for that there are a bunch of ways you could do things. you could integrate with something like redis or rabbit. or make your own broker thing
you can see sourcebroker.py and edgebroker.py in pushpin tools folder for some inspiration
edgebroker lives with each pushpin, connects to the local pushpin sub socket, and also connects out to one or more sourcebrokers
sourcebroker listens for connections from edgebroker and listens for connections from publishers. so publishers send to sourcebroker, which sends to edgebroker (which then sends to pushpin over localhost)
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 18:04
so I could have different ways to know that a new pushpin is on
to dispatch a message about it to handle that new pub socket
Adam McElwee
@acmcelwee
Jul 05 2018 18:08
For another data point that works well for us, we went with a model of a slightly more intelligent "sidecar" that's deployed with each pushpin "pod", to use the kubernetes terminology. That sidecar subscribes to one or more redis pub/sub channels and publishes the messages to the adjacent pushpin's port 5561.
This let us remove the discovery problem that you're talking about @jorgecuesta, and publishers just publish to a common redis channel. As our pushpin nodes scale out horizontally, no application-layer things need to know about any changes.
You could obviously swap zmq/rabbitmq/etc for redis.
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 18:12
@acmcelwee you say something like a node/python running with pushpin that handle (each one) the same rabbitmq/redis? Channel to listen about a common message path and push that message on adjacent pushpin instance, so pushpin will try to see is that subscription is handle by him
that is?
thanks @acmcelwee and @jkarneges :)
just one last thing some kind of config or setup that could improve to use pushpin in production?
Justin Karneges
@jkarneges
Jul 05 2018 18:26
@jorgecuesta use stats socket for instrumentation. be aware of the various tuning options in pushpin.conf, mainly message_rate and message_hwm
defaults are hopefully good though
Jorge S. Cuesta
@jorgecuesta
Jul 05 2018 18:31
Thanks @jkarneges