fanout/pushpin

Reverse proxy for realtime web services

Justin Karneges

@jkarneges

you mean the routes file? try setting the content to

*,debug localhost:8000

and then make a request

Justin Karneges

@jkarneges

@jeremyll for zmq with django grip, you can try something like:

GRIP_PROXIES = [{
    'control_zmq_uri': 'tcp://localhost:5563',
    'require_subscribers': True
}]

NIRANJAN

@niranjan-kmit

@jkarneges how do we know that specific channel doesn't have any subscribers ,the chanel which is created by backed service previously?

Justin Karneges

@jkarneges

@niranjan-kmit you can monitor for subscriptions via zmq, either the sub socket used for publishing or the stats socket. for what purpose?

NIRANJAN

@niranjan-kmit

Hello @jkarneges if any no clients subscribed to particular channel then i need to delete the channel?

Justin Karneges

@jkarneges

@niranjan-kmit there's no need to delete channels. they are created automatically and destroyed automatically

NIRANJAN

@niranjan-kmit

i mean i need to delete orphan channels

@jkarneges is there any particular time we have to destroy that orphan channel automatically?

Justin Karneges

@jkarneges

are you asking how long does it take pushpin to destroy a channel with no subscribers?

NIRANJAN

@niranjan-kmit

@jkarneges Yes

Justin Karneges

@jkarneges

for http stream and websocket transports it is immediately when the last subscriber going away. for http response (used for long polling) the channel lingers for about a minute in case a subscriber comes back

NIRANJAN

@niranjan-kmit

Thank you very much @jkarneges

NIRANJAN

@niranjan-kmit

@jkarneges Can you please provide some resource for getting insite about JWT validation in puspin proxy

Justin Karneges

@jkarneges

@niranjan-kmit validation of the Grip-Sig header? or something else?

NIRANJAN

@niranjan-kmit

@jkarneges Grip-Sig header

NIRANJAN

@niranjan-kmit

@jkarneges how can we validate token which is cumming from fronted using pushpin?

Justin Karneges

@jkarneges

@niranjan-kmit the signing key is specified in pushpin.conf sig_key

backend can validate using the same key

NIRANJAN

@niranjan-kmit

@jkarneges !GripControl.validateSig(session.getHeaders().get("grip-sig"), "<key>") above mentioned keys, from where grip-sig andand key comes?

Justin Karneges

@jkarneges

@niranjan-kmit Pushpin adds Grip-Sig to the proxied request. key is whatever is specified in pushpin.conf

NIRANJAN

@niranjan-kmit

if I want authorize valid user or request then how can do with the pushpin?

NIRANJAN

@niranjan-kmit

@jkarneges if I got the JWT Token from UI Service then how can I validate that token while making q request to the pushpin proxy?

@jkarneges Basically I want to implement Authentication/Autherization with Pushpin Proxy?

Justin Karneges

@jkarneges

@niranjan-kmit pushpin relays all headers sent by the client to the backend. you can then auth the client from the backend

ccbeloy

@ccbeloy

Hi @jkarneges ...is there an event pushpin raise when removing a channel? And how to subscribe to that event?v

Justin Karneges

@jkarneges

@ccbeloy see the zmq SUB socket. some info here: https://pushpin.org/docs/advanced/#subscription-forwarding

the code sample at the end of that section

NIRANJAN

@niranjan-kmit

@jkarneges How do I know that my channel X having N number subscriptions? can you please provide any reference since i am a beiginner for pushpin

Justin Karneges

@jkarneges

@niranjan-kmit you can find that on the subscription stats https://pushpin.org/docs/advanced/#stats-socket

NIRANJAN

@niranjan-kmit

Hi@jkarneges can we send json response with custome message using java grip library.

as of now i see newFixedLengthResponse in the code

Justin Karneges

@jkarneges

@niranjan-kmit you can send any data format you want

NIRANJAN

@niranjan-kmit

@jkarneges Json formate with status code with message

like {200: "deployed succesfully"}

Justin Karneges

@jkarneges

@niranjan-kmit pushpin sends bytes. so if you want to send json, just encode the json to bytes and send the bytes :)

NIRANJAN

@niranjan-kmit

Hi @jkarneges my backend service running to validate token.after validation of token ineed to send response standard way with response code.

Justin Karneges

@jkarneges

@niranjan-kmit if you just want to respond normally then you can simply do that. if no grip instructions are provided, the proxying is a passthrough

NIRANJAN

@niranjan-kmit

Hi @jkarneges how can we implement heart beat service for pushpin? i want know the active connections

NIRANJAN

@niranjan-kmit

Hello . Can any one guide me to the implementation of heart beat endpoint with puhpin?

Justin Karneges

@jkarneges

@niranjan-kmit there are a couple of ways to do this. one is to subscribe every connection to a unique channel, and monitor the subscriptions to see who is connected. however, TCP connections that go away may linger for a long time, so a more accurate way to determine active connections is for the clients to send heartbeats, which pushpin will then pass along to your backend

NIRANJAN

@niranjan-kmit

@jkarneges any ref documentation we have for this?

Justin Karneges

@jkarneges

@niranjan-kmit for the first approach, see the code sample at the end of this section for monitoring subscriptions https://pushpin.org/docs/advanced/#subscription-forwarding

the second approach is just generic proxying, so there's nothing specific written about it

NIRANJAN

@niranjan-kmit

Hi @jkarneges I want to include the timestamp to message before pushpin send the message to client? so that i can find out the latency from pushpin to client

Sasha K

@akovalev

Hey,

thanks for this project and writing very detailed documentation. I really enjoyed reading it.

At the moment I'm trying to understand the difference between Pusher and Fanout/Pushpin capabilities, especially with regards to possibility to subscribing to a given channel on-demand.

If I understood correctly, Pushpin doesn't provide any option to specify which channel a client should be subscribed to when using HTTP/Server-Sent Events as a realtime transport, except for the initial "negotiating" phase which takes place when a client sends HTTP request to the location of the remote resource serving the events/messages, e.g. a client might encode a channel it's interested in, say, as a query parameter of the remote resource URL. In other words, it looks like everything depends on the instructions sent back from origin server to Pushpin Proxy via Grip-Channel HTTP Response header.

At the same time, if one decides to use WebSocket/WebSocket-over-HTTP as a real transport, it seems that it's possible to communicate client's intention to subscribe to a given channel by sending a WebSocket frame from the client app and expect that frame to be handled the origin server.

To provide you with more context, I can describe the potential use-case in which on-demand subscription to channels might be useful: for example, we have an offline-friendly web app that manages/syncs the content of collaborative documents with the server. e.g. we persist local copies of the documents in the browser, say, in IndexedDB and periodically sync these document with server. Each user might have access to multiple documents, but we don't want to sync all documents in the background at the same time, but rather "postpone" the synchronisation until the moment when user explicitly opens it. Moreover, even if later on we decide to sync multiple documents (like last N recently opened documents) at the same time, we would like to do it efficiently via 1 HTTP/multiplexed connection rather than using 1 connection per document as there's a limit for 6 simultaneous TCP connections to a given hostname in modern browsers unless HTTP2 is used which is not the case with Pushpin if I understand correctly.

That being said, what would you recommend if it's needed to subscribe/unsubscribe to channels on-demand based on certain event occurring in the runtime of a client app? Is it feasible to achieve by choosing WebSocket as realtime transport and implementing ?

1 reply

Justin Karneges

@jkarneges

hi @akovalev , welcome to the room. indeed, the WebSocket transport would be more appropriate for managing multiple subscriptions dynamically. note that it is possible to change subscriptions with the HTTP streaming transport as well (by using reliable mode and triggering the change out-of-band), so for example a client could POST to some other endpoint to change the channels of a connection they have open, but this is too stateful of an API design for my taste

pushpin doesn't support HTTP/2 yet, but you could put another proxy in front of pushpin that does, such haproxy or nginx. that way the client could potentially use only one connection for multiple HTTP streams. I'd probably prefer that approach over trying to change the channels of an existing HTTP connection

Justin Karneges

@jkarneges

bidirectional WebSocket communication is absolutely an appropriate use-case for pushpin. in some sense, pushpin is more of a connection manager for stateless backends than purely a push solution. though, I think anyone using pushpin would also be pushing with it :)

Sasha K

@akovalev

thanks @jkarneges

I was also thinking about notifying an origin server via a separate endpoint and also didn't like this idea too much as it implies keeping a certain state on the server. Another option that I was considering is to to subscribe a given user to all documents they have access to, but I'm afraid it will result in an excessive payload on the channel since as I've said before client becomes interested in document updates only under certain circumstances, i.e. we are going to push updates from origin server to the Pushpin proxy even when the client isn't interested in them.

as for HTTP/2 proxy in front of pushpin -- having asked my initial question I found this thread 1 where you recommended pretty much the same and it turned out to be working pretty well.

Also, it's nice to hear that bi-directional WebSocket communication is a valid usage of Pushpin.

Brian Zier

@bzier

The Java GripControl library does not properly decode events containing unicode. I've submitted a pull request to the repo here:
fanout/java-gripcontrol#1
Please let me know if there is anything I need to do in order to get the change merged and a new version created. Thanks.

Where communities thrive

People

Repo info

Activity