by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 13 18:17

    jkarneges on develop

    handler: make http publish maxi… Merge pull request #47691 from … (compare)

  • Jul 29 18:41

    jkarneges on v1.30.0

    (compare)

  • Jul 29 18:40

    jkarneges on master

    limit body bytes to internal bu… runner: support condure service use client_maxconn and condure_… and 1 more (compare)

  • Jul 29 18:40

    jkarneges on develop

    update version (compare)

  • Jul 28 02:16

    jkarneges on develop

    use client_maxconn and condure_… (compare)

  • Jul 28 02:12

    jkarneges on develop

    runner: support condure service (compare)

  • Jul 22 01:42

    jkarneges on develop

    limit body bytes to internal bu… (compare)

  • Jul 15 18:45

    jkarneges on v1.29.0

    (compare)

  • Jul 15 18:45

    jkarneges on master

    fix crash when parsing accept h… provide credits on websocket ac… fix crash when response hold ti… and 10 more (compare)

  • Jul 15 18:45

    jkarneges on develop

    update version (compare)

  • Jul 13 21:36

    jkarneges on develop

    init field (compare)

  • Jul 13 18:59

    jkarneges on develop

    split websocket frames to fit i… (compare)

  • Jul 09 23:07

    jkarneges on develop

    update qzmq handler: only process body when… handler: don't allow pausing tw… (compare)

  • Jul 07 15:42

    jkarneges on develop

    update qzmq (compare)

  • Jun 26 20:54

    jkarneges on develop

    update submodules fix warnings (compare)

  • Jun 21 16:50

    jkarneges on develop

    fix crash when response hold ti… don't re-enable keep-alive if m… (compare)

  • May 22 03:40

    jkarneges on develop

    provide credits on websocket ac… (compare)

  • May 13 16:28

    jkarneges on develop

    fix crash when parsing accept h… (compare)

  • Apr 08 15:46

    jkarneges on master

    websocket-over-http: add option… update version (compare)

  • Apr 08 15:46

    jkarneges on v1.28.0

    (compare)

Nitesh Agarwal
@agarwal-nitesh
@jkarneges Thanks for the document link. A very simple setup is working for me. https://github.com/agarwal-nitesh/pushpin-prometheus-exporter A few more metrics would be helpful though. For e.g. currently, we get max active connections. If we can get the number of new connections, the number disconnected, message delivery time, etc. would be more helpful.
Justin Karneges
@jkarneges
good to see you got something working! I've made an issue about extra metrics: fanout/pushpin#47685
@agarwal-nitesh you can comment there if you want to suggest others. I'm not sure how message delivery time would work, but we could discuss
Brian Zier
@bzier

Hi @jkarneges , there are multiple references in the Pushpin documentation (e.g. here & here) to scaling out Pushpin and being able to use ZeroMQ to publish, rather than HTTP, and having that automatically forwarded to each instance. Are there any examples of how this can be configured? I am specifically experimenting with this using the docker-pushpin container in Kubernetes. I imagine that some of the ZeroMQ configuration values need to change, but it isn't apparent to me which ones, or how this would be configured. Please let me know if I've misunderstood something about what I read. I'm still familiarizing myself with ZeroMQ, so it is possible I am missing something obvious.

And slightly related, does the connection check work across multiple instances in this way as well, or does each instance need to be queried independently?

Thank you for the help.

Justin Karneges
@jkarneges
Hi @bzier , there are some samples in the tools folder https://github.com/fanout/pushpin/tree/master/tools
for example, edgebroker could run with each pushpin instance, connected to sourcebroker. then publishers could send to sourcebroker. there's also the push_in_sub_connect option which could obviate the need for a local component like edgebroker
the connection check is per instance
Nitesh Agarwal
@agarwal-nitesh
@jkarneges Wouldn't pushpin with a registry to something like consul would help to find all pushpin instances to publish to?
Justin Karneges
@jkarneges
@agarwal-nitesh yes that seems reasonable
Adrian
@picarsite

Hello @jkarneges I'm using Pushpin in a docker container. It seems that Mongrel2 has a problem with SSL. When it is starting, I get the following errors:

proxy_1 | [ERR] 2020-05-11 23:01:07.614 [m2 https:443] (src/unixy.c:123: errno: No such file or directory) Failed to open PID file /var/run/pushpin/mongrel2_443.pid for reading.
proxy_1 | [ERR] 2020-05-11 23:02:43.814 [m2 https:443] (src/connection.c:1082: errno: No such file or directory) Failed to find cert for api.MYDOMAIN.com

The last one is showing up couple of times when the container is running. I also get sometimes a handshake error in the logs. Both errors are probably connected together. My certs are under runner/certs (default_4430.crt and default_4430.key). It looks like Mongrel2 is looking for cert files with api.MYDOMAIN.com as filenames in the cert path but it is not finding it (I assume there are default_4430.crt and default_4430.key??). At least this is the comment in connection.c of Mongrel2:

// try to find a file named after the exact host, then try with a wildcard pattern at the
// same subdomain level. the file format uses underscores instead of asterisks. so, a
// domain of www.example.com will attempt to be matched against a file named
// www.example.com.crt and _.example.com.crt. wildcards at other levels are not supported.

Justin Karneges
@jkarneges
@picarsite a cert for the domain is looked up first before falling back to the default. that error should be harmless. but if you want it to go away, put two copies of your cert in the certs dir (as default_4430.crt and yourdomain.crt)
NIRANJAN
@niranjan-kmit
Hello every one, I am new to pushpin technology,Can any one guide or provide the path to write a back-end service to push pin using Java.
Justin Karneges
@jkarneges
welcome @niranjan-kmit . do you have any requirements for the client protocol, like sse or websockets? or not sure?
Jeremy Llewellyn
@jeremyll
Is there any examples of a multi-master pushpin deployments or centeralized pushpin pub/sub with proxies pointed to that? I cant' for the life of me find a solution that works outside of one process with pushpin.
NIRANJAN
@niranjan-kmit
@jkarneges it would be good to have websocket or Http in my case.
Justin Karneges
@jkarneges
@jeremyll a simple multi-instance approach is to publish messages to all pushpin instances. the instances that have applicable subscribers will deliver the message, the instances that don't will ignore
6 replies
Justin Karneges
@jkarneges
@niranjan-kmit I'd suggest starting with SSE over HTTP, as it is fairly straightforward. https://pushpin.org/docs/usage/#server-sent-events
basically, make an endpoint in your java backend that responds with Content-Type text/html and also sets Grip-Hold and Grip-Channel headers. then publish SSE-formatted data to the channel
you can publish using HTTP POST. there is an example of that in the section that immediately precedes the link I sent. just replace hello there\n with SSE format, like event: message\ndata: hello there\n\n
NIRANJAN
@niranjan-kmit
@jkarneges why i am getting "Error while proxying to origin." while openning the streaming connection to pushpin. I have changed pushpin conf file like * localhost:8000
Justin Karneges
@jkarneges

you mean the routes file? try setting the content to

*,debug localhost:8000

and then make a request

Justin Karneges
@jkarneges
@jeremyll for zmq with django grip, you can try something like:
GRIP_PROXIES = [{
    'control_zmq_uri': 'tcp://localhost:5563',
    'require_subscribers': True
}]
NIRANJAN
@niranjan-kmit
@jkarneges how do we know that specific channel doesn't have any subscribers ,the chanel which is created by backed service previously?
Justin Karneges
@jkarneges
@niranjan-kmit you can monitor for subscriptions via zmq, either the sub socket used for publishing or the stats socket. for what purpose?
NIRANJAN
@niranjan-kmit
Hello @jkarneges if any no clients subscribed to particular channel then i need to delete the channel?
Justin Karneges
@jkarneges
@niranjan-kmit there's no need to delete channels. they are created automatically and destroyed automatically
NIRANJAN
@niranjan-kmit
i mean i need to delete orphan channels
@jkarneges is there any particular time we have to destroy that orphan channel automatically?
Justin Karneges
@jkarneges
are you asking how long does it take pushpin to destroy a channel with no subscribers?
NIRANJAN
@niranjan-kmit
@jkarneges Yes
Justin Karneges
@jkarneges
for http stream and websocket transports it is immediately when the last subscriber going away. for http response (used for long polling) the channel lingers for about a minute in case a subscriber comes back
NIRANJAN
@niranjan-kmit
Thank you very much @jkarneges
NIRANJAN
@niranjan-kmit
@jkarneges Can you please provide some resource for getting insite about JWT validation in puspin proxy
Justin Karneges
@jkarneges
@niranjan-kmit validation of the Grip-Sig header? or something else?
NIRANJAN
@niranjan-kmit
@jkarneges Grip-Sig header
NIRANJAN
@niranjan-kmit
@jkarneges how can we validate token which is cumming from fronted using pushpin?
Justin Karneges
@jkarneges
@niranjan-kmit the signing key is specified in pushpin.conf sig_key
backend can validate using the same key
NIRANJAN
@niranjan-kmit
@jkarneges !GripControl.validateSig(session.getHeaders().get("grip-sig"), "<key>") above mentioned keys, from where grip-sig andand key comes?
Justin Karneges
@jkarneges
@niranjan-kmit Pushpin adds Grip-Sig to the proxied request. key is whatever is specified in pushpin.conf
NIRANJAN
@niranjan-kmit
if I want authorize valid user or request then how can do with the pushpin?
NIRANJAN
@niranjan-kmit
@jkarneges if I got the JWT Token from UI Service then how can I validate that token while making q request to the pushpin proxy?
@jkarneges Basically I want to implement Authentication/Autherization with Pushpin Proxy?
Justin Karneges
@jkarneges
@niranjan-kmit pushpin relays all headers sent by the client to the backend. you can then auth the client from the backend
ccbeloy
@ccbeloy
Hi @jkarneges ...is there an event pushpin raise when removing a channel? And how to subscribe to that event?v
Justin Karneges
@jkarneges
@ccbeloy see the zmq SUB socket. some info here: https://pushpin.org/docs/advanced/#subscription-forwarding
the code sample at the end of that section
NIRANJAN
@niranjan-kmit
@jkarneges How do I know that my channel X having N number subscriptions? can you please provide any reference since i am a beiginner for pushpin
Justin Karneges
@jkarneges
@niranjan-kmit you can find that on the subscription stats https://pushpin.org/docs/advanced/#stats-socket
NIRANJAN
@niranjan-kmit
Hi@jkarneges can we send json response with custome message using java grip library.