by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 13 18:17

    jkarneges on develop

    handler: make http publish maxi… Merge pull request #47691 from … (compare)

  • Jul 29 18:41

    jkarneges on v1.30.0

    (compare)

  • Jul 29 18:40

    jkarneges on master

    limit body bytes to internal bu… runner: support condure service use client_maxconn and condure_… and 1 more (compare)

  • Jul 29 18:40

    jkarneges on develop

    update version (compare)

  • Jul 28 02:16

    jkarneges on develop

    use client_maxconn and condure_… (compare)

  • Jul 28 02:12

    jkarneges on develop

    runner: support condure service (compare)

  • Jul 22 01:42

    jkarneges on develop

    limit body bytes to internal bu… (compare)

  • Jul 15 18:45

    jkarneges on v1.29.0

    (compare)

  • Jul 15 18:45

    jkarneges on master

    fix crash when parsing accept h… provide credits on websocket ac… fix crash when response hold ti… and 10 more (compare)

  • Jul 15 18:45

    jkarneges on develop

    update version (compare)

  • Jul 13 21:36

    jkarneges on develop

    init field (compare)

  • Jul 13 18:59

    jkarneges on develop

    split websocket frames to fit i… (compare)

  • Jul 09 23:07

    jkarneges on develop

    update qzmq handler: only process body when… handler: don't allow pausing tw… (compare)

  • Jul 07 15:42

    jkarneges on develop

    update qzmq (compare)

  • Jun 26 20:54

    jkarneges on develop

    update submodules fix warnings (compare)

  • Jun 21 16:50

    jkarneges on develop

    fix crash when response hold ti… don't re-enable keep-alive if m… (compare)

  • May 22 03:40

    jkarneges on develop

    provide credits on websocket ac… (compare)

  • May 13 16:28

    jkarneges on develop

    fix crash when parsing accept h… (compare)

  • Apr 08 15:46

    jkarneges on master

    websocket-over-http: add option… update version (compare)

  • Apr 08 15:46

    jkarneges on v1.28.0

    (compare)

Matt Huebert
@mhuebert
Hello all! I’m taking a first look at fanout/pushpin, liking what I see. I was wondering if there was any information about how one might implement “presence” for a chat app?
Adam McElwee
@acmcelwee
@jkarneges, it's been a bit since we've updated our pushpin version, and I'm looking at doing a round of updates. Configurable stats output format looks nice fanout/pushpin@863d994, and I'm actually re-examining our inclusion of zeromq in our app just for subscribing to the pushpin stats. Any thoughts on the option of potentially exposing those stats via a different mechanism, say a websocket?
The app that's collecting pushpin stats is a node app, and we seem to have found an unfortunate memory leak in the node 10 + zeromq.js 5.1.0 combo.
I suppose you get simple multi-subscriber support with zmq as the socket, though.
Justin Karneges
@jkarneges
hi @mhuebert , have the clients send heartbeats to the server, which can be used to update a presence table in a database. a user is considered online if they are in the table with a last heartbeat time that isn't too old. then use something like pushpin to send out the state changes
Justin Karneges
@jkarneges
@acmcelwee my preference is to have only the zmq output and people can make adapters
Adam McElwee
@acmcelwee
fair enough
I ended up writing a stats piping utility to pipe the stats from the zeromq socket in the container to a plain old unix domain socket, to make it simpler for other local things to connect and sub to stats.
~100 lines of python
so, that'll do
Matt Huebert
@mhuebert
I am trying out fanout for the first time, comparing using GripPubControl to using a plain fetch, passing a list of items. I seem to be doing something wrong, as in the second variant, I get a 200 response but no messages are actually published to the test channel (or, not that I see in my browser).
    // Version 1 - works
    const pub = new grip.GripPubControl({
        control_uri: CONTROL_URI,
        control_iss: realm.id,
        key: Buffer.from(realm.key, 'base64')
    })

    const res1 = await pub.publishHttpStream('test', 'event: message\ndata: "hello world"\n\n')

    // Version 2 - doesn't work (200 response, but browser doesn't see a message)
    const items = [
        { channel: "test", formats: { "http-stream": { content: "Hola" } } },
        { channel: "test", formats: { "http-stream": { content: 'event: message\ndata: "hola"\n\n' } } }]

    const res2 = await fetch(CONTROL_URI, {
        method: 'POST',
        headers: {
            'Authorization': `Bearer ${getToken()}`,
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ items: items })
    })
Matt Huebert
@mhuebert
^updated the CONTROL_URI to append /publish/, still get 200 but no publishing
Matt Huebert
@mhuebert
ok, I've figured out some of it. Definitely need to have the event: message\ndata: <data>\n\n wrapping.
Hmm interesting. I can publish via fetch repeatedly, get a 200/Published response, but see nothing in the browser... until I then publish via GripPubControl, and then all of those queued message show up at once
Matt Huebert
@mhuebert
eg. here is what my console in the browser looks like. It was blank while I published all of those fetch messages, then printed out all 7 immediately once I published via GripPubControl.
Sent via fetch, 1562184826176}
Sent via fetch, 1562184840649}
Sent via fetch, 1562184843494}
Sent via fetch, 1562184845946}
Sent via fetch, 1562184847815}
Sent via fetch, 1562184849443}
"Sent via GripPubControl, 1562184857648"
Matt Huebert
@mhuebert
Looks like the problem was incorrect formatting of the content
using a little helper function like this, and now I'm seeing reliable delivery:
const formatMessage = (s) => 
    `event: message\ndata: ${s.split("\n").join("\ndata: ")}\n\n`
Justin Karneges
@jkarneges
@mhuebert that all sounds right. the main thing to know is the HTTP streaming transport is just raw bytes, so in order to support SSE you need to publish data in the SSE format
getting this right can be a little fiddly though, as you found. we may consider adding SSE helpers to the libraries
Matt Huebert
@mhuebert
That makes sense. I am still wrapping my head around these format tradeoffs. Do most people using fanout/pushpin just choose one transport mechanism?
I’m curious about the practical ‘cost’ of using SSE for a situation where one wants to have a user’s channel subscriptions updated fairly often. I understand this would require reconnection, but if updates aren’t happening at a rapid pace then maybe this isn’t a big deal.
Matt Huebert
@mhuebert
Hm, are SSE connections always served via HTTP/1.1?
Justin Karneges
@jkarneges
most people choose one transport. I'm of the opinion that if the subscriptions don't change often, then a reconnect is probably not a big deal
SSE is not tied to any particular HTTP version. Pushpin doesn't support versions of HTTP higher than 1.1 yet, but you could put an HTTP/2 proxy in front and a client could maintain multiple SSE streams over one TCP connection
Matt Huebert
@mhuebert
Ah, excellent. I put a little test behind Cloudflare, and can go far beyond the 6 connection limit I was seeing with http1. I'd guess this will also make "reconnecting" when changing subscriptions smoother.
http2-sse.gif
Justin Karneges
@jkarneges
nice! :)
Matt Huebert
@mhuebert
Are PushPin docs in a repo somewhere? I would recommend changing the wording of: “To subscribe a request to multiple channels, respond with multiple Grip-Channel headers” to “respond with a comma-separated list of channels, or multiple Grip-Channel headers”. While technically correct that you can submit multiple distinct headers, it can be tricky to do so, eg. in situations where headers are represented as an object/map that doesn’t support duplicate keys. Though this did provide me with the opportunity to learn multiple-value headers work :) (https://stackoverflow.com/questions/3096888/standard-for-adding-multiple-values-of-a-single-http-header-to-a-request-or-resp)
Justin Karneges
@jkarneges
not a public repo. great suggestion though, I've changed the text
Michael DeRazon
@mderazon
Hi
First of all, very nice project and great documentation. Seems to be very well thought out
I was wondering how does Pushpin plays along with web push api and/or mobile push notifications. I know it is not supported out of the box, but I was wondering how this could be achieved.
Couple of things
  1. Send messages to clients that are not online, I want them to get them via the push notification and then once connected to the channel, have the message waiting for them there
  2. If the clients are connected, how do we prevent them from getting the message on the two channels - Pushpin and also the push api
Is there any blog post on the subject / material I can read from ?
Justin Karneges
@jkarneges
hi @mderazon , this is a really interesting topic that we have not written about anywhere. regarding your two questions, the main thing to understand is Pushpin only solves the problem of pushing data and nothing else. a good user notification system should probably have an inbox-like concept, where notifications are written into a database, which the user can later view/dismiss/etc. and knowing whether a user is online or not is a matter of receiving a heartbeat (or explicit "gone" message) from their client recently enough, and this could be managed by having a presence table in a database consisting of users and a timestamp. in my opinion, neither of these are strictly push problems, although they may be used with push in some way
I'd say, for 1) have a channel per inbox and always publish updates to it. if the user is offline they won't get anything. when they go online, read the inbox from the database. for 2) don't send via the push API if the user is considered online based on the presence table in the database
Michael DeRazon
@mderazon
@jkarneges that sounds good. I understand it's not in the scope of the project, it's just often very much part of the requirement of "real time apps" and surprisingly I haven't find a lot of material on it
GaneshMani
@ganeshmani
how to use pushpin with graphql?
Justin Karneges
@jkarneges
hi @ganeshmani , there's an example here https://github.com/fanout/apollo-serverless-demo
the readme describes fanout cloud, but the project works fine with pushpin
javier322
@javier322
Hello. How can I send data with json format using the HTTP control API?. The examples just show strings in content field.
Justin Karneges
@jkarneges
hi @javier322 , simply send the json as a string
Arye Shalev
@pantchox
Hi guys, who ever has a blog or post on Medium stuff that are related to programming, I am working on a new publishing platform, its my side project and i worked reallllly hard on it - I would love for your feedback! if you love publishing and curating content - PLEASE PM ME :)
if my message is not following chat room rules i will delete it! thanks.
Jason Cornez
@jcornez
Hello. I've made a change to pushpin.conf, changing the x_forward_for (from unset to "append"). I'd like to apply this setting without having to do a full restart. That is, I don't want to drop existing connections. I only want it so that new sub requests will supply the x-forward-for header. Is there a way to reload the config, or only restart partially?
Jason Cornez
@jcornez
Note, I'm using the pushpin runner, and typically would start things via service pushpin start
Jason Cornez
@jcornez
Is it fine to do killall -HUP pushpin-proxy, for example?
Jason Cornez
@jcornez
As far as I can tell, this is safe, but it doesn't cause pushpin.conf to be reloaded by the proxy.
Justin Karneges
@jkarneges
@jcornez indeed, pushpin.conf does not get reprocessed on SIGHUP. only the routes file gets reprocessed (for platforms where file change detection doesn't work)
Jason Cornez
@jcornez
Hello - thanks. So any other suggestion?
Justin Karneges
@jkarneges
the only other way I can think of handling this would be if you're using a load balancer and you have a way to route new traffic only to a pushpin with recent config while leaving the original one alone until everyone disconnects
Jason Cornez
@jcornez
@jkarneges thanks for your thoughts. Maybe this is a feature request? Seems like it would be useful.