Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 11 16:58

    jkarneges on v1.26.0

    (compare)

  • Dec 11 16:58

    jkarneges on master

    if root path requested on http … options should not require cont… update version (compare)

  • Dec 11 16:58

    jkarneges on develop

    update version (compare)

  • Dec 10 22:37

    jkarneges on develop

    if root path requested on http … options should not require cont… (compare)

  • Nov 20 15:47

    jkarneges on v1.25.0

    (compare)

  • Nov 20 15:47

    jkarneges on master

    handle DEBUG lines in mongrel2 … Merge pull request #47680 from … set log level in mongrel2 confi… and 3 more (compare)

  • Nov 20 15:46

    jkarneges on develop

    update version (compare)

  • Nov 20 05:55

    jkarneges on develop

    per-service log levels (compare)

  • Nov 09 02:01

    jkarneges on develop

    set log level in mongrel2 confi… Merge pull request #47681 from … (compare)

  • Oct 24 16:21

    jkarneges on develop

    handle DEBUG lines in mongrel2 … Merge pull request #47680 from … (compare)

  • Oct 12 15:11

    jkarneges on develop

    fix typo (compare)

  • Oct 12 15:11

    jkarneges on master

    fix typo (compare)

  • Aug 06 15:19

    jkarneges on v1.24.0

    (compare)

  • Aug 06 15:19

    jkarneges on master

    runner: don't create log dir if… Remove date parsing from mongre… Merge pull request #47676 from … and 1 more (compare)

  • Aug 06 15:19

    jkarneges on develop

    update version (compare)

  • Aug 03 14:14

    jkarneges on develop

    Remove date parsing from mongre… Merge pull request #47676 from … (compare)

  • Aug 03 01:58

    jkarneges on develop

    runner: don't create log dir if… (compare)

  • Aug 03 01:51

    jkarneges on develop

    send mongrel2 logging to /dev/s… format mongrel2 log messages ac… format mongrel2 access log acco… and 4 more (compare)

  • Aug 03 01:50

    jkarneges on master

    send mongrel2 logging to /dev/s… format mongrel2 log messages ac… format mongrel2 access log acco… and 4 more (compare)

  • Jul 03 17:13

    jkarneges on master

    option to have the push_in_sub … set mongrel2 option disable.acc… forward numeric log level to ha… and 6 more (compare)

Justin Karneges
@jkarneges
hi @javier322 , there's an sse + react example here https://github.com/fanout/leaderboard
Matteo Restelli
@reste85

Hi all!
First of all congrats for this wonderful project. I've a question about scalability:
In your docs is stated this "ìPushpin is horizontally scalable. Instances don’t talk to each other, and sticky routing is not needed. Backends must publish data to all instances to ensure clients connected to any instance will receive the data. Most of the backend libraries support configuring more than one Pushpin instance, so that a single publish call will send data to multiple instances at once."

What's the best way to handle autoscaling of pushpin instances in this case? I mean if i've configured 3 instances in my backend and automatically 2 instances gets added, what is the best way to handle those 2 instances?

Thank you!

Justin Karneges
@jkarneges
hi @reste85 , in that case you should route messages through some broker, and have pushpin receive from the broker (e.g. by making an adapter and running it in a sidecar with each pushpin instance)
Matteo Restelli
@reste85
ok @jkarneges thank you very much
Matteo Restelli
@reste85

@jkarneges Sorry for this others questions:
1) Have you tried to integrate with some broker like Kafka? Do you think it's possible to integrate several pushpin instances with Kafka?
2) How pushpin routes to clients messages, in this case? I mean, if i'm using a single kafka topic to deliver frontend messages, how they are delivered to clients subscribed to different channels? The message must be in some sort of format which indicates also the destination channel?

Thank you again, and again, congratulations for this wonderful technology

@jkarneges forget my first question, i've saw your previous reply:

"nervouscat it can certainly be wired up to kafka https://github.com/fanout/kafka-sse-example"

Justin Karneges
@jkarneges
@reste85 regarding your second question, yes, you could do something like that. you could include the channel explicitly in the message, or let the adapter derive a channel from the content
Matteo Restelli
@reste85
@jkarneges ok so this is something to be performed inside the adapter/sidecar, Pushpin is not able to do so, am i right?
@jkarneges sorry, last question: what about subscription to channels via wildcard? I saw an old open issue in the repo, maybe was let open due to low priority...
Justin Karneges
@jkarneges
@reste85 for the most part. the one thing pushpin can do on its own is connect out via zeromq, using the recently added push_in_sub_connect option. note that this would require the zeromq broker to carry messages in the format expected by pushpin as there would not be any transformation
wildcard is not possible yet
Matteo Restelli
@reste85
ok thank you very much @jkarneges
Anton Christensen
@anton-christensen
@jkarneges We observe that chrome seems to silently drop streaming HTTP connections if a device loses network connection (say the user closes the lid on their laptop). Do you know anything about why this might be? we'd prefer not to have to implement sockets and just keep using HTTP-streaming. Thanks, I know this is probably a bit off topic :)
Justin Karneges
@jkarneges
hi @anton-christensen , are you using eventsource? this may help: https://github.com/fanout/reconnecting-eventsource
Nishant karve
@KarveNishant_twitter
@jkarneges I am back! I am too excited for this tool. I am trying to write the relay.py code using Kafka's python client library instead of Confluent Kafka python library. I was able to get the relay,py working as well and tested it by producing some messages on the Kafka topic and receiving it with the relay. However, I am unable to receive these messages on to the subscribed channel through the API "curl -i http://localhost:7999/events/pushpin/". What could be the reason? Am I missing something obvious? The "relay.py" code works very well and it does display the messages I produce on the Kafka topic
Justin Karneges
@jkarneges
hi @KarveNishant_twitter , make sure you're subscribing and publishing to the same channel (obviously), and make sure you are publishing a proper format. note that you might not see output in curl unless your payloads contain newlines
Nishant karve
@KarveNishant_twitter
@jkarneges I did confirm that the channels are correct on both publishing and receiving side. I am publishing the message directly on Kafka topic using Kafka Console Producer library hoping that the subscribed channel will receive it through the bridge. There isn't any way for me to add a newline at the end of the JSON message I am trying to publish.
Justin Karneges
@jkarneges
@KarveNishant_twitter it is your code that is publishing data to pushpin, so you can send any data you want. what I mean is, you could add the newline there. it doesn't have to be in the original message from kafka. remember that you aren't sending json to the client, you're sending a chunk of HTTP response bytes. those bytes can be anything, and they should be what you want the client to receive
Anton Christensen
@anton-christensen
@jkarneges No we're using plain old XMLHttpRequest. Eventsource doesn't seem to be supported on IE and edge.
shahtajk7
@shahtajk7
Hi, I like to know some details about pushpin.
do they provide Rest APIs ?
Is there any link from which I can contact their customer support ?
Justin Karneges
@jkarneges
@anton-christensen you should be able to know when an XMLHttpRequest has errored out, using onerror or onreadystatechange or something like that
hi @shahtajk7 , the company behind Pushpin is Fanout, which offers commercial support. you can email sales@fanout.io about that
Justin Karneges
@jkarneges
what do you mean by "provide Rest APIs" ? data can be published to pushpin using HTTP requests, if that's what you mean. the subscriber-side API though is entirely up to you
Alois Napitalai
@anapitalai
I want o accomplish api management using http streams, what would be the config using express-grip in a nodejs backend?
API management using pushpin
Anton Christensen
@anton-christensen
@jkarneges I run against the 6-or-so connection limit in chrome, I saw a previous message here saying pushpin doesn't support HTTP2 but that accepting http2 connections would solve the problem. I haven't been able to set up nghttpx in my docker environment yet. I just wanted to hear if you could recommend a way of accepting HTTP2 connections for pushpin
Justin Karneges
@jkarneges
@anton-christensen yeah if you put a proxy in front with http2 support it should work
@anapitalai there is some example config in the readme https://github.com/fanout/express-grip . be sure to set gripProxies with an entry set to pushpin's control port (normally 5561)
if you have a more specific question I can try to help
Alois Napitalai
@anapitalai

@anton-christensen I am doing as whats in the express-grip doc, 'not implemented' is the error.Below is the express app code......const express = require('express');
const router = express.Router();
const grip = require('grip');
const expressGrip = require('express-grip');
router.use(expressGrip.preHandlerGripMiddleware);

// Pushpin and/or Fanout.io is used for sending realtime data to clients
const myConfigObject = {
gripProxyRequired: true,
gripProxies : [
// Pushpin
{
'control_uri': 'http://localhost:5561'
},
// Fanout.io
{

    }],

};

expressGrip.configure(myConfigObject);

router.get('/', function(req, res, next) {
try {
// If the request didn't come through a GRIP proxy, throw 501
if (!res.locals.gripProxied) {
res.sendStatus(501);
return;
}

    // Subscribe every incoming request to a channel in stream mode
    expressGrip.setHoldStream(res, 'channel');
    res.send('stream open\n');

    // Alternatively subscribe and long-poll
    //expressGrip.setHoldLongpoll(res, '<channel>', <timeout>);
    //res.end();
} finally {
    // next() must be called for the post-handler middleware to execute
    next();
}

});

router.post('/', function(req, res, next) {
var data = req.body;

// Publish stream data to subscribers
expressGrip.publish('channel', new grip.HttpStreamFormat(data + '\n'));

// Alternatively publish response data to long-poll clients
//expressGrip.publish('<channel>',
//        new grip.HttpResponseFormat(null, null, null, data));

res.send('Ok\n');
next();

});

// Add the post-handler middleware to the back of the stack
router.use(expressGrip.postHandlerGripMiddleware);

/** var pub = new grip.GripPubControl({
'control_uri': 'http://localhost:5561'
});

pub.publishHttpStream('channel', 'hello realtime mahn\n');

res.writeHead(200, {
    'Content-Type': 'text/plain',
    'Grip-Hold': 'stream',
    'Grip-Channel': 'channel'
});
res.end('Stream opened, prepare yourself!\n');

});**/

module.exports = router;

@anton-christensen If I have an existing express app server REST api, how do I go about connecting that to pushpin to enable realtime?
Justin Karneges
@jkarneges
@anapitalai are you making the request through pushpin?
also for sharing large amounts of code like this, better to put in a paste bin
Anton Christensen
@anton-christensen
@jkarneges Yeah i was wondering if you either had an example with a http2 proxy, or if you have experience with any particular one. I use Caddy as my reverse proxy, it supports http2 but it won't accept http2 and forward http1. If pushpin doesn't accept http2 then docker won't negotiate http2 with the client
Justin Karneges
@jkarneges
@anton-christensen I believe I was able to do this with nginx, which forwarded using http1
Alois Napitalai
@anapitalai
@anton-christensen sorry for that, I will put the code in pastbin and get back to you.
Nate Dudenhoeffer
@AgDude
I have pushpin running with a route pointed to a DNS entry. When ip address associated with that domain changes, how do I get pushpin to read the new value? It seems to be cached somewhere and restarting the pushpin service doesn't reload it. (this is not a ttl issue, using host from the same system resolves to the correct new address).
Alois Napitalai
@anapitalai
I have a route /teachers, how is that entered in the routes file? * localhost:3007/teachers?
Justin Karneges
@jkarneges
hi @AgDude , see if restarting zurl helps. if it does, consider upgrading zurl since there was a dns cache bug that was fixed
@anapitalai route targets don't have a path. the path specified by the client is forwarded to the target
Alois Napitalai
@anapitalai
I have two endpoints from a REST API I created,https://pastebin.com/YBMPtB05, I can I get that to work using express-grip ,the APIs are created using express.
Justin Karneges
@jkarneges
@anapitalai you can add express-grip to the app, sure. the middleware shouldn't cause existing endpoints to break or anything
then you can add new endpoints (or modify existing ones) to use grip features
Nate Dudenhoeffer
@AgDude
Thanks for the pointer. I have tried yet, but that is probably the issue. We are running zurl 1.9.1.
Rob Flores
@robflores13
I was hoping someone could help me troubleshoot a pushpin SSL issue? I'm attempting to evaluate pushpin for my company, and we're using the pushpin docker image. We're using a company root certificate, which I've installed into the container's root certs, and setup the PEM crt/key files in the pushpin/runner/certs named according to the ports in the .conf file. However, when starting up the container, we get nonstop SSL Handshake Failed errors from mongrel. I'm not sure where else to troubleshoot the SSL handshake failure, as I don't know what pushpin is trying to connect to without an incoming request?
Rob Flores
@robflores13
We were missing the root CA certificate on the docker host. After installing it we're able to connect via SSL to pushpin now. However, the log files are still filling up with (resource unavailable) SSL handshake errors. These happen even though there are no incoming REST requests to the pushpin gateway. It seems something else is trying to connect to it or pushpin is trying to connect to something else. Anyone know what connections pushpin might be experiencing?
Justin Karneges
@jkarneges
@robflores13 maybe that's from a health checker making requests?
Rob Flores
@robflores13
@jkarneges That's a good thought Justin, thanks. I'll go take a look.