Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 09 18:54

    jkarneges on master

    update dev version (compare)

  • Jun 09 18:49

    jkarneges on v1.32.2

    (compare)

  • Jun 09 18:49

    jkarneges on master

    update version (compare)

  • Jun 09 18:43

    jkarneges on master

    sockjs: handle websocket setup … (compare)

  • May 15 22:32

    jkarneges on master

    update changelog (compare)

  • May 13 19:10

    jkarneges on master

    update dev version (compare)

  • May 13 19:09

    jkarneges on v1.32.1

    (compare)

  • May 13 19:09

    jkarneges on master

    update version for patch release (compare)

  • May 12 21:47

    jkarneges on master

    ensure bin dir exists ensure install target is create… (compare)

  • May 12 17:14

    jkarneges on master

    use ':' for portability (compare)

  • May 11 17:23

    jkarneges on master

    update dev version (compare)

  • May 11 17:22

    jkarneges on v1.32.0

    (compare)

  • May 11 17:22

    jkarneges on master

    update changelog (compare)

  • May 11 17:17

    jkarneges on v1.32.0

    (compare)

  • May 11 17:16

    jkarneges on master

    update dev version (compare)

  • May 11 17:13

    jkarneges on v1.32.0

    (compare)

  • May 11 17:13

    jkarneges on master

    tweak package script (compare)

  • May 11 02:25

    jkarneges on master

    use rustls-native-certs use StreamOwned to simplify imp… (compare)

  • May 10 21:17

    jkarneges on master

    pushpin-publish: support basic … (compare)

  • May 10 20:47

    jkarneges on master

    pushpin-publish: support https (compare)

Justin Karneges
@jkarneges
@bkowzx sure, that's a good example. the base lib is serve-grip, which is here: https://github.com/fanout/js-serve-grip
bkowzx
@bkowzx
@jkarneges thank you, using that now. But now I'm getting a 502 error when trying to connect via websocket. I'm using JS / Express. Any idea what I could be doing wrong. I'm using the demo code but wsContext.isOpening() is always False.
Connection is not opening up because of this.
Justin Karneges
@jkarneges
@bkowzx what do the pushpin logs show? the proxy logs. Maybe you need over_http on the route target
Justin Karneges
@jkarneges
@/all Hi everyone! We (Fanout, the company that makes Pushpin) have announced a commercial version for Kubernetes. If you're using Pushpin at a company, please check it out! Our commercial offerings are how we fund development. https://blog.fanout.io/2021/04/27/cloud-native-platform-for-push-apis/
Jonathan Monette
@jmoney
i saw bintray is sunsetting and then i saw this issue: fanout/pushpin#47695
curious when the dockerfile will be updated to not use bintray?
also, is the recommended approach to do a apt-get install pushpin to install the binary or leverage the dockerfile? i think the docs only mention the dockerfile
Justin Karneges
@jkarneges
hi @jmoney , the Dockerfile will be updated immediately after the new repository is ready
https://pushpin.org/docs/install/ covers the various ways to install, including apt
personally I would just use apt unless docker is specifically needed
Jonathan Monette
@jmoney
thanks! any idea when the repo will be ready?
ASHMEET KANDHARI
@ashmeet-kandhari

Hi @jkarneges ,
For Websockets using puspin and custom backend, Is it possible to send a ping message from websocket client and pushpin handles it automatically?

For eg : According to https://pushpin.org/docs/advanced/#keep-alives
If I try to send message {"type": "ping"} from websocket client after a connection is established

From my observation the request is going to custom backend, but want utilize pushpin's out of the box pin-pong feature

Justin Karneges
@jkarneges
@ashmeet-kandhari it is not possible for pushpin to automatically handle websocket messages. doing that in a general way could be hard, but I'm open to ideas
3 replies
do you still want other client messages to go to the backend?
1 reply
ASHMEET KANDHARI
@ashmeet-kandhari

Hi @jkarneges ,
With regards to the pushpin with custom backend (java)
Can you guide us how to respond to PING and PONG events when the custom backend receives it.
Sharing sample code which we are trying to use to respond to PING events

Notes: we use Websocket over http and getting the request as String type

 public Mono<ResponseEntity<String>> serve(@RequestBody(required = false) String body) {
                    List<WebSocketEvent> inEvents = GripControl.decodeWebSocketEvents(body + "\r\n");
                    List<WebSocketEvent> outEvents = new ArrayList<>();
                     HttpHeaders responseHeaders = new HttpHeaders();
                    if (inEventsIsEmpty(inEvents) && (isEventType(inEvents, "PING"))) {
                            outEvents.addAll(inEvents);
                            Map<String, Object> pingMessage = new HashMap<>();
                            pingMessage.put("content", inEvents.get(0).content);
                            pingMessage.put("message-type", "pong");
                        for (WebSocketEvent event : inEvents) {
                            System.out.println("here");
                            System.out.println(event.type);
                            System.out.println(event.content);
                        }
                        System.out.println("end");
                        logger.info("Received PING event: {}", inEvents);
                        if (isEventType(inEvents, "PING")) {
                            outEvents.add(0, new WebSocketEvent("TEXT", "c:" + GripControl.webSocketControlMessage("keep-alive", pingMessage)));
                        } else {
                            outEvents.add(0, new WebSocketEvent("PING"));
                        }

                        // set metadata to the ws-gw header.
                        if (!ObjectUtils.isEmpty(authToken.getSubject())) {
                            responseHeaders.add("Set-Meta-User", authToken.getSubject());
                        }
                        String responseBody = createResponseBodyWithHeaders(authToken, outEvents, responseHeaders);
                        System.out.println("RESPONSE:" + responseBody);
                        ResponseEntity<String> stringResponseEntity =
//                                    new ResponseEntity<>(responseBody, responseHeaders, HttpStatus.OK);
                                ResponseEntity.ok()
                                        .headers(responseHeaders)
                                        .body(responseBody);
                        return Mono.just(stringResponseEntity);
                    }
}
 private boolean isEventType(List<WebSocketEvent> inEvents, String text) {
        return inEvents.get(0).type.equals(text);
    }
Justin Karneges
@jkarneges
@ashmeet-kandhari I'm not sure, but would (inEventsIsEmpty(inEvents) && (isEventType(inEvents, "PING"))) ever be true?
also, if you receive a PING and want to respond immediately with PONG, then you should outEvents.add(0, new WebSocketEvent("PONG"));
ASHMEET KANDHARI
@ashmeet-kandhari

@jkarneges sorry this should be inEventsIsNotEmpty(inEvents) and we were able to receive PING event
I tried to send outEvents.add(0, new WebSocketEvent("PONG")) immediately but was getting error

2021-04-30 21:52:03,166 - DEBUG - client - state = OPEN
2021-04-30 21:52:23,176 - DEBUG - client > Frame(fin=True, opcode=9, data=b'n\xdd.\xfd', rsv1=False, rsv2=False, rsv3=False)
2021-04-30 21:52:31,156 - DEBUG - client - event = eof_received()
2021-04-30 21:52:31,161 - DEBUG - client ! failing OPEN WebSocket connection with code 1006
2021-04-30 21:52:31,165 - DEBUG - client - event = connection_lost(None)
2021-04-30 21:52:31,170 - DEBUG - client - state = CLOSED
2021-04-30 21:52:31,174 - DEBUG - client x code = 1006, reason = [no reason]
2021-04-30 21:52:31,178 - DEBUG - client - aborted pending ping: 6edd2efd

We are using websockets python [1] library to connect to pushpin websocket which internally sends PING events with random 4 bytes added to it
[1] - https://pypi.org/project/websockets/

9 replies
Justin Karneges
@jkarneges
oh, be sure to send OPEN if you receive OPEN
1 reply
Rob Flores
@robflores13
A question about some design patterns around channel management on the pushpin connections. We have clients that request a stream be opened for a specific combination of query parameters. Typically, the queries are unique enough that there are really only 1 or 2 clients listening on a channel (the clients request a combination of streamed attributes from a list of thousands). The REST endpoint receives the query, sets up the channel with Grip-Headers, and then instructs another backend process to start streaming the requested attributes through pushpin messages with the correct channels . I'm having trouble figuring out how to determine when a channel no longer has a listening client, so I can tear down the stream from the back end. Otherwise, the backend is sending unwanted messages to the pushpin proxies. Any thoughts on handling terminated connections, or how to use Pushpin with custom streams for clients?
Justin Karneges
@jkarneges
Hi @robflores13 , Pushpin exposes a ZeroMQ socket that tells which channels have at least 1 subscriber. have a look at https://pushpin.org/docs/advanced/#subscription-forwarding
BTW, the commercial version makes this easier, if that's of interest
Rob Flores
@robflores13
Thanks @jkarneges. How do I find out the details of the commercial version?
Justin Karneges
@jkarneges
@robflores13 commercial page: https://fanout.io/enterprise/ . the subscription tracking feature isn't specifically mentioned there, but it's in the API docs: https://docs.fanout.io/docs/subscription-feeds
Antoine Boisier-Michaud
@Aboisier

Hi! I am trying to use Pushpin's Websocket-Over-Http/GRIP with .NET Core. The .Net sdk uses a very old version of Newtonsoft, so instead of doing dependencies binding-redirects shenanigans, I thought I'd just implement the protocol, since it is pretty simple.

I am having trouble though, it seems like the request just passes through the proxy. When my server receives "OPEN\r\n" on route /foo/, it replies this:

HTTP/1.1 200 OK
Content-Type: application/websocket-events
Sec-WebSocket-Extensions: grip

OPEN\r\n

It seems like this respects the procotol, but when I try connecting to ws://localhost:7999/foo using wscat, I get error: Unexpected server response: 200.

  • I tried returning status code 101 instead, but I then get error: Unexpected server response: 101.
  • I tried sending regular ws negotiation headers (Upgrade: websocket, Connection: Upgrade, Sec-WebSocket-Accept: <the base 64 thingy>), no success.

Pushpin logs were net really useful.

[INFO] 2021-05-07 12:24:55.419 [zurl] IN id=68ea998e-7032-45b3-a2af-9eabc64bcbb2, POST https://localhost:7999/opportunity/hub
[INFO] 2021-05-07 12:24:58.380 [zurl] OUT id=68ea998e-7032-45b3-a2af-9eabc64bcbb2 code=200 6 M
[INFO] 2021-05-07 12:24:58.381 [proxy] GET ws://localhost:7999/opportunity/hub -> elmer-quotes-dev:443[http] code=200 6

Any help is welcome! I am not sure how to find the root cause of this problem. :/

4 replies
ASHMEET KANDHARI
@ashmeet-kandhari

@jkarneges sorry this should be inEventsIsNotEmpty(inEvents) and we were able to receive PING event
I tried to send outEvents.add(0, new WebSocketEvent("PONG")) immediately but was getting error

2021-04-30 21:52:03,166 - DEBUG - client - state = OPEN
2021-04-30 21:52:23,176 - DEBUG - client > Frame(fin=True, opcode=9, data=b'n\xdd.\xfd', rsv1=False, rsv2=False, rsv3=False)
2021-04-30 21:52:31,156 - DEBUG - client - event = eof_received()
2021-04-30 21:52:31,161 - DEBUG - client ! failing OPEN WebSocket connection with code 1006
2021-04-30 21:52:31,165 - DEBUG - client - event = connection_lost(None)
2021-04-30 21:52:31,170 - DEBUG - client - state = CLOSED
2021-04-30 21:52:31,174 - DEBUG - client x code = 1006, reason = [no reason]
2021-04-30 21:52:31,178 - DEBUG - client - aborted pending ping: 6edd2efd

We are using websockets python [1] library to connect to pushpin websocket which internally sends PING events with random 4 bytes added to it
[1] - https://pypi.org/project/websockets/

Hi @jkarneges ,
Any update on this?

Justin Karneges
@jkarneges
@ashmeet-kandhari condure 1.2.0 should be packaged soon, with the fix. you could check here to see when it becomes available: https://fanout.jfrog.io/artifactory/debian/pool/
Justin Karneges
@jkarneges
@ashmeet-kandhari it's packaged and the pushpin docker image has been updated to use it
ASHMEET KANDHARI
@ashmeet-kandhari
Thanks @jkarneges , will test it and let you know
ASHMEET KANDHARI
@ashmeet-kandhari

Hi @jkarneges
I was able to use the new docker image and need some more help sending bytes as data.
The following were the findings

  • I was able to test ping requests when sending a ping request with custom string message

    2021-05-17 21:44:22,382 - DEBUG - client - state = OPEN
    2021-05-17 21:44:22,385 - DEBUG - client > Frame(fin=True, opcode=9, data=b'Test PING', rsv1=False, rsv2=False, rsv3=False)
    2021-05-17 21:44:22,410 - DEBUG - client - event = data_received(<11 bytes>)
    2021-05-17 21:44:22,414 - DEBUG - client < Frame(fin=True, opcode=10, data=b'Test PING', rsv1=False, rsv2=False, rsv3=False)
    2021-05-17 21:44:22,418 - DEBUG - client - received solicited pong: 546573742050494e47
    2021-05-17 21:44:22,422 - INFO - ponged
  • When the client sends PING automatically with some random bytes as data, for some reason the bytes are getting manipulated when sending the response back. Here are the logs

    2021-05-17 21:44:42,398 - DEBUG - client > Frame(fin=True, opcode=9, data=b'\x1b\x88\xf4\xee', rsv1=False, rsv2=False, rsv3=False)
    2021-05-17 21:44:42,424 - DEBUG - client - event = data_received(<4 bytes>)
    2021-05-17 21:44:42,428 - DEBUG - client < Frame(fin=True, opcode=10, data=b'\x1b\xef', rsv1=False, rsv2=False, rsv3=False)
    2021-05-17 21:44:42,432 - DEBUG - client - received unsolicited pong: 1bef
Justin Karneges
@jkarneges
@/all Pushpin 1.32.1 released, with improvements to the pushpin-publish tool (support for HTTP, auth, GRIP_URL). Also, Rust code has been introduced into the project.
@ashmeet-kandhari I was able to do a ping test with those bytes and get them back intact. maybe an encoding issue in the backend handler?
ASHMEET KANDHARI
@ashmeet-kandhari
@jkarneges yes, there seems to be some issue with java springboot webflux default encoder which is manipulating the response bytes
ASHMEET KANDHARI
@ashmeet-kandhari

Hi @jkarneges,
There seems to be issue with java-gripcontrol lib
I have raised an issue there
fanout/java-gripcontrol#4

PS: I posted it here as author is have fanout.io email

tuantmse90055
@tuantmse90055

After follow the usage of Pushpin, I implement application using WebSocket-over-HTTP protocol and SockJS client, then client only receive message (onmessage function) from response of POST endpoint but cannot receive published WebSocket message. Did I do anything wrong?
POST endpoint:

    @POST
    @Path("/ws-pushpin")
    public Response openConnection(String body) {
        ResponseBuilder resBuilder = Response.ok(body);
        List<WebSocketEvent> inEvents = GripControl.decodeWebSocketEvents(body);
        if (inEvents != null && !inEvents.isEmpty()) {
            if ("OPEN".equals(inEvents.get(0).type)) {
                Map<String, Object> channels = new HashMap<>();
                channels.put("channel", "pushpin-demo");
                List<WebSocketEvent> outEvents = Arrays.asList(
                    new WebSocketEvent("OPEN"),
                    new WebSocketEvent("TEXT", "c:" + GripControl.webSocketControlMessage("subscribe", channels))
                );
                resBuilder.entity(GripControl.encodeWebSocketEvents(outEvents));
            }
        }
        return resBuilder
                .type("application/websocket-events")
                .header("Sec-WebSocket-Extensions", "grip; message-prefix=\"\"")
                .build();
    }

Publish WebSocket message:

        Map<String, Object> entry = new HashMap<>();
        entry.put("control_uri", "http://172.17.0.3:5561");
        List<Map<String, Object>> config = Arrays.asList(entry);
        GripPubControl pub = new GripPubControl(config);
        List<String> channels = Arrays.asList("pushpin-demo");
        List<Format> formats = Arrays.asList(new WebSocketMessageFormat("WebSocket test publish!"));
        try {
            pub.publish(channels, new Item(formats, null, null));
        } catch (PublishFailedException exception) {
            System.out.println("Publish failed: " + exception.getMessage());
        }

Client:

    var sock = new SockJS('/ws-pushpin');

    function send() {
        sock.send('Hello!');
    }

    sock.onopen = function() {
        document.getElementById('sendText').addEventListener("click", send);
    };

    sock.onmessage = function(e) {
        console.log(e);
    };

    sock.onclose = function() {
        document.getElementById('sendText').removeEventListener("click", send);
    };
Justin Karneges
@jkarneges
hi @tuantmse90055 , you say the client is able to receive a message from the response of the POST endpoint, but in your POST handler I don't see any message being sent. do you have a different version of this code where you send a TEXT event containing a regular message?
what do the pushpin logs show?
tuantmse90055
@tuantmse90055

@jkarneges This is logs of pushpin:

[INFO] 2021-06-08 02:51:39.824 [zurl] IN id=ab6bc92f-c515-4f0c-a311-d3def0c604c4, POST http://localhost:7999/ws-pushpin

[INFO] 2021-06-08 02:51:39.826 [zurl] OUT id=ab6bc92f-c515-4f0c-a311-d3def0c604c4 code=200 64

[INFO] 2021-06-08 02:51:39.826 [proxy] GET ws://localhost:7999/ws-pushpin -> 172.17.0.2:8080[http] code=101 0

[INFO] 2021-06-08 02:51:39.831 [handler] subscribe ws://localhost:7999/ws-pushpin channel=pushpin-demo

[INFO] 2021-06-08 02:51:40.743 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:51:40.743 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:51:45.743 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:51:45.743 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:51:50.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:51:50.742 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:51:55.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:51:55.743 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:52:00.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:52:00.742 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:52:05.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:52:05.742 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:52:10.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:52:10.743 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:52:14.399 [zurl] IN id=39d629be-76ce-45cf-84b2-3499f5bd58cd, POST http://localhost:7999/ws-pushpin

[INFO] 2021-06-08 02:52:14.400 [zurl] OUT id=39d629be-76ce-45cf-84b2-3499f5bd58cd code=200 16

[INFO] 2021-06-08 02:52:15.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:52:15.743 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:52:20.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:52:20.742 [handler] publish channel=pushpin-demo receivers=1

[INFO] 2021-06-08 02:52:25.742 [handler] control: POST /publish/ code=200 10 items=1

[INFO] 2021-06-08 02:52:25.742 [handler] publish channel=pushpin-demo receivers=1

When function send() from client is called, POST endpoint receives body with content:

TEXT 6\r\n
Hello!\r\r

Then it responses the same body back and client receives message (onmessage function).
For publishing WebSocket message, I publish sample message "WebSocket test publish!" every 5s at you can see logs above, but client cannot receive any published WebSocker messages.
Periodic publishing WebSocket message:

        vertx.setPeriodic(5000, e -> {
            Map<String, Object> entry = new HashMap<>();
            entry.put("control_uri", "http://172.17.0.3:5561");
            List<Map<String, Object>> config = Arrays.asList(entry);
            GripPubControl pub = new GripPubControl(config);
            List<String> channels = Arrays.asList("pushpin-demo");
            List<Format> formats = Arrays.asList(new WebSocketMessageFormat("WebSocket test publish!"));
            try {
                pub.publish(channels, new Item(formats, null, null));
            } catch (PublishFailedException exception) {
                System.out.println("Publish failed: " + exception.getMessage());
            }
        });
tuantmse90055
@tuantmse90055
@jkarneges sorry, the correct content of body that POST endpoint receives:
TEXT 6\r\n
Hello!\r\n
Justin Karneges
@jkarneges
@tuantmse90055 the receivers=1 means there was a client to deliver to. strange that you aren't receiving it. what if you try without sockjs?
tuantmse90055
@tuantmse90055
@jkarneges without SockJS and using WebSocket API, client receives WebSocket messages normally.
BlockC
@bharath-naik
Hi @jkarneges , i have deployed the pushpin and a small reverse proxy backend server(mbackend1) in GKE with istio(service mesh) and in the routes file of pushpin conatiner, I have given mbackend1-service.default.svc.cluster.local:8000/* ...... while accesing the 7999 port from the browser it says No route for host: 1xx.1xx.1xx.4. can we give service urls ?
Justin Karneges
@jkarneges
@tuantmse90055 it seems the sockjs code got broken recently. this is now fixed in git, will release a new version soon
Justin Karneges
@jkarneges

hi @bharath-naik , "no route for host" means there wasn't a matching route condition in the routes file. maybe you mean for the route to be:

* mbackend1-service.default.svc.cluster.local:8000

?

BlockC
@bharath-naik
yes, not sure but something like this.
BlockC
@bharath-naik
Justin Karneges
@jkarneges
@bharath-naik ok I've answered your specific question on the issue. let's move the conversation over there
tuantmse90055
@tuantmse90055
@jkarneges thank you very much. Also, I don't know whether Pushpin supports STOMP or not. My purpose is using SockJs and STOMP over WebSocket.
Justin Karneges
@jkarneges
@tuantmse90055 Pushpin should be able to support most WebSocket protocols. hopefully you can get it to speak STOMP