Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 25 23:56

    jkarneges on master

    fix parsing of ports without an… (compare)

  • Apr 25 17:57

    jkarneges on master

    Improve port parsing in pushpin… Merge pull request #47707 from … (compare)

  • Mar 19 00:13

    jkarneges on master

    proxy: eat Pushpin-Route header (compare)

  • Mar 11 20:03

    jkarneges on v1.35.0

    (compare)

  • Mar 11 20:03

    jkarneges on master

    update version update dev version (compare)

  • Mar 10 21:15

    jkarneges on master

    be explicit (compare)

  • Mar 10 21:11

    jkarneges on master

    add local_ports option (compare)

  • Mar 09 19:49

    jkarneges on master

    fix for unix sockets with relat… add prometheus_prefix option (compare)

  • Mar 09 02:40

    jkarneges on master

    simplehttpserver: ability to li… statsmanager: ability to listen… (compare)

  • Feb 26 01:55

    jkarneges on master

    ensure pushpin-publish stays ex… (compare)

  • Feb 16 19:57

    jkarneges on master

    add requests received metric (compare)

  • Feb 15 20:01

    jkarneges on master

    only log route if explicitly set (compare)

  • Feb 12 02:38

    jkarneges on master

    move simplehttpserver into core… statsmanager: maintain combined… statsmanager: add prometheus se… and 4 more (compare)

  • Feb 10 18:14

    jkarneges on master

    add no_grip route condition par… (compare)

  • Feb 10 01:17

    jkarneges on master

    ensure each route has an ID, ev… if a route ID is shared and thu… use the route of the initial re… and 1 more (compare)

  • Jan 28 02:49

    jkarneges on routeheader

    (compare)

  • Jan 28 02:48

    jkarneges on master

    update routesfile tests (compare)

  • Jan 28 02:39

    jkarneges on master

    routesfile: move empty initial … use QMultiHash domainmap: add mode to look up … and 3 more (compare)

  • Jan 28 02:38

    jkarneges on routeheader

    routesfile: move empty initial … use QMultiHash domainmap: add mode to look up … and 3 more (compare)

  • Jan 26 16:24

    jkarneges on master

    runner: allow `:0` as a valid l… Merge pull request #47706 from … (compare)

ccbeloy
@ccbeloy
Here's the sample code...
dmu1313
@dmu1313
@jkarneges Okay thank you for the assistance! I don't have to fix it ASAP, so I'll probably just wait until you can get to it. I will add an issue on the github repo so that this investigation can be seen by others in the future, and also if it turns out you can't increase the number of allowed headers, then you can share that on the github issue.
Justin Karneges
@jkarneges
hi @ccbeloy, on the surface the code seems fine. maybe an incompatibility between the implementations. can you log the bytes of the client request somewhere? for example run nc -l {port} and make a non-ssl request to it on localhost. there are only a few ways that error message can happen, and it should be clear from the request what is causing it
ccbeloy
@ccbeloy
Hi @jkarneges - here's the log from netcat
GET /websocket/v1/subscribe HTTP/1.1
Connection: Upgrade
Content-Length: 0
Host: 127.0.0.1:7999
Upgrade: websocket
User-Agent: Java-http-client/11.0.14.1
Sec-WebSocket-Key: 47HJ6V9RCPBd3+motbPKwQ==
Sec-WebSocket-Version: 13
Justin Karneges
@jkarneges
thanks. yeah condure doesn't like that Content-Length: 0, and requires requests to have no body as opposed to having merely an empty body. I'll look into relaxing this
ccbeloy
@ccbeloy
Thanks a lot @jkarneges !
Justin Karneges
@jkarneges
@dmu1313 @ccbeloy pushpin docker image 1.35.0-1 has been posted, which includes the latest condure fixes
2 replies
Victor Kinelev
@vkinelev

Hi @jkarneges. I'm trying to replicate examples from the Quickstart guide for Websockets-Over HTTP.
Initial connect and echo work, but pushpin drops the connection in 8-10 seconds.

I noticed one error message in pushpin logs "[DEBUG] 2022-04-22 13:07:09.981 [proxy] wsproxysession: 0x6000032e8680 wscontrol session error
"

Here's my sinatra backend:

require 'sinatra'

post '/stream' do
  status 200
  data = request.body.read
  if !data.index("OPEN").nil?
    headers(
      'Content-Type' => 'application/websocket-events',
      'Sec-WebSocket-Extensions' => 'grip; message-prefix=""'
    )

    c = 'c:{"type": "subscribe", "channel": "mychannel"}'
    body "OPEN\r\nTEXT 2F\r\nc:#{c}\r\n"
  else
    headers('Content-Type' => 'application/websocket-events')

    body data
  end
end

pushpin version: 1.35.0

I'm connecting with "wscat", but it seems the same result is with curl or normal websocket clients

Any ideas?

Victor Kinelev
@vkinelev

Ok. I found the issue in my code. The grip control message was not correctly formed: I included "c:" two times by mistake.

Btw, pushpin logs pointed me to the right direction with this log entry:

[DEBUG] 2022-04-22 16:23:50.306 [handler] grip control message is not valid json

I may need look into gripcontrol gem again. That should help avoid such silly mistakes going forward.

RadiumBlock
@radiumblock:matrix.org
[m]
@jkarneges: Hello, We are trying to setup pushpin and would like to know if it is possible for pushpin stats socket to report websocket message count.
Justin Karneges
@jkarneges
@radiumblock:matrix.org the stats socket sends a message event which includes a count and the transport used. there's also the report event which includes an aggregate count of messages across all transports
1 reply
we also recently introduced prometheus metrics. it has to be explicitly enabled via the prometheus_port setting (see https://pushpin.org/docs/configuration/#pushpinconf-file). it provides aggregate values similar to the report event
RadiumBlock
@radiumblock:matrix.org
[m]
So we have been tryin gto use monitorstats.py from the /tools/ to get ws message reporting.
python3 monitorstats.py ipc:///var/run/pushpin/pushpin-stats report
report {'connections': 1, 'duration': 8501, 'from': 'pushpin-handler_90019', 'http-response-sent': 0, 'minutes': 1, 'received': 0, 'sent': 0}
but it doesnt look like ws message count is being reported
also tried python3 monitorstats.py ipc:///var/run/pushpin/pushpin-stats message but it does not seem to be triggering anything even when we connect to ws via pushping using wscat
@jkarneges: should we be listening to ipc:///var/run/pushpin/pushpin-proxy-stats ?
Justin Karneges
@jkarneges
@radiumblock:matrix.org are you trying to measure messages pushpin sends to the client? or messages the client sends to pushpin?
RadiumBlock
@radiumblock:matrix.org
[m]
Ideally both. I could parse the log but that seems slow and inefficient at high traffic
Justin Karneges
@jkarneges
at the moment, only messages sent from pushpin to the client are counted. that would be the sent field in the reports
RadiumBlock
@radiumblock:matrix.org
[m]
I tried that, however that keeps showing 0 even though the ws connection is being proxied and responses received correctly
Note we are doing a WS to ws backend proxy
Justin Karneges
@jkarneges
ah. only published messages are counted, not messages sent via the proxied connection
RadiumBlock
@radiumblock:matrix.org
[m]
I was afraid of that, would it be too far of a leap to try to add that? I.e. track down the code in proxy that counts the ws messages and use that to increment the report stats as well as latency.
Or are aware of any plans to add such a feature in the near future?
Justin Karneges
@jkarneges
counting non-published messages is a reasonable idea. maybe it's something we could implement in the next month or so
what do you mean about a latency metric?
RadiumBlock
@radiumblock:matrix.org
[m]
Time delta between client request and backend response in a sample period
Getting a metric counting non published messages would be awesome.
Matt Jarvis
@msivraj_gitlab
Has something changed with in the pushpin code to prevent the pushpin proxy from forwarding on client websocket messages to the backend? I had it all set up and working yesterday but today it appears like the pushpin proxy is not receiving messages form the websocket client. There are no errors thrown in either the pushpin logs or my personal logs . Opening up the websocket connection with pushpin works just fine and it proxies this connection to the backend which accepts the connection, but the websocket client cannot send messages to the server behind to pushpin proxy afterwards. Not sure what happened.
Justin Karneges
@jkarneges
hi @msivraj_gitlab , there is no such known issue at least. there should be a zurl log line for every HTTP request and response with the backend. if you aren't seeing that, you could try running with verbose logging and share your logs (DM ok)
Matt Jarvis
@msivraj_gitlab
Thx for getting back to me so quickly. I figured out the that the issue is with my company's firewall blocking the ws and wss protocols. Thanks for your suggestions.
RadiumBlock
@radiumblock:matrix.org
[m]
@jkarneges: from time to time, the proxy seems to fail to start. systemctl gives the following message "Failed to start HTTP reverse proxy server for streaming and long-polling services"
stop and restart of the service does not seem to clear things.. is there some sort of lock? thanks
RadiumBlock
@radiumblock:matrix.org
[m]
Never mind resolved this.
RadiumBlock
@radiumblock:matrix.org
[m]
@jkarneges: just wanted to say that we would be willing to beta test a patch to count non-published websocket request and response messages.
RadiumBlock
@radiumblock:matrix.org
[m]
Is it reasonable to expect to run the server at debug level 3 for long term provided the logs are rotated often?
Justin Karneges
@jkarneges
@radiumblock:matrix.org it's a lot of noise and will use more cpu. I wouldn't run a highly active server that way. but I suppose there's no problem doing it
RadiumBlock
@radiumblock:matrix.org
[m]
We tried that and we are finding that the proxy stops working in the debug mode after an hour or so.
We tried enable level 3 log only for the proxy (others at 1) and it still faces issues. It looks like it stops accepting connection
Justin Karneges
@jkarneges
that is very surprising. when connections seem to stop getting accepted, does the proxy log anything when a connection attempt is made?
RadiumBlock
@radiumblock:matrix.org
[m]
@jkarneges: as far as i can tell its the log has stopped. when a the new(failed) connection it attempted.
interestingly the update checker continues to log
RadiumBlock
@radiumblock:matrix.org
[m]
:point_up: Edit: @jkarneges: as far as i can tell, it has stopped logging. There is no new log when a the new(failed) connection it attempted.
wscat -c ws://localhost:8000 also hangs
Justin Karneges
@jkarneges
@radiumblock:matrix.org what if you increase condure's log level to 4? I wonder if anything shows up there when trying to connect after the proxy stops logging
supos7
@supos7
Hi, does pushpin work on AWS arm based server? There is no docker image for linux/arm64 and I got 502 Bad Gateway error after I built it myself.