These are chat archives for fanout/pushpin

20th
Mar 2017
fruch
@fruch
Mar 20 2017 07:16
@jkarneges thanks, can you show some examples of using the SUB input socket
I was trying to use the python gripcontrol package, but it seem a bit broken in that aspect
fruch
@fruch
Mar 20 2017 07:24
I'm trying to use pypubcontrol directly, but with no luck
fruch
@fruch
Mar 20 2017 07:43
here is the example i'm builing: https://github.com/fruch/compose-pushpin
the "require_subscribers: True" isn't working for me
and I couldn't figure out why...
Vaskevich Aleksander
@folt
Mar 20 2017 09:19

Using the code from the example:
@websocket_only
def echo(request):
ws = request.wscontext

if ws.is_opening():
    ws.accept()
    ws.subscribe('mychannel')

while ws.can_recv():
    message = ws.recv()

    if message is None:
        ws.close()
        break
    ws.send(message)

I make a request like this:
curl -d 'OPEN'$'\r'$'\n' 'TEXT 2F'$'\r'$'\n' -H 'Accept: application/websocket-events' http://localhost:8000/echo/

I get the answer:
OPEN
TEXT 38
c:{"type": "subscribe", "channel": "headline-mychannel"}

How can I make the terminal receive messages and do not interrupt the connection to the socket?

Justin Karneges
@jkarneges
Mar 20 2017 17:13
@fruch in the pushpin source repo, in the tools folder, have a look at publish2.py and monitorsubsock.py
if those work, then hopefully we can work backwards to figure out why pypubcontrol isn't working
Justin Karneges
@jkarneges
Mar 20 2017 17:21
@folt i'm not quite sure what you're asking, but if you want to make a real websocket connection that will stay open, then you need to open a websocket to pushpin. using curl to make http requests to the backend is good for testing the integration but won't result in a connection that stays open for message receipt
for example you could install wscat, and then do wscat -c ws://localhost:7999/echo/
Vaskevich Aleksander
@folt
Mar 20 2017 18:08
@jkarneges From the terminal comes not a string but bytes
Justin Karneges
@jkarneges
Mar 20 2017 18:54
@folt hmm, I still don't understand the problem. what are you expecting?
fruch
@fruch
Mar 20 2017 18:55
@jkarneges I found my issue, I was using python3
and the tnetstring3 seem to be a bit broken in my case
it was silently failing I guess
Vaskevich Aleksander
@folt
Mar 20 2017 18:56
@jkarneges I'm using python 3
Due to the fact that in the terminal curl returns bytes and not a string, the socket event is not valid
fruch
@fruch
Mar 20 2017 18:56
trying the sub2.py from the repo, I got it working
now I've added consul to my example, so I can pub/sub to all the nodes
Justin Karneges
@jkarneges
Mar 20 2017 18:58
@fruch does this mean you had to use python 2 to publish?
@folt the server response looks correct to me. although your request looks strange. TEXT 2F\r\n with nothing after it is bogus
Vaskevich Aleksander
@folt
Mar 20 2017 19:04
Now I'm working on a patch for the python library 3
I already did today pull request
fruch
@fruch
Mar 20 2017 19:37
@jkarneges yes, when I move to python2, thing started to work as I expected
I think I'll have a complete docker-compose setup to demoenstrate scaling of my use case
I would be happy to share it
Justin Karneges
@jkarneges
Mar 20 2017 19:40
@fruch that'd be great :)
fruch
@fruch
Mar 20 2017 19:40
I'm trying to gather enough numbers for my idea
we have a system that works with xmpp (on jabberd), which we find is not very reliable
there are case that message get lost, and not send to clients
or client that gets disconnected a lot
our initial soultion was based on longpoll, and we were working a lot to squeeze it via existing xmpp we had in place.
Justin Karneges
@jkarneges
Mar 20 2017 19:44
makes sense. in my experience, xmpp is a lot of complexity if you're translating for browser use, and also very stateful
with pushpin you can be lean and stateless and it's easier to manage
fruch
@fruch
Mar 20 2017 19:44
it's not even a browser...
Justin Karneges
@jkarneges
Mar 20 2017 19:45
well then :)
Justin Karneges
@jkarneges
Mar 20 2017 20:25
@fruch very cool. do you only produce the stack or do you also work with deployments?
fruch
@fruch
Mar 20 2017 20:25
some of it all, we try to shift to devops, as much as we can
question, why would zmq pub/sub would be better then sending a publish to all pushpin nodes ?
Justin Karneges
@jkarneges
Mar 20 2017 20:28
zmq does publisher-side filtering. so when you write to a PUB socket, if there's no subscriber then the message doesn't even leave the machine
the way this works under the hood is the SUB socket sends notifications to the PUB socket, and libzmq tracks the list of channels in memory. but this is all behind the scenes zmq magic
fruch
@fruch
Mar 20 2017 20:29
hmm, sound cools
but still doesn't work for me exactly
the 'control_zmq_uri'
I got it to work with zmq directly
but with python gripcontrol
not so much
I think my problem is that the docker image publish the tcp://*:5562 address
and that no so useful in my case, when run inside docker
Justin Karneges
@jkarneges
Mar 20 2017 20:31
ah yes, the SUB socket endpoint is discovered
you'll need to make push_in_sub_spec specify the external interface
so that it'll work if used by a remote machine
fruch
@fruch
Mar 20 2017 20:33
hmm, this is kind of a chicken and an egg this...
*thing
maybe we could do a trick and replace the address with address I give at
control_zmq_uri
Justin Karneges
@jkarneges
Mar 20 2017 20:34
i think you can also bypass the discovery by setting zmq_pub_uri.
fruch
@fruch
Mar 20 2017 20:34
I think not in gripcontol, only in pubcontol
Justin Karneges
@jkarneges
Mar 20 2017 20:38
so GripPubControl is a convenience subclass of PubControl and it doesn't do a whole lot
and you could call add_client() on it directly with pre-constructed client classes
fruch
@fruch
Mar 20 2017 20:42
publish still work with PubControl
*still doesn't work
I think this flag here should be True, it's disabling the sub/pub
this line should change to
require_subscribers, not require_subscribers, None, self._zmq_ctx,
and now it's working...
now I need to think of my test
fruch
@fruch
Mar 20 2017 20:49
I love being PR#1 on repos...
fanout/pypubcontrol#1
look like the discovery was the major way people would use the library...
fruch
@fruch
Mar 20 2017 22:02
I'm seeing very strange things with overloading pushpin with longpoll requests
I'm getting lots of 502
much more then I expected

max messages per second

message_rate=2500

max rate-limited messages

message_hwm=25000

is those has anything todo with it ?
also memory keep raising, and doesn't seem to be lowering, after there's no connections
Justin Karneges
@jkarneges
Mar 20 2017 22:12
ah whoops, i forgot add_client() may not actually work in this case, because PUB is managed by the pubcontrol container and not the client
Vaskevich Aleksander
@folt
Mar 20 2017 22:13
@fruch
1) you can change the settings
2) you can raise more images
Justin Karneges
@jkarneges
Mar 20 2017 22:14
normally when you call publish on PubControl, it iterates all the clients and publishes to each. however there is a special case with PUB sockets, whereby a single socket is used and we let libzmq do the sending. this is why all the clients have disable_pub=True
Vaskevich Aleksander
@folt
Mar 20 2017 22:18
@jkarneges
fanout/django-grip#5
I ask you to see the patch
Justin Karneges
@jkarneges
Mar 20 2017 22:21
@fruch those settings are for publishing and shouldn't be related to 502 errors which would come from proxying. in the logs you might see where the 502 is coming from, e.g. from the upstream server, or if it's a failure from one of pushpin's processes
also make sure your sysctl / limits stuff is set up well
fruch
@fruch
Mar 20 2017 22:22
hmmm, so I'm loading my upstream server...
could be, I just wrote off something in bottle...
after giving some more cpu to the nodes
it's working a bit better
I need to put some numbers into my test
Justin Karneges
@jkarneges
Mar 20 2017 22:24
thanks @folt i will definitely look
fruch
@fruch
Mar 20 2017 22:27
@jkarneges I would love to hear thoughts and ideas on how to make this example better: https://github.com/fruch/compose-pushpin