Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Feb 24 19:28
    dcgloe commented #1510
  • Feb 23 11:55
    zhanggyarcher commented #1505
  • Feb 22 10:05
    minrk commented #1505
  • Feb 22 09:10
    minrk commented #1510
  • Feb 19 20:43
    dcgloe opened #1510
  • Feb 19 08:40
    minrk closed #1025
  • Feb 19 08:39
    minrk closed #1077
  • Feb 19 08:36
    minrk closed #1098
  • Feb 19 08:35
    minrk commented #1098
  • Feb 19 08:35
    minrk closed #1100
  • Feb 19 08:32
    minrk closed #887
  • Feb 19 08:28
    minrk closed #161
  • Feb 19 08:28
    minrk commented #161
  • Feb 19 08:27
    minrk closed #1231
  • Feb 18 15:26
    zhanggyarcher commented #1505
  • Feb 18 11:42
    minrk closed #1250
  • Feb 18 11:42
    minrk commented #1250
  • Feb 18 11:41
    minrk closed #1283
  • Feb 18 11:39
    minrk commented #1324
  • Feb 18 11:38
    minrk closed #1324
Min RK
@minrk
ROUTER messages are all prefixed with an identity, which lets you identify the sender and route replies. You'll probably want ROUTER sockets for both directions in the scheduler, and DEALER sockets on the workers and clients. Workers can send an "I'm here" message when they connect, which the scheduler can then store along with the identity that came with the message.
François Tessier
@hephtaicie
Hmm... Interesting. Why not |CLIENTS|(REQ) <-> (ROUTER)|SCHEDULER|(ROUTER) <-> (REQ)|SERVER|?
Min RK
@minrk
why DEALER instead of REQ, you mean?
François Tessier
@hephtaicie
Yes
Min RK
@minrk
DEALER and REQ are really the same, but REQ adds unnecessary enforcement of send/recv/send/recv sequencing. The only thing it accomplishes is making it impossible to have messages without reply, retrying sends, etc.
REQ/REP are ~never right for production
François Tessier
@hephtaicie
I see! Thanks for your help. That's very helpful. I'm gonna try with DEALER <-> ROUTER + ROUTER <-> DEALER and see how it goes. The REQ/REP sequencing was making my code really heavy...
Min RK
@minrk
François Tessier
@hephtaicie
@minrk Thanks a lot!
Just a couple of questions to be sure I understand everything correctly:
1) In recv_msg, you use identities implying multiple identities for a single message. Is that correct?
François Tessier
@hephtaicie
2) What's the difference between client_socket.send_multipart([client_identity, encode_msg(msg)]) (L98) and client_socket.send_multipart(identities + [encode_msg(msg)]) (L122)?
Min RK
@minrk
1-many, yes
In particular, I kept the client identity in the worker message, meaning that worker replies are [worker identity, client identity, msg_content] (it would also have been sensible to add it to the msg body, but then it needs to be base64-encoded or something)
sending a reply with identities means "go back the way you came, no matter how many identity prefixes were stacked up"
whereas forwarding reply from worker to client requires manipulation, specifically removing the worker identity from the front
François Tessier
@hephtaicie
I see. So it also answers 2): you just merge two lists.
Min RK
@minrk
Yes, the difference is one is sending a message back the way it came, and the other is relaying it on a different socket
for any given socket, only the first identity is used for routing, the rest is opaque 'content' to be used at the application level
François Tessier
@hephtaicie
So what is returned by recv_multipart is something like [[identity_1, identity_2, ... , identity_n], empty, message], right?
Min RK
@minrk
recv_multipart returns a list of bytes, e.g. [identity1, msg] or just [msg] depending on which socket is receiving
in that example, there are 1-, 2-, and 3- part messages
(client recvs one-part, scheduler recvs two-part from client, three-part from worker, worker recvs two-part from scheduler)
François Tessier
@hephtaicie
Oh ok. And the last part is always msg I suppose
Min RK
@minrk
yes
François Tessier
@hephtaicie
@minrk Thanks again! I worked actively on my project those days and your example and our discussion here have been very helpful.
Fogang Fokoa
@fokoa
Hello
I'am installing jupyter notebook and my installation stop when pyzmq-21.0.1 want to install.
I get this error :
Downloading https://files.pythonhosted.org/packages/29/84/4b0e1f95a9305861507e7d60bd2368c8939766335d716b7cb05d0200a5ed/pyzmq-21.0.1.tar.gz (1.2MB) 100% |████████████████████████████████| 1.2MB 420kB/s Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-btceje2b/pyzmq/setup.py", line 687 warn(f"platform={get_platform()}, vcvars=") ^ SyntaxError: invalid syntax
And when I take a version <21.0.1, pip automatically try to install 21.0.1.
I need help. Thank you!
Min RK
@minrk
@fokoa what platform, Python version, and pip version?
That syntax error suggests you are on an unsupported version of Python, which in turn suggests you are using a pip that's too old to respect the python version requirement metadata
Fogang Fokoa
@fokoa
@minrk I use ubuntu 16.04, Python3(3.5.2) so pip3 (version 8.1.1).
Min RK
@minrk
pyzmq 21 requires Python 3.6. You need pip >=9.0 to implement Python version requirements. It's always a good idea to start with: python3 -m pip install --upgrade pip to make sure you have pip up-to-date.
If you had a relatively recent pip (since 2016), it would get the latest pyzmq to support your Python (20.0)
Fogang Fokoa
@fokoa
I did it and version 20.3.3 install but when I type pip3 -V, I get version 8.1.1 yet.
Min RK
@minrk
try python3 -m pip
Fogang Fokoa
@fokoa
python3 -m pip send me into utility. and when I type python3 -m pip install i get this :
You must give at least one requirement to install (see "pip help install") You are using pip version 8.1.1, however version 20.3.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
Okay. I think I see the problem. When I type this sudo -H pip3 install --upgrade pip and I get :
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
I will set pip3 to run via python3.7.
Min RK
@minrk
Note that that message is about a deprecation, not an error. But yes, since 3.5 is EOL, you do need to be careful.
Fogang Fokoa
@fokoa
Okay.
Min RK
@minrk
it's tricky because you need pip new enough to support the Requires-Python metadata, which it has for ~5 years. However, if your pip is still that old, pip install --upgrade pip can jump to a version of pip that doesn't support your Python. python3 -mpip install --upgrade 'pip<21' will ensure you get a version of pip that still supports Python 3.5
Johan Mabille
@JohanMabille
Hi there, I have some question regarding the usage of pyzqm
I have a TCP socket connected to a server that can send events
so I use a ZMQStream (which uses the tornado ioloop in ipykernel to be more precise)
the issue is that I also need to wait for a specific response on this socket after I sent a request
and ZMQStream does not provide the blocking recv method in its API
the only mean I see to achieve this is to keep a reference on the underlying socket, unregister the callback in the zmqstream when I need blocking calls to receive, and register the callback again when I have received the answer
Johan Mabille
@JohanMabille
but this does not seem super clean
(and a bit cumbersome to implement)
Johan Mabille
@JohanMabille
is there a better way to achieve what I need?
Allie
@cite-reader
There's a canonical answer, but you probably won't like it.