Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Feb 24 19:28
    dcgloe commented #1510
  • Feb 23 11:55
    zhanggyarcher commented #1505
  • Feb 22 10:05
    minrk commented #1505
  • Feb 22 09:10
    minrk commented #1510
  • Feb 19 20:43
    dcgloe opened #1510
  • Feb 19 08:40
    minrk closed #1025
  • Feb 19 08:39
    minrk closed #1077
  • Feb 19 08:36
    minrk closed #1098
  • Feb 19 08:35
    minrk commented #1098
  • Feb 19 08:35
    minrk closed #1100
  • Feb 19 08:32
    minrk closed #887
  • Feb 19 08:28
    minrk closed #161
  • Feb 19 08:28
    minrk commented #161
  • Feb 19 08:27
    minrk closed #1231
  • Feb 18 15:26
    zhanggyarcher commented #1505
  • Feb 18 11:42
    minrk closed #1250
  • Feb 18 11:42
    minrk commented #1250
  • Feb 18 11:41
    minrk closed #1283
  • Feb 18 11:39
    minrk commented #1324
  • Feb 18 11:38
    minrk closed #1324
Min RK
@minrk
yes, or use a poller
Yassine Benyahia
@Wronskia
would something like this work
def run(self, request_data):
        self.socket.setsockopt(zmq.RCVTIMEO, self.timeout)
        self.send(json.dumps(request_data))
        try:
            result = self.recv_array(self.socket)
        except zmq.error.Again as e:
            # log e
            raise TimeoutError("timeout")

        return result
Min RK
@minrk
I expect so, yes
François Tessier
@hephtaicie
Hello everyone! I'm trying to build a simple scheduler using zmq (Python). On one side, there are multiple clients that submit requests to the scheduler (a single app). On the other side, there are servers that register to the scheduler and offer a list of available resources. When a client submit a request, the scheduler will determine the resources that will be allocated to this client. Then, this scheduler will reach the corresponding server to let it know that this set of resources is now allocated by a client. The server returns info about the resources to the scheduler which forwards it to the calling client.
I can't figure out how to identify who is who. For instance, I took a look at the "extended request-reply" example using ROUTER and DEALER but I can't see which client sends which request and which worker replies to it.
François Tessier
@hephtaicie
Going though identities. That could answer my question.
Min RK
@minrk
ROUTER messages are all prefixed with an identity, which lets you identify the sender and route replies. You'll probably want ROUTER sockets for both directions in the scheduler, and DEALER sockets on the workers and clients. Workers can send an "I'm here" message when they connect, which the scheduler can then store along with the identity that came with the message.
François Tessier
@hephtaicie
Hmm... Interesting. Why not |CLIENTS|(REQ) <-> (ROUTER)|SCHEDULER|(ROUTER) <-> (REQ)|SERVER|?
Min RK
@minrk
why DEALER instead of REQ, you mean?
François Tessier
@hephtaicie
Yes
Min RK
@minrk
DEALER and REQ are really the same, but REQ adds unnecessary enforcement of send/recv/send/recv sequencing. The only thing it accomplishes is making it impossible to have messages without reply, retrying sends, etc.
REQ/REP are ~never right for production
François Tessier
@hephtaicie
I see! Thanks for your help. That's very helpful. I'm gonna try with DEALER <-> ROUTER + ROUTER <-> DEALER and see how it goes. The REQ/REP sequencing was making my code really heavy...
Min RK
@minrk
François Tessier
@hephtaicie
@minrk Thanks a lot!
Just a couple of questions to be sure I understand everything correctly:
1) In recv_msg, you use identities implying multiple identities for a single message. Is that correct?
François Tessier
@hephtaicie
2) What's the difference between client_socket.send_multipart([client_identity, encode_msg(msg)]) (L98) and client_socket.send_multipart(identities + [encode_msg(msg)]) (L122)?
Min RK
@minrk
1-many, yes
In particular, I kept the client identity in the worker message, meaning that worker replies are [worker identity, client identity, msg_content] (it would also have been sensible to add it to the msg body, but then it needs to be base64-encoded or something)
sending a reply with identities means "go back the way you came, no matter how many identity prefixes were stacked up"
whereas forwarding reply from worker to client requires manipulation, specifically removing the worker identity from the front
François Tessier
@hephtaicie
I see. So it also answers 2): you just merge two lists.
Min RK
@minrk
Yes, the difference is one is sending a message back the way it came, and the other is relaying it on a different socket
for any given socket, only the first identity is used for routing, the rest is opaque 'content' to be used at the application level
François Tessier
@hephtaicie
So what is returned by recv_multipart is something like [[identity_1, identity_2, ... , identity_n], empty, message], right?
Min RK
@minrk
recv_multipart returns a list of bytes, e.g. [identity1, msg] or just [msg] depending on which socket is receiving
in that example, there are 1-, 2-, and 3- part messages
(client recvs one-part, scheduler recvs two-part from client, three-part from worker, worker recvs two-part from scheduler)
François Tessier
@hephtaicie
Oh ok. And the last part is always msg I suppose
Min RK
@minrk
yes
François Tessier
@hephtaicie
@minrk Thanks again! I worked actively on my project those days and your example and our discussion here have been very helpful.
Fogang Fokoa
@fokoa
Hello
I'am installing jupyter notebook and my installation stop when pyzmq-21.0.1 want to install.
I get this error :
Downloading https://files.pythonhosted.org/packages/29/84/4b0e1f95a9305861507e7d60bd2368c8939766335d716b7cb05d0200a5ed/pyzmq-21.0.1.tar.gz (1.2MB) 100% |████████████████████████████████| 1.2MB 420kB/s Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-btceje2b/pyzmq/setup.py", line 687 warn(f"platform={get_platform()}, vcvars=") ^ SyntaxError: invalid syntax
And when I take a version <21.0.1, pip automatically try to install 21.0.1.
I need help. Thank you!
Min RK
@minrk
@fokoa what platform, Python version, and pip version?
That syntax error suggests you are on an unsupported version of Python, which in turn suggests you are using a pip that's too old to respect the python version requirement metadata
Fogang Fokoa
@fokoa
@minrk I use ubuntu 16.04, Python3(3.5.2) so pip3 (version 8.1.1).
Min RK
@minrk
pyzmq 21 requires Python 3.6. You need pip >=9.0 to implement Python version requirements. It's always a good idea to start with: python3 -m pip install --upgrade pip to make sure you have pip up-to-date.
If you had a relatively recent pip (since 2016), it would get the latest pyzmq to support your Python (20.0)
Fogang Fokoa
@fokoa
I did it and version 20.3.3 install but when I type pip3 -V, I get version 8.1.1 yet.
Min RK
@minrk
try python3 -m pip
Fogang Fokoa
@fokoa
python3 -m pip send me into utility. and when I type python3 -m pip install i get this :
You must give at least one requirement to install (see "pip help install") You are using pip version 8.1.1, however version 20.3.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
Okay. I think I see the problem. When I type this sudo -H pip3 install --upgrade pip and I get :
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
I will set pip3 to run via python3.7.
Min RK
@minrk
Note that that message is about a deprecation, not an error. But yes, since 3.5 is EOL, you do need to be careful.
Fogang Fokoa
@fokoa
Okay.
Min RK
@minrk
it's tricky because you need pip new enough to support the Requires-Python metadata, which it has for ~5 years. However, if your pip is still that old, pip install --upgrade pip can jump to a version of pip that doesn't support your Python. python3 -mpip install --upgrade 'pip<21' will ensure you get a version of pip that still supports Python 3.5
Johan Mabille
@JohanMabille
Hi there, I have some question regarding the usage of pyzqm
I have a TCP socket connected to a server that can send events
so I use a ZMQStream (which uses the tornado ioloop in ipykernel to be more precise)