Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 04 2021 12:02
    minrk synchronize #1577
  • Aug 04 2021 12:00
    minrk edited #1556
  • Aug 04 2021 11:08
    minrk opened #1577
  • Aug 03 2021 13:48
    minrk edited #1572
  • Aug 03 2021 13:47

    minrk on move-wepoll

    (compare)

  • Aug 03 2021 13:47

    minrk on main

    bundle wepoll the same way we d… try testing wheels for ipc supp… Merge pull request #1576 from z… (compare)

  • Aug 03 2021 13:47
    minrk closed #1576
  • Aug 03 2021 13:29

    minrk on main

    run pre-commit on pre-commit.ci… Merge pull request #1575 from m… (compare)

  • Aug 03 2021 13:29
    minrk closed #1575
  • Aug 03 2021 13:29
    minrk opened #1576
  • Aug 03 2021 13:28

    minrk on move-wepoll

    bundle wepoll the same way we d… try testing wheels for ipc supp… (compare)

  • Aug 03 2021 13:22
    minrk opened #1575
  • Aug 03 2021 12:49

    minrk on main

    Updating bundle.py to use v142 … Merge pull request #1574 from m… (compare)

  • Aug 03 2021 12:49
    minrk closed #1574
  • Aug 03 2021 12:49
    minrk commented #1574
  • Aug 03 2021 12:08
    AntiTenzor commented #1570
  • Aug 03 2021 12:02
    mattrjackson commented #1574
  • Aug 03 2021 11:58
    mattrjackson commented #1574
  • Aug 03 2021 11:35
    mattrjackson opened #1574
  • Aug 03 2021 07:54
    minrk commented #1563
Cassiel-girl
@Cassiel-girl
https://zguide.zeromq.org/docs/chapter3/
In the code example “rtreq: ROUTER-to-REQ in C”, why does the broker call the statement "s_sendmore(broker, identity)" to construct <identity,empty delimiter,data>? Doesn't the req socket only need empty delimiter?
Cassiel-girl
@Cassiel-girl
Why do we need zpoller when we already have zloop? What are the advantages, disadvantages and applicability of both?
Min RK
@minrk
Those are czmq APIs, not pyzmq, though pyzmq has equivalent features in e.g. asyncio integration vs zmq.Poller. But zloop gives you an event-driven API ("when there is a message on this socket, call this function"), while zpoller is for waiting for events ("block here until there's an event on a socket"). poll is a lower-level construct - you can implement zloop on top of zpoller, but since zloop exists, you don't have to.
Cassiel-girl
@Cassiel-girl
thanks
Endogen
@Endogen
hey guys, i'd like to get a sub/pub going with encryption. is that possible? i saw the examples in here: https://github.com/zeromq/pyzmq/tree/main/examples/security but none of them is for PUB SUB
Endogen
@Endogen

i think what i need is CurveZMQ and since i want to use it with pub/sub i'd need it in the transport layer i guess.

To secure a single hop between client and server, which is the CurveCP use case. For this use case we would embed CurveZMQ in the transport layer so that it can work for all patterns (publish-subscribe, pipeline, and so on).

how do i embed that lib in my python application?
Fabian Beitler
@swamper123
Hey ho together,
I am pretty new to zmq and I try to get into the topic. To make some comparison between zmq and other frameworks, I want to make a simple "ping-pong" thing for IPCs (just send a string "ping" and recv. "pong").
I figured out to built a REQ-REP version, but I am struggeling with a DEALER-ROUTER example.
Intuitive, I would replace REQ-REQ with DEALER-ROUTER and await send/recv_pyobj messages....but this doesn't seem to work. :/
Allie
@cite-reader
Are you handling the message envelopes in your DEALER-ROUTER code?
Fabian Beitler
@swamper123
Stupid question:
If I send a msg from DEALER to ROUTER, do I receive a list like [identifier, message]?
And if this is true, how should an answer from ROUTER back to DEALER look like (in an async context)?
Fabian Beitler
@swamper123
Ok, I am starting to understand how this whole stuff works. ^^
Indeed, since the ROUTER is a N-1 connection, it needs to know to whom it has to send what....
CPickens42
@CPickens42
Hello guys! I'm looking for a low-latency IPC mechanism right now. ZeroMQ on the top of my list. However, I was wondering... Is it possible to setup an inproc connection using the multiprocessing module in Python?
Since Python threading are not real threads, I was thinking about doing some kind of multiprocessing instead.
import threading
from multiprocessing import Process
import zmq

def step1(context=None):
    """Step 1"""
    context = context or zmq.Context.instance()
    # Signal downstream to step 2
    sender = context.socket(zmq.PAIR)
    sender.connect("inproc://step2")

    sender.send(b"from step1")

def step2(context=None):
    """Step 2"""
    context = context or zmq.Context.instance()
    # Bind to inproc: endpoint, then start upstream thread
    receiver = context.socket(zmq.PAIR)
    receiver.bind("inproc://step2")

    #thread = threading.Thread(target=step1)
    thread = Process(target=step1, args=(context,))
    thread.start()

    # Wait for signal
    msg = receiver.recv()
    print("%s" % msg)

    # Signal downstream to step 3
    sender = context.socket(zmq.PAIR)
    sender.connect("inproc://step3")
    sender.send(b"from step2")

def main():
    """ server routine """
    # Prepare our context and sockets
    context = zmq.Context.instance()

    # Bind to inproc: endpoint, then start upstream thread
    receiver = context.socket(zmq.PAIR)
    receiver.bind("inproc://step3")

    #thread = threading.Thread(target=step2)
    thread = Process(target=step2, args=(context,))
    thread.start()

    # Wait for signal
    msg = receiver.recv()
    print("%s" % msg)

    print("Test successful!")

    receiver.close()
    context.term()

if __name__ == "__main__":
    main()

So far this example (changed with multiprocessing in mind) fails. I took, and changed it from here: https://github.com/companycy/cpp_by_example/blob/master/inproc_example.py

It gives this result:

Assertion failed: ok (src/mailbox.cpp:99)
Assertion failed: ok (src/mailbox.cpp:99)
[1]    3015255 abort (core dumped)  python test-pymq.py
Fabian Beitler
@swamper123
Hey ho,
I know that a Project exists which implemented an IPC socket communication in the python zeromq lib. I already tested it and worked pretty well. If I find the link, i will let you know :)
Fabian Beitler
@swamper123
Keep in mind that Python is not the fastest language, if you really need low latency.
CPickens42
@CPickens42
I would love too! Yes I know that Python is not the fastest language for that. But all our stack is in Python soooo :)
I think I just need to find a "fast enough solution"... And if I don't find it, I will try others alternatives in others languages. But it will surely add more complexity to the rest of our code
Fabian Beitler
@swamper123
yeah, zmq via unix socket is pretty fast :) Hope your os is unix compatibel?
CPickens42
@CPickens42
I'm on linux :p
Fabian Beitler
@swamper123
@CPickens42 Meh, can't find the project atm, haven't starred it. Just found this example in the normal repo, but that wasn't what I was looking for:
https://github.com/zeromq/pyzmq/blob/29aded1bf017385866dcbf7b92a954f272360060/examples/gevent/simple.py
Min RK
@minrk

inproc connection using the multiprocessing module in Python?

inproc is in-process, it can't be used across processes, which includes forks created by multiprocessing. Contexts cannot be passed across forks, either. You'll need to use the ipc or tcp transports for inter-process communication, and start a Context in each process.

CPickens42
@CPickens42
Okay, very clear ! Thank you for help guys
Cassiel-girl
@Cassiel-girl
Can zmq.Context() be called multiple times in the same process context?
3 replies
Cassiel-girl
@Cassiel-girl
How does zeromq enable reliable transmission from one sender to multiple receivers. pub/sub sockets seem to have packet loss issues, right?
Cassiel-girl
@Cassiel-girl

from zmq import Socket
mycontext = zmq.Context()
class MySocket(Socket):

#how to write this class?

sock = MySocket()
sock.xx()

how to implement MySocket to use like socket defined in zmq.

Min RK
@minrk
'relaible messages' are more of an application-level feature
libzmq handles low-level disruptions like packet loss, so you won't ever get incomplete messages
The guilde describes some patterns for reliability.
But this gist is usually something along the lines of
  1. store history (or state, depending on the nature of messages) on the sender, or somewhere else, and
  2. use counters in messages to enable receivers to see when they've missed something, and
  3. enable out-of-band bulk requests, so receivers can ask for what they've missed to catch up
Min RK
@minrk
exactly what and how much to make available for these 'catch-up' or 'start subscribing' requests will depend on the nature of your application. A generic version might be to keep the last N messages in an always-expiring buffer, and allow a request for retrieving some/all of those messages by counter.
The clone pattern in the guide is used for e.g. state reflection, where a 'resubscribe' is getting the whole state as a starting point.
Cassiel-girl
@Cassiel-girl
thank you
divflex
@divflex
If I use send_multipart() to send ['X', 'data'], ['Y', 'data'], ['Z', 'data']
Use recv_multipart(), the result will be ['X', 'Y', 'Z', 'data']
How can I recv ['X', 'data'], not ['X', 'Y', 'Z', 'data']
Min RK
@minrk
No, if you use send_multipart to send those 3 messages, 3 separate calls to recv_multipart will receive the same 3 two-part messages
Mehran Kholdi
@semekh
Is that a fair assumption that a REQ-REP architecture should be resilient to arbitrary transport layer disconnections? If so, I suppose I have a reproducible situation in which breaking the TCP connection, results in a state in which both parties get stuck in a recv() statement even though the REQ side successfully reconnects to the REP.
Am I missing something? or do I need to file a bug report on that?
Mehran Kholdi
@semekh
Here's the code to reproduce it. I connect the REQ-REP side using a socat port-forward, and break it arbitrarily. It results in a broken state most of the times.
Min RK
@minrk
This probably isn't the channel for it, since it's libzmq behavior, but I don't think you should assume REQ-REP is resilient to much of anything. For resiliency, DEALER-ROUTER is ~always the right choice.
Mehran Kholdi
@semekh
Thank you. DEALER-ROUTER seems too complicated for my use case, since I have exactly one client and exactly one server, both of which do not fail. I just want to make sure I can recover from transport failures without writing lots of retry logic.
Min RK
@minrk
If they are transport failures, I think libzmq will ensure delivery or raise. You can't really retry with req/rep because on the strict send/recv/send/recv requirement (this is what DEALER-ROUTER avoids).
The only way to recover from an issue with req/rep is to close and reopen (probably both sides) sockets.
dealer/router avoids that unnecessary limitation, in exchange for keeping track of exactly one string on the router/rep side
David Alejandro Torrado Guzmán
@datorrado:matrix.org
[m]
hi
EPzhang
@EPzhang
image.png
Hi everyone. I'm a super new rookie to zeromq and have to ask a very basic quesiton. I just copy-pasted the helloworld test case into my pycharm and it seems like the tcp couldn't be connected. Any hints on what's happening and how should I fix this? Stuck at the "message = socket.recv()". Many thanks in advance.