Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    bv
    @bv:matrix.org
    [m]
    looks reasonable
    Cassiel-girl
    @Cassiel-girl
    This is the official zmq example code for sending and receiving numpy arrays. Do you think this has any advantages over using json/protobufd etc? This code seems to be more complex.
    bv
    @bv:matrix.org
    [m]
    if you do not care about non-Python languages you could also use "pickle" or use Numpy.load/save directly via io.BytesIO
    Actually, looking at the code closer, socket.send(A, ...) I'm not sure how A is sent. maybe via pickle serialization
    Cassiel-girl
    @Cassiel-girl
    Several data
    Method 1: Use zmq.SNDMORE to send the data several times.
    1. Method 2: We manually pack the data into a json structure and send it once.
      What are the advantages of each of the two methods? Will the former be more performant than the latter?
    bv
    @bv:matrix.org
    [m]
    I'd figure out is non-Python code ever needs to be involved. If so, I'd not use a Python-specific serialization.
    it's hard to say about performance without testing. often, things I think will be slow are not and vice versa.
    what is the general application meant for?
    Cassiel-girl
    @Cassiel-girl
    zmq.SNDMORE and json are functionally equivalent, right? Anything that zmq.SNDMORE can do, we can do based on json, right?
    bv
    @bv:matrix.org
    [m]
    as for SNDMORE vs packing, I don't think there's all that much difference. SNDMORE will make zeromq do the packing instead of your own code.
    no. SNDMORE and use of JSON are orthogonal
    SNDMORE tells zeromq you want to make a "multi part message"
    Cassiel-girl
    @Cassiel-girl
    how to understand SNDMORE and use of JSON are orthogonal
    bv
    @bv:matrix.org
    [m]
    on the wire a multi-part message looks like: [size|bytes][size|bytes] where each size gives the size of the bytes part.
    without SNDMORE the message is simply "single part"
    with JSON the bytes part is filled specifically with JSON text
    so, I can send a single part message holding JSON or I can send a multi-part message with each part holding JSON (or some other format). the two choices are independent / orthogonal
    Cassiel-girl
    @Cassiel-girl
    I can understand from you explain
    Both sendmore and json can be used to implement message protocols.
    bv
    @bv:matrix.org
    [m]
    yes.
    though, you need not use sendmore. the "draft" sockets like SERVER/CLIENT do not even support multi-part messages.
    and anytime you want something like multiple parts, you can always create them at the application layer
    Cassiel-girl
    @Cassiel-girl
    nanomsg will replace zeromq?
    bv
    @bv:matrix.org
    [m]
    I doubt it. Both have been around for a long time. Is nanomsg even maintained anymore?
    nanomsg has had one set of commits this year. so it's not neglected.
    Cassiel-girl
    @Cassiel-girl
    Do you mean that nanomsg is no longer maintained?
    bv
    @bv:matrix.org
    [m]
    It was a question. Then, I checked it's github and see one group of commits this year. So, it seems still maintained.
    Cassiel-girl
    @Cassiel-girl
    What is the biggest advantage of nanomsg over zeromq?
    bv
    @bv:matrix.org
    [m]
    I have never used nanomsg so I can't comment.
    Cassiel-girl
    @Cassiel-girl
    Is zeromq a message queue?
    bv
    @bv:matrix.org
    [m]
    yes and no. it has message queues inside each of its socket. but many people hear "message queue" and they think "centralized message broker". zeromq is NOT that, but zeromq can be used to make "centralized message brokers".
    we can see many blogs or articles comparing zeromq and rabbitmq and other "centralized message brokers". These are "apples and oranges" comparisons. A more correct comparison would be "zeromq" (or nanomsg) and BSD sockets.
    Cassiel-girl
    @Cassiel-girl
    Many people say that zeromq is not a message queue, it is just a socket-like library. The performance is a bit higher because there is a lot less stuff.
    Is zeromq still being maintained? Does it support rmda?
    bv
    @bv:matrix.org
    [m]
    zeromq is being maintained and new language bindings are in healthy development
    I don't know about rdma. but I know that stock zeromq can saturate 25 Gbps and with more I/O threads can saturate 100 Gbps.
    Cassiel-girl
    @Cassiel-girl
    how to saturate 100Gbps?
    could you show some examples?
    bv
    @bv:matrix.org
    [m]
    the context can be told to use multiple I/O threads
    however, this leads to out-of-order messages which must then be handled by the application layer
    easier would be to use multiple sockets
    in the wiki there is some tests of 100 Gbps but since the hack that's not available
    I have an informal report of a test I did
    Cassiel-girl
    @Cassiel-girl
    I have recently encountered this problem. Could you please send me an example to learn it?
    bv
    @bv:matrix.org
    [m]
    which problem?
    Cassiel-girl
    @Cassiel-girl
    I have a 200Gbps network, how to saturate it?
    bv
    @bv:matrix.org
    [m]
    Oh, I see. here's my informal note. https://www.phy.bnl.gov/~bviren/tmp/zmq/latthr.pdf
    libzmq has latency and throughput test programs in the source tree. I also wrote my own https://github.com/brettviren/zperfmq
    I don't think I've seen a test at 200 Gbps yet so if you test please share your results.
    Cassiel-girl
    @Cassiel-girl
    sure, thank you