by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Allie
    @cite-reader
    Not ZMQ, the extra API layer you need to go through will just distract you from what you need to do to implement the MQTT wire protocol. Go through uh... it's been years since I used java and I can't remember what they call their network sockets, one minute.
    just java.net.Socket.
    And then whatever other libraries you think might help you with implementing a binary network protocol.
    KiranKumar BS
    @kirankbs

    @cite-reader This is incredible feedback. It is my second day with ZMQ and I really liked until I reached a STREAM socket. You really cut down my efforts.
    I will check things with java.net.Socket and other libs for my current use case MQTT.

    I really continue my general learning with ZMQ as it is offering so many sockets to try different use cases.

    Allie
    @cite-reader
    Yeah, ZMQ is a great thing, but it's also a very specific thing in some ways.
    cyberquarks
    @cyberquarks

    Hi, is ZMQ a good option for real-time Browser-based Video/Audio streaming (reference: https://github.com/zeromq/JSMQ)? If yes, what would be the general capacity (# of streams that ZMQ can handle, say each payload is 50kb per 100ms) on a 16GB 6 cores example Linode machine. Can it handle 1000 or at least 500 concurrent streaming on such server?

    Atm, already using websockets to stream audio/video but the big issue I face with it is the more clients connect to the "video channel" the slower the video stream playback to all other clients. I am wondering also if ZMQ is vulnerable to this limitations?

    Kevin Sapper
    @sappo
    @somdoron as you did implement the websockets feature, what do you think.
    @cyberquarks this topic is rather specific. IMO you'll get more responses on the zeromq mailing list https://lists.zeromq.org/mailman/listinfo/zeromq-dev
    Doron Somech
    @somdoron
    @cyberquarks I would use https://github.com/zeromq/jszmq as it talks directly to zeromq (if you build from master)
    One thing regarding websocket transport is, that it is lacking a back-pressure, so I wouldn't use it for high frequency
    I plan to build the back-pressure into the protocol, but that would take time
    cyberquarks
    @cyberquarks
    @somdoron so jszmq already have back-pressure built in?
    Doron Somech
    @somdoron
    No, it is also based on websockets
    Shishir Pandey
    @shishirpy
    What can be done to have more adoption of zeromq library? When ever one thinks of messaging - they generally think of rabbitmq, kafka, etc. If it comes to request reply people generally think of web frameworks it is about django, spring, etc.
    Kevin Sapper
    @sappo
    That is a very good question @shishirpy :)
    Generally zeromq is the library you'll build software like rabbitmq, kafka, django or spring.
    IMO to get more adoption in that market we need better higher level API that integrate in the likes of spring etc.
    Shishir Pandey
    @shishirpy
    @sappo I think mojordomo is one such high level API, I don't know much but I guess malamute was also designed for that.
    Kevin Sapper
    @sappo
    Yes, malamute is a zeromq broker, zyre a peer to peer framework and dafka a pub/sub system similar to kafka.
    Doron Somech
    @somdoron
    Zeromq is low-level, if you think of Kafka and GRPC (the main competitor IMHO) they are high level
    I think the future should be to make such high level libraries in the ZEROMQ way
    We should make zeromq an alternative for GRPC, for that we first need a serialization framework and we can adopt some kind of standard (like apache avro)
    We can call it majordomo, make one majordomo broker project in C with a docker image and helm chart
    can support the client and worker on multiple langauges
    Next is dafka, which I plan to use in a project soon. Some features are still missing for me in order to use Dafka for production usage, mainly persistent subscriber (and transactional publisher, which is also not supported on Kafka)
    Doron Somech
    @somdoron
    @cyberquarks I would take a look at https://github.com/zeromq/jszmq, it is still work-in-progress but it talks to libzmq over websocket
    Zeromq (from master) has native support for WebSocket and should scale almost the same as regular ZMTP protocol
    I would give it a try
    Also, zeromq has zero-copy built-in, so if you use PUB-SUB, you will not make a copy of the msg, which should also improve throughput
    Shishir Pandey
    @shishirpy

    We should make zeromq an alternative for GRPC, for that we first need a serialization framework and we can adopt some kind of standard (like apache avro)

    This will be interesting to have.

    OliverNChalk
    @OliverNChalk
    is it advisable for a new project looking to work with zeromq.js to start with v6 or v5? I see there are significant API changes, however, wondering what the likelihood of running into issues on v6 is?
    Kevin Sapper
    @sappo
    In the README it is explicitly recommended to start a new project with v6 which sounds to me that the likelihood of running into serious issues is quite low.
    rodened
    @rodened
    Hi All! I am thinking about using zero MQ in a Google Cloud Function to send info to subscribers if a firestore db has been updated to subscribers. Is this a good idea? Or maybe Google has a solution? What do you recommend? I have just started to investigate the idea..
    Allie
    @cite-reader

    Off the top of my head, cloud functions are extremely transient, which means something else in the messaging topology needs to be stable. Either subscribers get listed in the cloud function's config, they register themselves in some KV store you have lying around, or you put a broker on a compute engine instance. Depending on your deliverability needs that broker might be as simple as running zmq_proxy between an XPUB and XSUB socket, or you might need to run with DEALER/ROUTER and write custom code to handle acknowledgements and maybe disk persistence.

    If you do want stronger deliverability guarantees than you can get out of PUB/SUB sockets by themselves, it's worth evaluating the development work of doing all that against using GCP's "Cloud Pub/Sub" product.

    rodened
    @rodened
    Yes, thanks for the thoughts. I have just checked out Google Pub/Sub and also found Node-RED flows that I will examine: https://flows.nodered.org/node/node-red-contrib-google-pubsub
    Shishir Pandey
    @shishirpy
    @rodened one has to be careful while sending message from a newly created socket. Since, the sockets are created in the background and it takes some time. The socket might not be created when you send the message, so you will have to keep you function running a little longer to make sure the socket is created when you send the message.
    Tom Kwong
    @tk3369
    Hi there. Noob here :-) Is there a list of how zeromq is used in large scale projects?
    Kevin Sapper
    @sappo
    Hi @tk3369, I'm not aware of such a list.
    I can tell you that ZeroMQ is used in a wide range of service from satellite networks to web portals.
    Because ZeroMQ gives you the ability to adopt to your personal needs we don't have this one pattern "How to apply ZeroMQ in large scale projects!"
    Is there anything more specific you like to know?
    OliverNChalk
    @OliverNChalk

    I'm trying to develop asynchronous RPC and am wondering if there's a reason to use PAIR, vs DEALER ROUTER. It seems like PAIR would be the natural choice because both sides can initiate communication where my understanding of DEALER ROUTER is that you would need to send using DEALER and would REPLY using ROUTER. Or you could setup DEALER <> DEALER I guess.

    I guess my main question is there any reason not to use PAIR for this use case, even though it's not strictly a multi-threading use case

    Allie
    @cite-reader
    Practically speaking, PAIR sockets are restricted to single-process use cases with topologies known at design time. I wouldn't normally call that a remote procedure call, but if it does describe your use case then PAIR would be the most natural socket type. If your design looks like the more traditional structure of a server processing requests from a variable number of clients, or if you're reaching across process or node boundaries, the DEALER/ROUTER pair would become more natural.
    OliverNChalk
    @OliverNChalk

    okay, that's a good point. I would need to know how many clients would connect at design time. I guess my design is more accurately a single server with N clients that need to perform concurrent requests to the server (ideally just message sending). Another point I should keep in mind is that the docs say PAIR is unsuitable for TPC based communication, due to lack of reconnection.

    Given that, I think the more appropriate design is DEALER<>ROUTER. Here clients would send requests to the service using DEALER, which would process them and then return the responses using ROUTER and the routing id. There may be a reason why certain recommendations are made in the docs :sweat_smile:

    Shishir Pandey
    @shishirpy
    @OliverNChalk : Yes, here is the set of allowed combinations from the doc. It explains it pretty well - http://zguide.zeromq.org/page:chapter3#toc6
    crimsoncor
    @crimsoncor

    I'm seeing some unusual behavior with our ZMQ setup and I'm curious if anyone has some ideas on where to start poking around. We are using v3.2 (upgrading is on the roadmap, but not any time super soon) and also using jzqm which we are building against our zmq build. We have services using jzmq that have a ROUTER socket listening for client connections and a second ROUTER that is used to distribute work to a bunch of worker threads. This is all on Linux. On the client side, we are primarily Windows and are using cppzmq (generally inside a library wrapped in python bindings) to talk to the services. Clients send messages using a DEALER socket and then poll for a given timeout waiting on a reply (the server will send an ack message to the client if the request is taking a long time to process to keep it from timing out). If there is a long enough period with receiving a reply from server (either ack or the answer to the query) an exception is raised and the socket is eventually remade. Generally this has all seemed to work fine.

    Recently with the whole pandemic, we've had lots of people transition to working over VPN and have now started to notice some unusual behavior on the service side. Inspecting the open sockets on the service side, I saw well over 3000 open connections today (much higher than normal) to one of our services and many IPs with 30-60 open sockets, almost exclusively from machines with VPN related IP addresses. Investigating some of those client IPs it seems like in many of the cases whichever computer made those sockets is no longer the same machine as the one which currently has the VPN address (our VPN system is a big pool of IPs, so there is no guarantee the same machine gets the same IP every day). So something in ZMQ is hanging onto those connections even though the client socket has clearly been cleaned up.

    From this stackoverflow post it seems like there might be something happening with messages being queued, but not sent with a ROUTER (https://stackoverflow.com/questions/28558274/how-to-drop-inactive-disconnected-peers-in-zmq). Does that seem like a likely candidate for what is happening here? Are there things I can do on the ZMQ side to try and prevent this from happening, noting that I don't have any control over the client IPs or when they might be dropping off VPN (Either intentionally or because the Internet is cruel)?

    Allie
    @cite-reader
    I'd suspect the problem exists below the the ZMQ level; it can take hours for the operating system to detect that a TCP connection has been broken by network weather, and ZMQ isn't going to know it needs to dispose of any associated resources until that detection happens. You may want to investigate the ZMQ_TCP_KEEPALIVE family of socket options to tighten that detection window.
    On libzmq 4.x you'd add ZMQ_HEARTBEAT_IVL and ZMQ_HEARTBEAT_TIMEOUT to your repertoire of tools to deal with it.
    crimsoncor
    @crimsoncor
    so the TCP keepalive on the linux server system is set to the default which is two hours and the phantom connections are persisting for longer than that
    Allie
    @cite-reader
    One thing I can never remember is whether SO_KEEPALIVE is on by default. If it's not, I don't think the keepalive logic runs at all.
    crimsoncor
    @crimsoncor
    Remember that keepalive support, even if configured in the kernel, is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockopt interface.
    So I guess keepalive is not on for us
    should resetting the ROUTER socket with a linger of zero clean-up all those extra connections?
    Allie
    @cite-reader
    Meaning closing the socket and then binding again? That should get you back to a clean state, yes. Clients will probably have to wait for a timeout before they'll restart their side of the protocol but it sounds like you're already set up for that.