zeromq/general

General community discussion

Allie

@cite-reader

How is your test rig getting the router socket to have anything to receive?

Ralph J. Steinhagen

@RalphSteinhagen

I disabled the forwarding part and simply output each received frame.

Allie

@cite-reader

yes but where are those frames coming from

Ralph J. Steinhagen

@RalphSteinhagen

... ah... from an external web browser ... (poor mans REST client ...)

Allie

@cite-reader

... how? An HTTP exchange doesn't look anything like a ZMTP handshake, you should be getting a connection error.

Ralph J. Steinhagen

@RalphSteinhagen

Nope 😁. Connecting to the REST/STREAM socket directly works correctly and I get the request header, body, can reply to the browser and the browser display the returned content. Connecting via the router port seems to acknowledge the connection and calls the handler... can see the header frame but then gets stuck ...

I took the forwarding from other proxy examples but these may not work with the stream socket.

Am unsure about the repackaging/framing that is possibly needed.

The underlying hypothesis was/is: if the first frame is a valid ID -> continue with the other ZeroMQ sockets else -> forward to the REST socket.

Allie

@cite-reader

oh my god apparently hitting a zmq socket with an http client just makes libzmq enter its version-negotiation logic, that's... not ideal.

I forgot zmtp/1 didn't have a sensible preamble.

I don't have a handy jeromq setup to dissect that particular interaction in wireshark but wow.

Okay so the thing that's going to trip you up with "is the first frame a valid ID" type thinking is that the first frame off a router or stream socket is synthesized by the ZMQ library. It doesn't exist on the wire, and it's valid by definition.

Allie

@cite-reader

You absolutely need to be working at the TCP level first, and proxy the connection to ZMQ infrastructure only if you recognize that protocol. The high-level plan there is something like reading ten bytes from a fresh connection, and if it's "\xFF\0\0\0\0\0\0\0\1\x7F" that's a ZMTP connection; anything else and you can let the HTTP infrastructure deal with it.

ZMTP/1 connections might look different. You probably don't care, that hasn't been current for ages.

Ralph J. Steinhagen

@RalphSteinhagen

... oops, did I break something? 😳 am also fine with C/C++ code ... just prototyped it with JeroMQ. Yes, the ID will be valid but not the next frames where the API protocols differ. My guess was that how I forward the frames after the ID to the STREAM socket might be wrong (haven't worked with it before).

Ralph J. Steinhagen

@RalphSteinhagen

I thought of this TCP proxy/router before the actual router but felt that these were too many moving parts and didn't want too much time (and long-term maintenance) on writing a reliable/performant proxy if ZeroMQ already provides facilities for that. Been down that road before with low-level TCP communication...

Allie

@cite-reader

By the time the router socket gives you any frames, what you see has already been through several decoding steps. There's a multi-round-trip handshake where the ZMQ library decides whether the connection is going to be using ancient ZMTP/1, old ZMTP/2, or modern ZMTP/3; the security mechanism makes a permit/deny decision; metadata is exchanged. There's a specific format for how those frames are separated and how many compose a single message. The only way an HTTP message is making it through all of that is if it's already been specifically encapsulated for the purpose. There's no way around this, if you're going to stuff HTTP and ZMQ on the same TCP port you need to work at the TCP level first.

Ralph J. Steinhagen

@RalphSteinhagen

@cite-reader I understand. Maybe I just needed someone to confirm my suspicion... In any case, thanks a lot for your time and patience to go with me through this nuts idea. Much appreciated. :+1:

OliverNChalk

@OliverNChalk

Is there a way to detect when a PUB socket is dropping messages from the PUB end? I.e. have the publisher notice that messages are dropping and then do something with that information? Currently, I have the subscriber that can see missing nonces and then take action but was wondering if there was something I could do on the publisher side

Is there a way to detect when a PUB socket is dropping messages from the PUB end? I.e. have the publisher notice that messages are dropping and then do something with that information? Currently, I have the subscriber that can see missing nonces and then take action but was wondering if there was something I could do on the publisher side

Found noDrop in the zeromq.js library

ZMQ_XPUB_NODROP

Иван Сердюк

@oceanfish81_twitter

Hi. I am curious if there are any end-user of LLVM/Clang/libc++ present here

Ralph J. Steinhagen

@RalphSteinhagen

A ZeroMQ MDP protocol-related question since I could not find any new documentation other than zeromq/majordomo#34. I am (re-)implementing a custom Majordomo protocol in view of extending the sub-protocols by adding some extra client/worker commands. In MDP V0.2 the first empty frame was removed limiting clients from REP/DEALER to DEALER sockets only -- which I sort of can understand -- but what is the purpose of the second empty frame in the worker protocol definition? Does it serve a particular purpose? The protocol could work without it. Any insight would be much appreciated!

Kevin Sapper

@sappo

According to the rfc https://rfc.zeromq.org/spec/18/ the empty frame is the envelope delimiter

Ralph J. Steinhagen

@RalphSteinhagen

Yes, this is clear... but what is this delimiter used for... the protocol would work also without it.

More specifically, why is the delimiter needed for the worker sub-protocol but not the client sub-protocol...

Ralph J. Steinhagen

@RalphSteinhagen

@sappo would you know whether there is any additional functionality associated with this empty frame other than convention or because it is written as such in the spec?

Samuel

@samuel-zhen

Hello, is it possible to use zeromq with nginx?

Kevin Sapper

@sappo

@samuel-zhen what is your use case? There seems to be a miss understanding about zeromq is/does.

lumenian

@lumenian

Hello,
Are there any existing resources on linking zeromq statically? I'm using MSYS2 gcc toolchain on Windows, and struggling to find a way to do so. Any help would be greatly appreciated!

Diego Molina

@diemol

Hi everyone,
Does anyone have an example of XPUB/XSUB working with Curve?

Adriano Ribeiro

@drr00t

Hi guys, i wondering to implement a mediator design pattern framework to used as dispatcher for domain events in DDD applications. So, considering zmq patterns, i'm considering two approaches: 1) a variation of MDP with message handler registered like a workers, so i will have a static but more responsive architecture; 2) instantiate handler as soon first message is received, i will have a "dynamic" architecture but the allocation problem; For reference the framework will at some how same functionality that MediatR https://github.com/jbogard/MediatR

Kevin Sapper

@sappo

@drr00t are you dispatching your domain events within one application or between applications? Do you require an ordered stream of events? Do you require to archive and replay events at some later point in time?

raul-parada

@raul-parada

Hi! I just started with ZMQ on Python, I'm trying this PUB/SUB example using two VMs but it doesn't work (I don't receive data from SUB side). The example is from this website https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pubsub.html. Someone tried on two separated machines within the same network? thanks

Adriano Ribeiro

@drr00t

@drr00t are you dispatching your domain events within one application or between applications? Do you require an ordered stream of events? Do you require to archive and replay events at some later point in time?

Hi @sappo sorry by delay, yes my ideia i kind use zmq as a layer to propagate ordered domain events as they were generated, but i think that i can handler this with incremental id field for the generated events (all events are groups as a "transaction"), so i can propagate inside application (like a observer) and cross-applications (syncing different microservices)

Name

@kwinz

@diemol I found this: https://gist.github.com/Panchatcharam/9407fc14611035a499faa4e169272668 (haven't tried it)

Kevin Sapper

@sappo

@drr00t are you dispatching your domain events within one application or between applications? Do you require an ordered stream of events? Do you require to archive and replay events at some later point in time?

Hi @sappo sorry by delay, yes my ideia i kind use zmq as a layer to propagate ordered domain events as they were generated, but i think that i can handler this with incremental id field for the generated events (all events are groups as a "transaction"), so i can propagate inside application (like a observer) and cross-applications (syncing different microservices)

@drr00t You might want to look into dafka. I'll make some time next week to think more about your problem.

Kostya

@KKostya

This message was deleted

Kostya

@KKostya

Hello zeroers! Are there any guidelines related to the integration of zeromq into cooperative threading frameworks?
Cooperative threads usually have some event loop under the hood that does event polling (say, using select(2)) and
evaluates promises reacting to the incoming events. Every blocking operation gets wrapped into a promise and a file
descriptor is added to the polling event loop.
Now, if I want to perform a blocking receive on a zmq socket, I have to, somehow, do the same wrapping.
Is there any discussion/documentation on how to do that properly?

Kevin Sapper

@sappo

@KKostya you'll get a broader audience for an answer on the mailing list

Adriano Ribeiro

@drr00t

@drr00t are you dispatching your domain events within one application or between applications? Do you require an ordered stream of events? Do you require to archive and replay events at some later point in time?

Hi @sappo sorry by delay, yes my ideia i kind use zmq as a layer to propagate ordered domain events as they were generated, but i think that i can handler this with incremental id field for the generated events (all events are groups as a "transaction"), so i can propagate inside application (like a observer) and cross-applications (syncing different microservices)

@drr00t You might want to look into dafka. I'll make some time next week to think more about your problem.

ok, jdafka is it some kind of sdk for dafka? is it possibli generate a dotnet version? thank you @sappo

Adriano Ribeiro

@drr00t

Hi guys, anybody succeed to build dafka? I'm stuck on cucumber-cpp dependency because of change in gmock. Is there a docker image for dafka?

Kevin Sapper

@sappo

@drr00t jdafka is native java implementation of the dafka protocol.

The cucumber dependency is optional. It's only required for the tests

You should be able to compile without it

Adriano Ribeiro

@drr00t

The cucumber dependency is optional. It's only required for the tests
Hum, ok. jdafka has same features that dafka?

OliverNChalk

@OliverNChalk

zeromq.js XPUB<>SUB is currently only notifying me when subscriber count goes from 0 to 1 and 1 to 0, is this intended behavior? I thought it would have notified me each time a subscriber joined or left?

James Ramm

@JamesRamm

Hello! Does anyone know the status of zeromq.js? Any active maintainers? V6.0.0 is declared as being in 'beta' but is recommended for new projects (which seems conflicting to me). There has also been no release for over a year now - is it still in beta? Furthermore, NodeJS has moved on and LTS is now v14. The current 'stable' version of zeromq.js (v5) is not compatible with Node v14+. So, all seems a bit of a mess....

Dmitri Peredera

@Muhomorik

Hello, any usees of NetMq Beacon here?
I am trying to figure out why broadcast is not received on any of sockets in the sample.

Where communities thrive

People

Repo info

Activity