minrk on can-connect-aain
use SIGALRM for test timeout s… (compare)
minrk on main
remove optional JSON imports fr… Merge pull request #1549 from m… (compare)
fsyncor similar, so an accepted message can easily be lost, or in the event of a crash the files it leaves behind may be in some indeterminate, garbage state that may cause it to send arbitrary nonsense on to peers. Even worse, the example uses the
zmsg_savefunction from czmq to write the message to the file, which explicitly is not stable across library versions, preventing easy upgrades. Definitely do the work to define a stable message serialization format, and pull in an actual database to do the actual saving. (A simple on disk key-value store is likely fine, you just want it to treat your valuable data with the care it deserves.) At that point there's a variety of ways to trade off throughput for latency, but you do have to measure your implementation on your hardware.
Right, okay. That's an eyebrow-raising sample rate.
I'm going to suggest inverting your current flow between Postgres and Kafka. When using both, it's really unusual to hit Postgres first: its APIs are poorly suited to streaming workloads, and Kafka is highly tuned for persisting a bunch of messages quickly in ways Postgres's design priorities make difficult. You can then use a Kafka consumer application to put the events into a more structured database in larger chunks; hopefully large enough to make up for the work that needs to be done to write indices and whatever else.
Also, I don't know how many clients you have or what these events look like, but under modest assumptions at 100 Hz even a single client will need large amounts of storage relatively quickly. Depending on the shape of your events and how long you want to keep them, it may be beneficial to evaluate column stores sooner rather than later for better compression.
Hi all! I have what is (perhaps) a simple question. I am a complete newbie when it comes to sockets etc., so bear with me.
TLDR: How fast can a message be sent and received by the REP/REQ pattern?
I have a REQ socket that is asking for some data from a REP socket and vice versa. It does this in several send/recv steps. The client is currently simulating a real-time application. The most common messages being sent (after some initialisation) are some arrays of floating point numbers (of about length 50).
line-profiler on a test function, I can see that the
socket.recv() method in the REP server is taking about 90-95% of the time of the entire process. This equates to ~ 4 ms. I have a "feeling" that this could be faster, seeing as it is a local operation. I have tried to speed this up with different approaches, but haven't come across anything that might help. These approaches:
I am hoping for a transfer speed per message of about 0.5-1 ms. Is this possible with REP/REQ, or with another pattern? Or has it something to do with my PC aswell?
The REQ client function is using a different thread. I read somewhere that is isn't good, so please let me know.
As I am a newb in this world of sockets, I can't think of a way to implement a MWE that is much simpler than the base code I have. Luckily, one can clone our repo and just run
line-profiler (or any profiler) on the test function "tests/test_choose_minimum_function.py". This will send the above array of about 50 floats in each message.
line-profiler, I tag the
tests.test_choose_minimum_function.test_run_plugin_with_user_defined_features function and the
shapelink.shapelink_plugin.ShapeLinkPlugin.handle_messages method with the
The repo: https://github.com/ZELLMECHANIK-DRESDEN/shapelink
I can also just copy the results of the
Thanks for your time :)
recv_multipartsI feel like I cannot send a simple message without an identity. Is that correct? Am I suppose to create a fake identity?