-cn --cluster-nameand I don't have a suggestion to
-a --addresses(Yep, we have conflict in jobList subcommand)...
-nis a valued option, so we could use
-cnin this case
-ais used for
list-jobs, so we need other name for
addressesin this case
We can use
-cfor cluster name and
-afor "addresses". It will break list-jobs, but that's probably OK, we can only have
--allin list-jobs. Another option is to use something like
-c is used to
submit too ahahah, I liked the
-t to address, also remove
list-job. Any other suggestion for cluster name?
I am facing a problem with event discorder and max lag.
My jet job is supposed to:
My problem is that the devices/sensors upload their measurements to the kafka topic (source for jet job) in parallel and only in 5-minute intervals.
This results in event disorder. I thought that I could fix this by setting withNativeTimestamps(300_000) (5 min) to counteract the upload interval.
No events are skipped now, but I am not getting any window result either.
What am I doing wrong? Do I use a wrong time unit for conversion or is my understand of the max lag concept wrong?
Thanks in advance!
@viliam-durina @mtopolnik @cangencer
We have around 100 (and rising) Raspberry Pis collecting measurements from various sensors in mostly 5 minute intervals.
The measurements are stored in a redis database on the Pis first and uploaded in 5-minute intervals to a Spring backend.
The backend parses the measurements (sent via HTTP Request) and sends them to Kafka.
We thereby specify the timestamp of the measurement as the Kafka timestamp, so that timestamp is what Jet uses as the "native timestamp", isn't it?
We have 4 kafka partitions and use a Raspberry Pi unique ID as the key, so we probably have like 25 Pis in each partition.
Given the fact that we have multiple Pis per partition, which upload their data independently of each other, we necessarily have event disorder on partition level.
This 1-minute tumbling window was probably chosen unfortunately in this case, I agree.
It is just for testing though and from my understanding I thought that I should at least get one window result every 5 window aggregations then.
So tomorrow I will test to: