My use case is probably not the most ideal, but we have many (100+) small jobs with 4-5 tasks which we would like to be scheduled onto the same peer-group if possible since onyx seems to buckle when too much network (status msgs, nack msgs and bytes flying back & forth)
when using the
:onyx.task-scheduler/colcocated it assigns a large number of peers to a task where only 1 is semantically correct, ie. reading/writing from/to a file, is this implemented by design?
I can only see how to fire a trigger. Not how to pass on each windowed list to the next node in my workflow.
You can use http://www.onyxplatform.org/docs/cheat-sheet/latest/#trigger-entry/:trigger/emit to do so
(onto-chan A (range 20))
(pipeline concurrency B partition-by-5-xform A)
(pipeline concurrency C process-partitions-of-5-xform B)
When setting the parameter in the catalog I think it whines because it cannot be serialized.
@jsab Probably best to look at the code @ https://github.com/onyx-platform/onyx-sql (or simply use the plugin as is)
@jsab Here is Michael Drogalis talking about lifecycles and how they manage state and facilitate side-effects.
Each job will receive connection parameters and initialize a connection in the lifecycle/before-task-start
Anyone interested to make a tryout to port Onyx for Clojure CLR?
Might be helpful, I think it’s pretty neat.
?
@advancedwebdeveloper we don’t have the time to investigate this (see onyx-platform/onyx#887), and there is other high priority work to be done on the JVM. All I can really tell you is that Onyx leverages a number of JVM libraries, so it would not be a small job. You can do a survey of the code base to see which libraries are depended on and do your own analysis of how much work it would be.
I am new to distributed computing, and also new to onyx. I tried to setup 2 onyx in 2 machines, each with 4 v-peers running. When i submitted a job (very simple), I println the segment in one of the tasks, but i cannot find where the println result goes... Anyone can help? I am really appreciate onyx, and want to use it in production. I found it's awesome and better than apache spark, storm, flink...
anyone have any idea about below exception?
2019-09-07 13:34:27 feifei WARN [onyx.peer.task-lifecycle:177] -
java.lang.Thread.run Thread.java: 748
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 624
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1149
...
clojure.core.async/thread-call/fn async.clj: 441
onyx.peer.task-lifecycle/start-task-lifecycle!/fn task_lifecycle.clj: 1155
onyx.peer.task-lifecycle/run-task-lifecycle! task_lifecycle.clj: 550
onyx.peer.task-lifecycle.TaskStateMachine/exec task_lifecycle.clj: 1070
onyx.peer.task-lifecycle/poll-recover-input-function task_lifecycle.clj: 510
onyx.messaging.aeron.subscriber.Subscriber/poll! subscriber.clj: 206
io.aeron.Subscription.controlledPoll Subscription.java: 246
io.aeron.Image.controlledPoll Image.java: 343
io.aeron.ControlledFragmentAssembler.onFragment ControlledFragmentAssembler.java: 121
onyx.messaging.aeron.subscriber.Subscriber/onFragment subscriber.clj: 307
onyx.messaging.aeron.status-publisher.StatusPublisher/set-session-id! status_publisher.clj: 74
java.lang.AssertionError: Assert failed: (or (nil? session-id) (= session-id session-id))
clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle :lifecycle/poll-recover. Killing the job. -> Exception type: java.lang.AssertionError. Exception message: Assert failed: (or (nil? session-id) (= session-id session-id))
job-id: #uuid "840c0668-6332-4923-62b5-699b5c4efcaa"
metadata: {:activity-id #uuid "2e380184-d962-4910-a86d-f5467fce5432", :activity-kind :mgeovsl, :job-id #uuid "840c0668-6332-4923-62b5-699b5c4efcaa", :job-hash "df8d4dea3c8f2f57df98d3d285edc9eaa3e79ecbe71349f27e65ff2f47764abe"}
peer-id: #uuid "7d40bbf0-bbf6-09d0-5eac-1b29b0bcb53a"
task-name: :load-goals
2019-09-07 13:34:27 feifei WARN [onyx.peer.task-lifecycle:177] -
java.lang.Thread.run Thread.java: 748
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 624
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1149
...
clojure.core.async/thread-call/fn async.clj: 441
onyx.peer.task-lifecycle/start-task-lifecycle!/fn task_lifecycle.clj: 1155
onyx.peer.task-lifecycle/run-task-lifecycle! task_lifecycle.clj: 550
onyx.peer.task-lifecycle.TaskStateMachine/exec task_lifecycle.clj: 1070
onyx.peer.task-lifecycle/poll-recover-input-function task_lifecycle.clj: 510
onyx.messaging.aeron.subscriber.Subscriber/poll! subscriber.clj: 206
io.aeron.Subscription.controlledPoll Subscription.java: 246
io.aeron.Image.controlledPoll Image.java: 343
io.aeron.ControlledFragmentAssembler.onFragment ControlledFragmentAssembler.java: 121
onyx.messaging.aeron.subscriber.Subscriber/onFragment subscriber.clj: 307
onyx.messaging.aeron.status-publisher.StatusPublisher/set-session-id! status_publisher.clj: 74
java.lang.AssertionError: Assert failed: (or (nil? session-id) (= session-id session-id))
clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle :lifecycle/poll-recover. Killing the job. -> Exception type: java.lang.AssertionError. Exception message: Assert failed: (or (nil? session-id) (= session-id session-id))
job-id: #uuid "840c0668-6332-4923-62b5-699b5c4efcaa"
metadata: {:activity-id #uuid "2e380184-d962-4910-a86d-f5467fce5432", :activity-kind :mgeovsl, :job-id #uuid "840c0668-6332-4923-62b5-699b5c4efcaa", :job-hash "df8d4dea3c8f2f57df98d3d285edc9eaa3e79ecbe71349f27e65ff2f47764abe"}
peer-id: #uuid "7d40bbf0-bbf6-09d0-5eac-1b29b0bcb53a"
task-name: :load-goals
Is anyone using onyx to join a stream against a large dataset via copartitioning (a la confluent's KTables)? Is there a good way to build something like ktables? Does the onyx kafka plugin expose the kafka consumer it is using or provide any control/insight into partition assignment?