Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 31 10:58

    solatis on master

    Mark next release cycle (compare)

  • Aug 31 10:57

    solatis on master

    Temporarily roll back version (compare)

  • Aug 31 10:51

    solatis on 0.14.x

    Temporarily roll back version Mark version 0.14.6 (compare)

  • Aug 31 10:50

    solatis on 0.14.6

    (compare)

  • Aug 31 10:41

    solatis on 0.14.x

    Prepare for next release cycle Additional documentation improv… Log as :info level when Aeron s… and 4 more (compare)

  • Aug 31 10:33

    solatis on master

    Fix incorrect versions documentation: fix hyperlink fo… Merge branch '0.14.x' and 1 more (compare)

  • Jul 28 13:59

    lbradstreet on master

    documentation: fix hyperlink fo… (compare)

  • Jul 28 13:55

    lbradstreet on 0.14.x

    documentation: fix hyperlink fo… (compare)

  • Jul 28 13:55
    lbradstreet closed #899
  • Jul 27 01:05
    The-Alchemist opened #899
  • Jul 05 06:40
    solatis commented #887
  • Jul 05 06:39

    solatis on master

    Re-introduce support for :seq/c… (compare)

  • Jul 05 06:39
    solatis closed #898
  • Jul 05 06:39

    solatis on master

    Log as :info level when Aeron s… (compare)

  • Jul 05 06:39
    solatis closed #897
  • Jul 05 06:38
    solatis commented #897
  • Jul 03 11:18
    thenonameguy commented #887
  • Jun 08 00:24
    sundbry opened #898
  • May 30 18:59
    sundbry opened #897
  • May 23 09:36
    sundbry edited #896
Lucas Bradstreet
@lbradstreet
If it’s not that, then it’s pretty odd.
Lutz Büch
@lutz-100worte
That might be the case, yes.
Lutz Büch
@lutz-100worte
Is there a way to inspect the watermark at each peer during runtime?
Lutz Büch
@lutz-100worte
Is there any documentation on how refinement works? What are create-state-update and apply-state-update intended for?
Lutz Büch
@lutz-100worte
Is there any documentation on how :reduce tasks work? Is there any difference between a terminal :reduce task and an terminal (i.e., :output) task with plugin :onyx.peer.function/function?
Lutz Büch
@lutz-100worte
(I am aware of the section about :reduce tasks in the user-guide and in the cheat-sheet...)
Nicolas Estrada
@neuromantik33
Is there any way to use colocated task scheduling in order to minimize network traffic (I have a lot of issues with aeron) yet have it respect the max-peers/n-peers affected to tasks?
My use case is probably not the most ideal, but we have many (100+) small jobs with 4-5 tasks which we would like to be scheduled onto the same peer-group if possible since onyx seems to buckle when too much network (status msgs, nack msgs and bytes flying back & forth)
when using the :onyx.task-scheduler/colcocated it assigns a large number of peers to a task where only 1 is semantically correct, ie. reading/writing from/to a file, is this implemented by design?
Lutz Büch
@lutz-100worte
@neuromantik33 I think you should try your luck in clojurians.slack.com.
Danie Roux
@danieroux
Does anyone have a working example of resume-points? When I resubmit my job using onyx.api/build-resume-point, I get a :incorrect-job-hash error
Timothy Washington
@twashing
Does anyone know how to pass a sliding window of items from point A to B in a workflow?
I can only see how to fire a trigger. Not how to pass on each windowed list to the next node in my workflow.
Timothy Washington
@twashing
Hey @lbradstreet , would you use emit instead of :trigger/sync.
Timothy Washington
@twashing
Ok, emit seems to be passing data downstream.
Timothy Washington
@twashing
But I can't seem to get clean partitions flowing to the next node. Ie, I'd like to be able to do the equivalent of core.async channels.
(onto-chan A (range 20))
(pipeline concurrency B partition-by-5-xform A)
(pipeline concurrency C process-partitions-of-5-xform B)
Lutz Büch
@lutz-100worte
Can anyone recommend Apache Beam as an alternative to Onyx?
Lutz Büch
@lutz-100worte
:question:
Roland Thiolliere
@RolT
@lutz-100worte it works with clojure with a few quirks, but repl dev is harder.
Lutz Büch
@lutz-100worte
Thanks!
Jean-Sebastien A. Beaudry
@jsab
How do I pass connection to the db to my onyx job? (i.e. not access it using a global var)
When setting the parameter in the catalog I think it whines because it cannot be serialized.
Jean-Sebastien A. Beaudry
@jsab
Looking at how learn-onyx 4.1 and 4.2 are built, I'm not sure it is even possible, both use global vars
Nicolas Estrada
@neuromantik33
@jsab I think the best practice is to pass the connection info to a lifecycle and create the db connection there and assoc it to the event map for use in tasks
@jsab Probably best to look at the code @ https://github.com/onyx-platform/onyx-sql (or simply use the plugin as is)
Lutz Büch
@lutz-100worte
@jsab Here is Michael Drogalis talking about lifecycles and how they manage state and facilitate side-effects.
Jean-Sebastien A. Beaudry
@jsab
So if I understand correctly, if I have 10 onyx jobs requiring a datadog connection, I will have 10 datadog connections?
Each job will receive connection parameters and initialize a connection in the lifecycle/before-task-start
Ivan
@advancedwebdeveloper
Hello
Anyone interested to make a tryout to port Onyx for Clojure CLR?
Lucas Bradstreet
@lbradstreet
It would be incredibly difficult to make it work in a distributed way because we depend on too many java libraries
Ivan
@advancedwebdeveloper
@lbradstreet , anyone has a full map of dependencies and correlated API usage stats?
Jean-Sebastien A. Beaudry
@jsab
Is there a way to disable checkpointing? (infinite streaming task)
Eugene Pakhomov
@p-himik
Is it possible/feasible to use Onyx without ZooKeeper and with the Redis plugin? I want to deploy an app to Heroku, and I don't think there's any way to use ZooKeeper there without actually writing a Heroku add-on and/or providing your own servers.
Eugene Pakhomov
@p-himik
If no, what would be an alternative where I can have manually managed nodes and schedule tasks via a Redis queue?
Travis Camechis
@camechis
@p-himik I am rusty but I am pretty sure you have to use Zookeeper
joefromct
@joefromct
Check out onyx-local-rt https://github.com/onyx-platform/onyx-local-rt
@p-himik
Might be helpful, I think it’s pretty neat.
Ivan
@advancedwebdeveloper
http://vijaykiran.com/2019/04/defn-48-david-miller/ - how about porting to Clojure CLR
?
Ivan
@advancedwebdeveloper
who can investigate that and check which challenges would be expected, while porting?
Lucas Bradstreet
@lbradstreet
The challenges are significant. Onyx depends on quite a number of Java libraries.
Ivan
@advancedwebdeveloper
we require a full roadmap - not just a generalisation
Eugene Pakhomov
@p-himik
@joefromct Thanks! Alas, it's the opposite extreme since apparently it doesn't allow any distribution at all. But I can definitely see how it could potentially be useful.
Lucas Bradstreet
@lbradstreet
@advancedwebdeveloper we don’t have the time to investigate this (see onyx-platform/onyx#887), and there is other high priority work to be done on the JVM. All I can really tell you is that Onyx leverages a number of JVM libraries, so it would not be a small job. You can do a survey of the code base to see which libraries are depended on and do your own analysis of how much work it would be.
liloxboy
@liloxboy
I am new to distributed computing, and also new to onyx. I tried to setup 2 onyx in 2 machines, each with 4 v-peers running. When i submitted a job (very simple), I println the segment in one of the tasks, but i cannot find where the println result goes... Anyone can help? I am really appreciate onyx, and want to use it in production. I found it's awesome and better than apache spark, storm, flink...
liloxboy
@liloxboy
Forget my question, i used the key :onyx.messaging/external-addr in peer config and resolved it.
liloxboy
@liloxboy
anyone have any idea about below exception?
2019-09-07 13:34:27 feifei WARN [onyx.peer.task-lifecycle:177] -
java.lang.Thread.run Thread.java: 748
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 624
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1149
...
clojure.core.async/thread-call/fn async.clj: 441
onyx.peer.task-lifecycle/start-task-lifecycle!/fn task_lifecycle.clj: 1155
onyx.peer.task-lifecycle/run-task-lifecycle! task_lifecycle.clj: 550
onyx.peer.task-lifecycle.TaskStateMachine/exec task_lifecycle.clj: 1070
onyx.peer.task-lifecycle/poll-recover-input-function task_lifecycle.clj: 510
onyx.messaging.aeron.subscriber.Subscriber/poll! subscriber.clj: 206
io.aeron.Subscription.controlledPoll Subscription.java: 246
io.aeron.Image.controlledPoll Image.java: 343
io.aeron.ControlledFragmentAssembler.onFragment ControlledFragmentAssembler.java: 121
onyx.messaging.aeron.subscriber.Subscriber/onFragment subscriber.clj: 307
onyx.messaging.aeron.status-publisher.StatusPublisher/set-session-id! status_publisher.clj: 74
java.lang.AssertionError: Assert failed: (or (nil? session-id) (= session-id session-id))
clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle :lifecycle/poll-recover. Killing the job. -> Exception type: java.lang.AssertionError. Exception message: Assert failed: (or (nil? session-id) (= session-id session-id
))
job-id: #uuid "840c0668-6332-4923-62b5-699b5c4efcaa"
metadata: {:activity-id #uuid "2e380184-d962-4910-a86d-f5467fce5432", :activity-kind :mgeovsl, :job-id #uuid "840c0668-6332-4923-62b5-699b5c4efcaa", :job-hash "df8d4dea3c8f2f57df98d3d285edc9eaa3e79ecbe71349f27e65ff2f47764abe"}
peer-id: #uuid "7d40bbf0-bbf6-09d0-5eac-1b29b0bcb53a"
task-name: :load-goals