Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Alex Buchanan
    @buchanae
    ohsu-comp-bio/funnel#414
    chirag
    @jain_chirag04_twitter
    @buchanae interesting. we have our own abstraction like funnel that integrates with k8. I will keep an eye on this.
    Alex Buchanan
    @buchanae
    Cool. Which is that?
    chirag
    @jain_chirag04_twitter
    not sure i understand your question there
    Alex Buchanan
    @buchanae
    Does your abstraction which integrates with k8s have name and is it open source?
    chirag
    @jain_chirag04_twitter
    got it. it's a http api which calls into k8 api and has some business logic coupled with it as well. It's not open source yet but something we want to do at some point.
    which is why i was interested in funnel. if there was k8 support, we could probably use it as is
    Alex Buchanan
    @buchanae
    Cool. Would love to see where Funnel is different/similar.
    Adding basic support for k8s would be fairly straightfoward. The main hurdle is that there's potentially a docker-in-docker situation, which brings weird quirks sometimes.
    Usually Funnel has a worker which calls out to docker to start containers, so it's sort of at odds with a kublet.
    chirag
    @jain_chirag04_twitter
    right. we are using DIND image as a sidecar container right now
    Alex Buchanan
    @buchanae
    Cool. And do you use the k8s job scheduler?
    chirag
    @jain_chirag04_twitter
    k8 is the scheduler but we don't use the Job mechanism
    k8 schedules ever task a regular Pod
    Alex Buchanan
    @buchanae
    Gotcha. So the pods start up, run a task, and then exit? Or do they stay online for multiple tasks?
    chirag
    @jain_chirag04_twitter
    because we want run-once and Job does not give that guarantee
    Alex Buchanan
    @buchanae
    Ah interesting, I misunderstood Job then
    chirag
    @jain_chirag04_twitter
    yes, that's the idea. k8 starts a pod, runs a task(step) and then exits. every new/subsequent task(step) is a new pod
    Alex Buchanan
    @buchanae
    Great, thanks. We can make something like that work.
    I'll try to write up a plan this week, and we'll go from there. If you're interested, the compute backends are here: https://github.com/ohsu-comp-bio/funnel/tree/master/compute
    They're fairly simple, although we're working on adding better integration for cancel/monitoring/etc.
    chirag
    @jain_chirag04_twitter
    so in this case k8 will be a new compute backend right?
    Alex Buchanan
    @buchanae
    right
    chirag
    @jain_chirag04_twitter
    i haven't done a deep dive into funnel so not entirely sure
    ok great
    Azza Ahmed
    @azzaea
    @buchanae Thank you for the prompt response and clarification
    Michael R. Crusoe
    @mr-c
    @zo0z @buchanae @jain_chirag04_twitter REANA is a Kubernetes based workflow system and @anton-khodak is adding CWL support to it: reanahub/reana-workflow-engine-cwl#9
    GESS THOMS XAVIER
    @gezthxbio
    Hi Im trying to run whole genome cwl on a small data using rabix which has SBG Prepare Intervals
    My interval bed looks like this
    chr1 1 249250621
    chr2 1 243199373
    It throws error evaluating the expression.
    Any help
    Adrian Sharma
    @sharmatime
    @gezthxbio I see you left an issue on github with lots of detail. Thanks
    GESS THOMS XAVIER
    @gezthxbio
    @sharmatime Thanks indeed
    Keiran Raine
    @keiranmraine

    I'm having some trouble using the TES back end with SE/Ceph. I've tested that funnel is able to process a simple request with S3 using curl already.

    When I try to do the same with ./rabix having modified the config I get an incomprehensible error:

    $ ./rabix -c ./config examples/dna2protein/dna2protein.cwl.json examples/dna2protein/inputs.json
    [2018-05-05 08:46:37.019] [ERROR] EventProcessor failed to process event InitEvent [value={input_file=FileValue [size=null, path=/private/var/tmp/rabix-cli-1.0.5/examples/dna2protein/data/input.txt, location=null, checksum=null, secondaryFiles=[], properties={sbg:metadata=null}]}, rootId=e6e70855-e5b1-42c3-99be-49a61956fed1, config=null].
    java.util.NoSuchElementException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1444) ~[na:1.8.0_161]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1466) ~[na:1.8.0_161]
        at org.rabix.engine.service.impl.BackendServiceImpl.sendToExecution(BackendServiceImpl.java:258) ~[rabix-cli.jar:na]
        at org.rabix.engine.status.impl.DefaultEngineStatusCallback.onJobReady(DefaultEngineStatusCallback.java:29) ~[rabix-cli.jar:na]
        at org.rabix.cli.status.LocalBackendEngineStatusCallback.onJobReady(LocalBackendEngineStatusCallback.java:69) ~[rabix-cli.jar:na]
        at org.rabix.engine.status.impl.DefaultEngineStatusCallback.onJobsReady(DefaultEngineStatusCallback.java:35) ~[rabix-cli.jar:na]
        at org.rabix.engine.service.impl.JobServiceImpl.handleJobsReady(JobServiceImpl.java:242) ~[rabix-cli.jar:na]
        at org.rabix.engine.processor.impl.EventProcessorImpl.processReadyJobs(EventProcessorImpl.java:166) [rabix-cli.jar:na]
        at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$doProcessEvent$3(EventProcessorImpl.java:113) [rabix-cli.jar:na]
        at org.rabix.engine.store.memory.InMemoryRepositoryRegistry.doInTransaction(InMemoryRepositoryRegistry.java:92) ~[rabix-cli.jar:na]
        at org.rabix.engine.processor.impl.EventProcessorImpl.doProcessEvent(EventProcessorImpl.java:107) [rabix-cli.jar:na]
        at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$null$1(EventProcessorImpl.java:91) [rabix-cli.jar:na]
        at org.rabix.engine.metrics.impl.MetricsHelperImpl.time(MetricsHelperImpl.java:78) ~[rabix-cli.jar:na]
        at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$start$2(EventProcessorImpl.java:91) [rabix-cli.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
    EventProcessor failed to process event:
    InitEvent [value={input_file=FileValue [size=null, path=/private/var/tmp/rabix-cli-1.0.5/examples/dna2protein/data/input.txt, location=null, checksum=null, secondaryFiles=[], properties={sbg:metadata=null}]}, rootId=e6e70855-e5b1-42c3-99be-49a61956fed1, config=null]

    In config I've changed:

    backend.embedded.types=TES
    
    tes.client_scheme=http
    tes.client_host=172.27.18.81
    tes.client_port=8000
    
    tes.storage_base=s3://cog.sanger.ac.uk/funnel-bucket/rabix

    and added:

    s3.endpoints= s3://cog.sanger.ac.uk/
    s3.protocol=HTTPS
    s3.access_key=...
    s3.secret_key=...
    s3.path_style_access=true
    s3.signer_override=AWS3SignerType
    Keiran Raine
    @keiranmraine
    Is this area no longer monitored?
    Adam Struck
    @adamstruck
    @keiranmraine sorry I haven't been monitoring gitter lately. I think its a config issue. First of all the s3 config values should follow the format: s3.<some arbitrary name>.<option> (e.g. s3.ceph.endpoints=s3://cog.sanger.ac.uk/). Second I think your signer type should be either AWS4SignerType or S3SignerType.
    Keiran Raine
    @keiranmraine
    @adamstruck thanks for the info. Will give that a try in a few hours.
    Manabu ISHII
    @manabuishii
    Hello I found small typo in Rabix Composer Document http://docs.rabix.io/introduction-to-tool-wrapping
    Where is good place to feedback ?
    lfaller-zymergen
    @lfaller-zymergen

    Hello! I am trying to play around with rabix on a Mac but it's telling me I need to install java. However, java is installed and on my path:

    $> which java
    /usr/bin/java

    How can I tell rabix how to find java? Thanks!!

    lfaller-zymergen
    @lfaller-zymergen
    :point_up: solved -- I needed to install the complete JDK from here: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
    Kostas Mavrommatis
    @kmavrommatis
    Hi, I am trying to play with Rabix on MacOS. I have wrapped a simple java application which is stored as a docker image at the AWS ECR. Although I can pull the image from the command line, when i execute the workflow from Rabix I get the error that it cannot pull the image. What are the steps/requirements to be able to pull from ECR?
    Milos Ivanovic
    @imilos
    Are there any guidelines of how to run Rabix Composer as a standard web app (without Electron)? Perhaps already available build?
    Fran Sánchez
    @fj-sanchez
    Hello! I am completely new to CWL and Rrabix Composer. I would like to have a central repo for the tool wrappers which I guess is equivalent to the "connect to platform" option, is this part also open source?
    Kaushik Ghose
    @kaushik-work
    Hi @imilos there are no such guidelines unfortunately.
    Hi @fj-sanchez one public tool repo is Dockstore, and many people upload their tools to github. There is no interface from Rabix Composer to these two endpoints, but you can easily download a tool you need from them onto your local file system and point Composer to them.
    Michael R. Crusoe
    @mr-c
    @imilos Aravdos forked Rabix Composer to make that happen (and to connect to their platform): https://github.com/curoverse/composer
    Fran Sánchez
    @fj-sanchez
    @kaushik-work but is there any spec for implementing the "platform" (I don't know how to call it) side of rabix composer?
    For example, I would like to have also the ability to deploy workflows in my compute from the UI
    Kaushik Ghose
    @kaushik-work
    Hi @fj-sanchez Composer talks to the SBG platform via it's public API. This API is of course specialized to the SBG platforms.
    Fran Sánchez
    @fj-sanchez
    Thanks @kaushik-work I'll try to find the API or read the code to understand it. I guess I could follow the download/sync folder route or try to implement this interface in a small server
    Are you aware of any openApi spec for this?
    Kaushik Ghose
    @kaushik-work
    Hi @fj-sanchez the SBG platform API specs are here https://docs.sevenbridges.com/page/api