which is why i was interested in funnel. if there was k8 support, we could probably use it as is
Adding basic support for k8s would be fairly straightfoward. The main hurdle is that there's potentially a docker-in-docker situation, which brings weird quirks sometimes.
Usually Funnel has a worker which calls out to docker to start containers, so it's sort of at odds with a kublet.
k8 schedules ever task a regular
Pod
I'll try to write up a plan this week, and we'll go from there. If you're interested, the compute backends are here: https://github.com/ohsu-comp-bio/funnel/tree/master/compute
They're fairly simple, although we're working on adding better integration for cancel/monitoring/etc.
ok great
Hi Im trying to run whole genome cwl on a small data using rabix which has SBG Prepare Intervals
My interval bed looks like this
chr1 1 249250621
chr2 1 243199373
It throws error evaluating the expression.
Any help
My interval bed looks like this
chr1 1 249250621
chr2 1 243199373
It throws error evaluating the expression.
Any help
I'm having some trouble using the TES back end with SE/Ceph. I've tested that funnel is able to process a simple request with S3 using curl already.
When I try to do the same with ./rabix having modified the config I get an incomprehensible error:
$ ./rabix -c ./config examples/dna2protein/dna2protein.cwl.json examples/dna2protein/inputs.json
[2018-05-05 08:46:37.019] [ERROR] EventProcessor failed to process event InitEvent [value={input_file=FileValue [size=null, path=/private/var/tmp/rabix-cli-1.0.5/examples/dna2protein/data/input.txt, location=null, checksum=null, secondaryFiles=[], properties={sbg:metadata=null}]}, rootId=e6e70855-e5b1-42c3-99be-49a61956fed1, config=null].
java.util.NoSuchElementException: null
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1444) ~[na:1.8.0_161]
at java.util.HashMap$KeyIterator.next(HashMap.java:1466) ~[na:1.8.0_161]
at org.rabix.engine.service.impl.BackendServiceImpl.sendToExecution(BackendServiceImpl.java:258) ~[rabix-cli.jar:na]
at org.rabix.engine.status.impl.DefaultEngineStatusCallback.onJobReady(DefaultEngineStatusCallback.java:29) ~[rabix-cli.jar:na]
at org.rabix.cli.status.LocalBackendEngineStatusCallback.onJobReady(LocalBackendEngineStatusCallback.java:69) ~[rabix-cli.jar:na]
at org.rabix.engine.status.impl.DefaultEngineStatusCallback.onJobsReady(DefaultEngineStatusCallback.java:35) ~[rabix-cli.jar:na]
at org.rabix.engine.service.impl.JobServiceImpl.handleJobsReady(JobServiceImpl.java:242) ~[rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.processReadyJobs(EventProcessorImpl.java:166) [rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$doProcessEvent$3(EventProcessorImpl.java:113) [rabix-cli.jar:na]
at org.rabix.engine.store.memory.InMemoryRepositoryRegistry.doInTransaction(InMemoryRepositoryRegistry.java:92) ~[rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.doProcessEvent(EventProcessorImpl.java:107) [rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$null$1(EventProcessorImpl.java:91) [rabix-cli.jar:na]
at org.rabix.engine.metrics.impl.MetricsHelperImpl.time(MetricsHelperImpl.java:78) ~[rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$start$2(EventProcessorImpl.java:91) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
EventProcessor failed to process event:
InitEvent [value={input_file=FileValue [size=null, path=/private/var/tmp/rabix-cli-1.0.5/examples/dna2protein/data/input.txt, location=null, checksum=null, secondaryFiles=[], properties={sbg:metadata=null}]}, rootId=e6e70855-e5b1-42c3-99be-49a61956fed1, config=null]In config I've changed:
backend.embedded.types=TES
tes.client_scheme=http
tes.client_host=172.27.18.81
tes.client_port=8000
tes.storage_base=s3://cog.sanger.ac.uk/funnel-bucket/rabixand added:
s3.endpoints= s3://cog.sanger.ac.uk/
s3.protocol=HTTPS
s3.access_key=...
s3.secret_key=...
s3.path_style_access=true
s3.signer_override=AWS3SignerType
@keiranmraine sorry I haven't been monitoring gitter lately. I think its a config issue. First of all the s3 config values should follow the format:
s3.<some arbitrary name>.<option> (e.g. s3.ceph.endpoints=s3://cog.sanger.ac.uk/). Second I think your signer type should be either AWS4SignerType or S3SignerType.
Hello I found small typo in Rabix Composer Document http://docs.rabix.io/introduction-to-tool-wrapping
Where is good place to feedback ?
Where is good place to feedback ?
:point_up: solved -- I needed to install the complete JDK from here: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Hi, I am trying to play with Rabix on MacOS. I have wrapped a simple java application which is stored as a docker image at the AWS ECR. Although I can pull the image from the command line, when i execute the workflow from Rabix I get the error that it cannot pull the image. What are the steps/requirements to be able to pull from ECR?
Hi @fj-sanchez one public tool repo is Dockstore, and many people upload their tools to github. There is no interface from Rabix Composer to these two endpoints, but you can easily download a tool you need from them onto your local file system and point Composer to them.
@imilos Aravdos forked Rabix Composer to make that happen (and to connect to their platform): https://github.com/curoverse/composer
For example, I would like to have also the ability to deploy workflows in my compute from the UI
Are you aware of any openApi spec for this?