Pod
I'm having some trouble using the TES back end with SE/Ceph. I've tested that funnel is able to process a simple request with S3 using curl already.
When I try to do the same with ./rabix
having modified the config I get an incomprehensible error:
$ ./rabix -c ./config examples/dna2protein/dna2protein.cwl.json examples/dna2protein/inputs.json
[2018-05-05 08:46:37.019] [ERROR] EventProcessor failed to process event InitEvent [value={input_file=FileValue [size=null, path=/private/var/tmp/rabix-cli-1.0.5/examples/dna2protein/data/input.txt, location=null, checksum=null, secondaryFiles=[], properties={sbg:metadata=null}]}, rootId=e6e70855-e5b1-42c3-99be-49a61956fed1, config=null].
java.util.NoSuchElementException: null
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1444) ~[na:1.8.0_161]
at java.util.HashMap$KeyIterator.next(HashMap.java:1466) ~[na:1.8.0_161]
at org.rabix.engine.service.impl.BackendServiceImpl.sendToExecution(BackendServiceImpl.java:258) ~[rabix-cli.jar:na]
at org.rabix.engine.status.impl.DefaultEngineStatusCallback.onJobReady(DefaultEngineStatusCallback.java:29) ~[rabix-cli.jar:na]
at org.rabix.cli.status.LocalBackendEngineStatusCallback.onJobReady(LocalBackendEngineStatusCallback.java:69) ~[rabix-cli.jar:na]
at org.rabix.engine.status.impl.DefaultEngineStatusCallback.onJobsReady(DefaultEngineStatusCallback.java:35) ~[rabix-cli.jar:na]
at org.rabix.engine.service.impl.JobServiceImpl.handleJobsReady(JobServiceImpl.java:242) ~[rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.processReadyJobs(EventProcessorImpl.java:166) [rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$doProcessEvent$3(EventProcessorImpl.java:113) [rabix-cli.jar:na]
at org.rabix.engine.store.memory.InMemoryRepositoryRegistry.doInTransaction(InMemoryRepositoryRegistry.java:92) ~[rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.doProcessEvent(EventProcessorImpl.java:107) [rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$null$1(EventProcessorImpl.java:91) [rabix-cli.jar:na]
at org.rabix.engine.metrics.impl.MetricsHelperImpl.time(MetricsHelperImpl.java:78) ~[rabix-cli.jar:na]
at org.rabix.engine.processor.impl.EventProcessorImpl.lambda$start$2(EventProcessorImpl.java:91) [rabix-cli.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
EventProcessor failed to process event:
InitEvent [value={input_file=FileValue [size=null, path=/private/var/tmp/rabix-cli-1.0.5/examples/dna2protein/data/input.txt, location=null, checksum=null, secondaryFiles=[], properties={sbg:metadata=null}]}, rootId=e6e70855-e5b1-42c3-99be-49a61956fed1, config=null]
In config I've changed:
backend.embedded.types=TES
tes.client_scheme=http
tes.client_host=172.27.18.81
tes.client_port=8000
tes.storage_base=s3://cog.sanger.ac.uk/funnel-bucket/rabix
and added:
s3.endpoints= s3://cog.sanger.ac.uk/
s3.protocol=HTTPS
s3.access_key=...
s3.secret_key=...
s3.path_style_access=true
s3.signer_override=AWS3SignerType
s3.<some arbitrary name>.<option>
(e.g. s3.ceph.endpoints=s3://cog.sanger.ac.uk/
). Second I think your signer type should be either AWS4SignerType
or S3SignerType
.