These are chat archives for nextflow-io/nextflow

25th
Oct 2016
Jason Byars
@jbyars
Oct 25 2016 20:40
Has something changed recently about using s3 buckets for channel input? I swear this used to work, but now I get an
ERROR ~ Relative path cannot be made absolute: foo/bar/tmap
params.bamDir = 's3://foo/bar/tmap/*.bam'
Channel
    .fromPath( "${params.bamDir}" )
    .ifEmpty { error "Cannot find any bam files matching: ${params.bamDir}" }
    .set { bamfiles }
Paolo Di Tommaso
@pditommaso
Oct 25 2016 20:45
Please include the stack trace you can find in the log file
Jason Byars
@jbyars
Oct 25 2016 20:49
come to think of it, I don't remember if wildcards worked with s3 objects in the past. Here's the trace
`` Oct-25 20:45:08.845 [main] DEBUG nextflow.cli.Launcher - $> /usr/local/bin/nextflow run nextflow-scripts/legos/unleashthekraken.nf Oct-25 20:45:08.976 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 0.22.3 Oct-25 20:45:08.987 [main] INFO nextflow.cli.CmdRun - Launchingnextflow-scripts/legos/unleashthekraken.nf` [berserk_mayer] - revision: 754b0516d7
Oct-25 20:45:09.299 [main] DEBUG nextflow.Session - Session uuid: 49f28bc6-d63d-428e-90c9-5adbae9b27a3
Oct-25 20:45:09.299 [main] DEBUG nextflow.Session - Run name: berserk_mayer
Oct-25 20:45:09.301 [main] DEBUG nextflow.Session - Executor pool size: 32
Oct-25 20:45:09.320 [main] DEBUG nextflow.cli.CmdRun -
Version: 0.22.3 build 4095
Modified: 12-10-2016 09:31 UTC
System: Linux 3.13.0-93-generic
Runtime: Groovy 2.4.7 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_101-b13
Encoding: UTF-8 (UTF-8)
Process: 4050@ip-172-31-22-229 [172.31.22.229]
CPUs: 32 - Mem: 240.2 GB (15 GB) - Swap: 0 (0)
Oct-25 20:45:09.345 [main] DEBUG nextflow.Session - Work-dir: /data2/work [ext2/ext3]
Oct-25 20:45:09.433 [main] DEBUG nextflow.Session - Session start invoked
Oct-25 20:45:09.440 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Oct-25 20:45:09.441 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
Oct-25 20:45:09.599 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Oct-25 20:45:09.624 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Oct-25 20:45:09.642 [main] DEBUG nextflow.file.FileHelper - AWS S3 config details: {}
Oct-25 20:45:11.198 [main] DEBUG nextflow.Channel - files for syntax: glob; folder: atg-datalake-test/acctumorRNA.g1k/tmap/; pattern: *.bam; options: null
Oct-25 20:45:11.199 [main] DEBUG nextflow.Session - Session aborted -- Cause: Relative path cannot be made absolute: atg-datalake-test/acctumorRNA.g1k/tmap
Oct-25 20:45:11.226 [main] ERROR nextflow.cli.Launcher - @unknown
java.lang.IllegalStateException: Relative path cannot be made absolute: atg-datalake-test/acctumorRNA.g1k/tmap
at com.upplication.s3fs.S3Path.toAbsolutePath(S3Path.java:435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:79)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at nextflow.extension.FilesEx.complete(FilesEx.groovy:1385)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMeth
Jason Byars
@jbyars
Oct 25 2016 21:01
If the syntax seems valid I can look into the IAM role settings
Paolo Di Tommaso
@pditommaso
Oct 25 2016 21:10
I can't check now, but I think a workaround is to use a path like this s3:///some/dir
Jason Byars
@jbyars
Oct 25 2016 21:10
NP, I'll test it
Paolo Di Tommaso
@pditommaso
Oct 25 2016 21:11
I mean with three slashes
Jason Byars
@jbyars
Oct 25 2016 21:13
winner
I should have results in 5 minutes. BTW, congratulations on getting the cloud autoscaling integrated.
Paolo Di Tommaso
@pditommaso
Oct 25 2016 21:35
:+1:
Jason Byars
@jbyars
Oct 25 2016 21:35
@pditommaso ok, the extra slash solution is working.