These are chat archives for nextflow-io/nextflow

9th
Jun 2017
Paolo Di Tommaso
@pditommaso
Jun 09 2017 07:43
@amacbride is should be just a warning message, if it reports an error please open an issue on GH including the error stack trace
amacbride
@amacbride
Jun 09 2017 13:15
Will do. (It's in HistoryFile.groovy, the IllegalStateException in the locking closure.) Will re-run and paste in the exact stack trace.
tonywang0613
@tonywang0613
Jun 09 2017 16:03
Hi, I try to download file from s3 in the process, like this, aws s3 cp $reference ./
gives me an error
File "xx" is out of the scope of process working dir
What's that mean?
Paolo Di Tommaso
@pditommaso
Jun 09 2017 16:06
please report the exact code and exact error message
tonywang0613
@tonywang0613
Jun 09 2017 16:07
process downRefAws{
tag {"download: "+reference}
echo true
    input:
        val(reference) from params[params.refversion].aws.values()

    output:
        file(reference) into downloadFiles

    """
    aws s3 cp $reference ./
    """

}

ERROR ~ Error executing process > 'downRefAws (download: s3://referencefolder/xxx)'

Caused by:
File //referencefolder/xxx is out of the scope of process working dir: /mnt/work/57/5ad6b2d5b15e733c24a56c06a7e698

Paolo Di Tommaso
@pditommaso
Jun 09 2017 16:10
formatting correctly would help
tonywang0613
@tonywang0613
Jun 09 2017 16:10
I store all the samples on aws, we are now write a flow to first download all the files to ec2 instance
Paolo Di Tommaso
@pditommaso
Jun 09 2017 16:11
is that the exact s3 path ?
tonywang0613
@tonywang0613
Jun 09 2017 16:12
what is the good way to download files from s3, using the process?Do you have some working example?
yes
Paolo Di Tommaso
@pditommaso
Jun 09 2017 16:13
I'm not understanding why reference is s3://referencefolder/xxx and instead is reported as //referencefolder/xxx in the error message
tonywang0613
@tonywang0613
Jun 09 2017 16:18
maybe that's because output: file(reference) , the reference variable is the full path
Paolo Di Tommaso
@pditommaso
Jun 09 2017 16:19
can you please copy & paste the complete error message ?
tonywang0613
@tonywang0613
Jun 09 2017 16:21
Now it works, my mistake, in the output, I use same variable reference which is the full path s3://xx
I changed to file '*' into download files , works now
thanks
Phil Ewels
@ewels
Jun 09 2017 23:47
Note that NF has built in support for s3, so no need for a s3 cp command in a process. You can just specify the s3 path for the input file names for a regular process and NF will automatically fetch the files.