These are chat archives for nextflow-io/nextflow

24th
Jan 2018
Francesco Strozzi
@fstrozzi
Jan 24 2018 09:12
Hi guys, anyone expericing problems on AWS S3 with the publishDir directive ? I got an error from the AWS SDK saying that the COPY operation failed for objects larger than 5GB
WARN  nextflow.processor.PublishDir - Failed to publish file: /nextflow-tmp-fst/8f/1ec21b33cd6e660f74010d18190dcf/table.txt; to: /final/table/table.txt
com.amazonaws.services.s3.model.AmazonS3Exception: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest; Request ID: F7CA17FFB876EC35
I wonder if this is due to the COPY operation issued by the publishDir directive that is not using multipart uploads
since the 5 GB limit is usually for non-multipart uploads
Paolo Di Tommaso
@pditommaso
Jan 24 2018 09:43
multipart upload is supposed to be used for files > 5MB
enable the logging how I was suggesting in the issue you opened
Francesco Strozzi
@fstrozzi
Jan 24 2018 10:01
done