.command.*files to my S3 bucket. Literally, all folders in
workdirectory contain only input and output files. I need to collect
.command.logfrom several process and at the moment cannot do that. Has anyone seen similar behaviour? When I ran AWS Batch from my laptop before I remember seeing a couple warnings in
.nextflow.logrelated to S3, but in Tower I don't seem to be able to find
.nextflow.log... When I setup up Tower and role policies on AWS I strictly followed Tower documentation.
Just rerun it from my local laptop on AWS and check
.nextflow.log. There does not seem to be any warnings or errors except for:
WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant
Though I still do not see any of the
.command.* files in my work directory. The pipeline is working with no problem until I specifically ask for
.command.log and then it fails as the file does not exist.
.command*files are written to folders in work directory and the pipeline runs OK until it gets to the process where
.command.logis needed to be copied
task-1.command.err.txtwas empty and when tried to download
Unable to download file: .command.logmessage on the website. I've also checked a corresponding work directory here:
/fusion/s3/my-bucket/scratch/5DQRTBKJsIIay1/bf/71c55281bdbbb44ead372c5acf3746and it was empty.
s3:list. Is that the correct one?
.command.*files in the
workdirectories. Am I missing anything?
maxRetriesto some reasonable value for all of the processes by default? (At the moment my
maxRetriesdepends on the exit code, but as there is not exit code provided by AWS, it should be applied to any reason)
process.errorStrategy = 'retry' process.maxRetries = 5 // or more
-resumeoption in NF Tower, so that the outputs from these initial processes can be reused? Thanks!