nextflow-io/nextflow

Hello @pditommaso and the rest of the nextflow community,

I am a big fan of the approach taken by Nextflow, and I am actively pushing people to adopt it as the principal data flow control language for our genetics community at the major pharmaceutical firm where I work. Our local high-performance UNIX cluster is entirely inadequate to the multitude of algorithms that we need to run, however, so running on a cloud provider is essential. Lately I’ve run up against a challenge when running with the AWS Batch mode, and I’m hoping that someone can suggest a workaround.

In our module-based, DSL-2 driven environment I’ve implemented a structure that I thought would be flexible, using some initial processes to gather (and check for the existence) of files, and then making symbolic links to those files wherever they originate so that I could access them in subsequent steps managed by Nextflow. I’ve been using an approach along these lines

process gather_essaential_files{
    input:
        val genotype_bim_name
        val genotype_bed_name
        val genotype_fam_name
        val working_dir    
output:
        path("$genotype_bim_name"), optional: true, emit: geno_bim
        path("$genotype_bed_name"), optional: true, emit: geno_bed
        path("$genotype_fam_name"), optional: true, emit: geno_fam

script:
        """
           ln -s $working_dir/$genotype_bim_name $genotype_bim_name
           ln -s $working_dir/$genotype_bed_name $genotype_bed_name
           ln -s $working_dir/$genotype_fam_name $genotype_fam_name      
        """

After a process such as this one I can treat all the files in subsequent processes as if there local, and the logic all seems very clean. I find that this approach runs wonderfully in my local environment, and making those file links is naturally super fast. When I try to run it on AWS BATCH, however, I don’t get file links, but instead those files each get uploaded to the S3 bucket where next flow is managing its process-specific work directories. This uploading/coping approach would make sense to me if we were dealing with s3-based data (given that S3 isn’t mounted like a real file system), but we aren’t. I’m using AWS FSx to allow access to those files, and therefore they act like any other local files. Here’s my question: is there some parameter/trick that I can use that would allow me to actually create symbolic links instead of copying the files every time they are referenced? We use lots of large data files, and we want to run in a massively parallel environment, and if we are forced to copy those data files every time they are needed by process and the whole approach becomes less viable. I would love a configuration parameter that would allow me to say “link-instead-of-copying” in the context of the AWSBATCH profile. Does anybody have a recommendation for me?

Thanks for any insights, Ben