These are chat archives for nextflow-io/nextflow
I have my pipeline working in multiple environments: local, cluster, single AWS instance. Now I want to get it working with an ignite cluster in AWS. I'm having a hard time understanding how ignite pairs the worker to master node. AWS does not support multicast so I understand this may not be straightforwad. In order to get it to work I have to do these steps:
cluster.join = "path:/mnt/efs/joincast.
nextflow node -bg -cluster.join path:/mnt/efs/joincast
nextflow run <your pipeline> -process.executor ignite
IS this the recommended approach? Seems a bit tedious.
nextflow cloud createto start a cluster but my experience is that the state is not ready to deploy jobs since the master/worker nodes are not paired
workDirwas necessary. In my case I am using EFS. Im not sure if 1)NF is expected to run this way or 2) Since EFS mounts are not taking place on workers that my cluster is failing to communicate
sharedStorageMountspecified in your
nextflow.config? I seem to recall having a hard time getting
nextflow cloudworking with my own AMI, but then having much better luck with the one @pditommaso specifies in the tutorials.
nextflow cloudand EFS
2) Since EFS mounts are not taking place on workers that my cluster is failing to communicate
sharedStorageMount? Need to understand if this is an issue related to environment or Nextflow design.
.command.runfor that process that is 22MB in size! This runs fine locally, but
slurmchokes on a script that big (their limit is apparently 4MB). Ideally I'd just store all of these files in a directory somewhere and pass the directory around, but I'm not quite sure what the best way to do that would be. If I pass all of the files into a process and return a directory then I have the same problem, but if I store things from a channel (maybe using
reduce) then I'm not sure where to put the directory... Any ideas?
reducethat returns a dir...
I was having issues with running a nextflow cloud cluster, but I got it working after @pditommaso mentioned the need for an EFS.
Here are the four things I did to get a nextflow cloud cluster working:
I think this functionality makes nextflow extremely useful.
nextflow cloud create clustername -c 3), is there a way to programmatically answer 'y'?
Please confirm you really want to launch the cluster with above configuration [y/n]