These are chat archives for nextflow-io/nextflow

12th
Jul 2018
Rad Suchecki
@rsuchecki
Jul 12 2018 02:10
Is there a way of accessing properties of files declared as input? For example I would like to :
process foo {
  input:
    set (file f1, file f2) from inChannel

  when:
    !f1.isEmpty() && !f2.isEmpty()
Paolo Di Tommaso
@pditommaso
Jul 12 2018 06:31
@gnetsanet please report a bug
@rsuchecki always the same problem and workaround nextflow-io/nextflow#731
Rad Suchecki
@rsuchecki
Jul 12 2018 07:48
I'll print this one @pditommaso and pin it somewhere in sight :-)
Paolo Di Tommaso
@pditommaso
Jul 12 2018 07:49
LOL
Tim Dudgeon
@tdudgeon
Jul 12 2018 09:03

OK, so I sort of go this directory traversal working like this (slightly contrived example):

input = Channel.fromPath( '*/input.smi' )
receptor = Channel.fromPath( '*/receptor.pdb')

process listThemAll {

  publishDir './'

  input:
  file input
  file receptor

  output:
  file 'results'

  """
  cat $input $receptor > results
  """
}

My problem now is with the output. As it stands the output file for every directory has the same name so the publishDir option won't work correctly.
How can I either capture the directory name (the wildcard) so that I can name the output file in a unique manner or include it in the publishDir spec?

Paolo Di Tommaso
@pditommaso
Jul 12 2018 09:05
where input, receptor or both ?
Tim Dudgeon
@tdudgeon
Jul 12 2018 09:13
not sure what you mean. I'm wanting to either generate a unique name by including the dir name (the bit that matches the wildcard in the inputs) in the output file name (e.g. dirname_results) or to use that dirname as part of the publishDir expression.
What happens for me in the above example is that for each directory that is matched by the widlcard a process is excecuted and the work dir for that process contains the input.smi and receptor.pdb files from that directory (through some nextflow black magic!)
Paolo Di Tommaso
@pditommaso
Jul 12 2018 09:14
you need to retain the parent path name eg.
input = Channel.fromPath( '*/input.smi' )
receptor = Channel.fromPath( '*/receptor.pdb')
                          .map { file-> tuple(file.parent.name, file) }

process listThemAll {

  publishDir "$name"

  input:
  file input
  set val(name), file(receptor) from  receptor

  output:
  file 'results'

  """
  cat $input $receptor > results
  """
}
Tim Dudgeon
@tdudgeon
Jul 12 2018 09:23
Great. That works. Thanks!
Francesco Strozzi
@fstrozzi
Jul 12 2018 10:37
Question: why changing the output definition and adding optional true triggers a new execution of already completed processes using -resume?
Paolo Di Tommaso
@pditommaso
Jul 12 2018 10:38
not sure the two things are related
Francesco Strozzi
@fstrozzi
Jul 12 2018 10:39
ok so NF per se does not flag a process as changed if I add optional true to the outputs
Paolo Di Tommaso
@pditommaso
Jul 12 2018 10:47
it should not, if it does please open a bug
Francesco Strozzi
@fstrozzi
Jul 12 2018 10:51
Ok
But I think the problem was elsewhere
KochTobi
@KochTobi
Jul 12 2018 13:16
Hi there,
I am currently trying to run the nf-core/rnaseq nextflow workflow on aws batch. I configured a custom ami (java 1.8, docker, nf-utils, git and awscli), created a job queue and the compute environment.
Nextflow now creates jobs which are in the Runnable state but my compute enivronment becomes INVALID with the following error:
SpotFleetRequestConfig.IamFleetRole
The compute environment is configured as MANAGED and to choose optimal Instance types. There is no EC running so the limit should not be reached. I'm not able to find the problem. Does anyone have an idea where to look? Thanks a lot, Tobi
Paolo Di Tommaso
@pditommaso
Jul 12 2018 13:45
are you able to run a basic Batch job without using NF?
Tim Dudgeon
@tdudgeon
Jul 12 2018 13:51

I'm using the groupBy operator to group a set of files:

results.groupBy { f ->
    f.name[0]
}

This is working and returning a LinkedhashMap. Now I know this should be easy but how to I then flatten this out into a List of tuples that I can work through.
I was thinking that .flatten() would do this, but alas no.

KochTobi
@KochTobi
Jul 12 2018 14:06
@pditommaso no I also tried to run a simple job (tutorial example Hello world with splitting letters). It just gets added to the job queue and is stuck in RUNNABLE.
The compute environment changes its Status from VALID to INVALID for this example also.
Alan B. Christie
@alanbchristie
Jul 12 2018 14:12
I was just tripped-up by the executor.queueSize on my 144-core run. I wandered why the throughput seemed to get throttled at 100! What's the recommended way of using this for a cloud deployment, especially one that can grow dynamically? It would be really cool if the master instance could increase the queue size to match the largest number of observed cores in the cluster (for me, it doesn't have to downsize). Can it do this? If not can the queue size be changed at run-time?
Paolo Di Tommaso
@pditommaso
Jul 12 2018 14:24
@tdudgeon don't use groupBy, is mostly useless
@KochTobi there's a problem with the config of your compute env, make sure you are able to run a basic job WITHOUT using NF
Paolo Di Tommaso
@pditommaso
Jul 12 2018 14:30
@alanbchristie increase it to a bigger number, generally is not a problem
Francesco Strozzi
@fstrozzi
Jul 12 2018 14:39
@KochTobi it seems there is something wrong with the configuration of your permissions for Batch and the computing environment.
It seems that your computing environment is not allowed to launch Spot Fleet, I can suggest you create a simple Computing Environment from the AWS Batch dashboard using the wizard to create all the IAM roles that are necessary to create EC2 instances and run the jobs
then test it with a very simple Job with NF
Raoul J.P. Bonnal
@helios
Jul 12 2018 15:17
When NF runs in resume mode, are the channels cached or not ? If they are cached can I invalidate the channel ? My channel reads lines from a file (sampleid, filepath) I noticed that if i add a new line the channels does not read it anymore.
Paolo Di Tommaso
@pditommaso
Jul 12 2018 15:20
No, generally operator attached to channels are just re-executed (except splitXxx)
Raoul J.P. Bonnal
@helios
Jul 12 2018 15:58
@pditommaso ok tanks I'll look into my script, sure is something on my side.
Btw I can confirm that aws batch is working smoothly with NF
Francesco Strozzi
@fstrozzi
Jul 12 2018 17:19
:+1:
Alan B. Christie
@alanbchristie
Jul 12 2018 19:45
OK, so I have over 500 cores spread over 7 machines (7x72) and for the first hour or so the machines have been running full speed. Now, about 3 hours into what should be a 24-hour run, with millions of jobs still to run the machine load (which should be 72) has collapsed to about 5 or 6 on each machine. I have hundreds of cores doing nothing and its turning out to be rather expensive. How can I find out where the bottle-next is and why Nextflow isn't launching jobs - each machine should be running with a load factor of nearly 72!!!
Félix C. Morency
@fmorency
Jul 12 2018 20:03
@alanbchristie maybe you can take a subset of your data, run your pipeline and plot the dag using nextflow run -with-dag my_dag.png ...
Alan B. Christie
@alanbchristie
Jul 12 2018 20:18
@fmorency Thanks, it's just one simple processing job replicated about 1.6 million times over an ignite cluster on AWS with seven 72-coree machines. The machines were maxed out for the first hour - everything running fine but for some reason they're now all more than 90% idle with over a million jobs still to be run but only 3 or 4 running on each node. I've got to kill the run because the costs are going to go through the roof on this. I have tested a large subset on 2 nodes and it all runs perfectly well with expected throughput. But this grand run of 1.6 million identical tasks over 7 machines just seems to have decided to grind to a halt.
Alan B. Christie
@alanbchristie
Jul 12 2018 20:25
@fmorency Interestingly the javaprocess (Nextflow) on the designated master is running at full bore (100% of one of the cores). I hadn't noticed this before but is that right? Could it be struggling to keep up in real-time?
The master node itself is also only running with a load factor of 7 when it should be 72. Anyway - need to kill the cluster and takes stats and look at this tomorrow.
Félix C. Morency
@fmorency
Jul 12 2018 20:27
@alanbchristie Paolo could answer best. Did you see any memory issue on your head node? IO issue?
Alan B. Christie
@alanbchristie
Jul 12 2018 20:30
Yeah, they look OK, plenty of idle CPU and memory. These are c5.18xlarge nodes with not much going on. Hay ho. Thanks.
The full bore java process is worrying. It may be nothing, we'll see.
Félix C. Morency
@fmorency
Jul 12 2018 20:34
Np. Ttyl
Alan B. Christie
@alanbchristie
Jul 12 2018 20:57
@pditommaso With regard to executor.queueSize (thanks). Let's say I set it to 576 (72 cores on 8 nodes). Then, after a while I want to add new nodes, say anther 8 to speed things up. To use these fully I need to set the queueSize to 1,152. Do I need to bounce (start & stop) the master to do this?
Len Trigg
@Lenbok
Jul 12 2018 22:02
Hi, I am trying to run a pipeline in AWS Batch. My AMI has an EFS volume attached containing resources such as reference genome etc but this is not visible in the docker container. Is there a way to ensure the Job Definition that gets created sets up the volume?
(I tried using process.containerOptions = '--volume /resources:/resources' but that didn't work)
Len Trigg
@Lenbok
Jul 12 2018 22:11
@helios or @pditommaso hopefully you have some insight on getting AWS Batch working. (BTW, If i create my own Job Description that has the volume mapping I can access the /resources fine, i think it's just the Job Description that NF makes that can't do the volume mapping). Alternatively how should I supply my external resources like reference / various index files etc?
Len Trigg
@Lenbok
Jul 12 2018 23:36
In AWS console I can take the Job Description that NF makes, and manually create a new revision that has the volume/mount configured. NF will use that new revision for subsequent jobs and they run fine accessing the data off that volume. We just need NF to allow creating that volume setup itself