These are chat archives for nextflow-io/nextflow

9th
Nov 2018
Rad Suchecki
@rsuchecki
Nov 09 2018 00:27
@srynobio
  • config file: :+1:
  • nf script: instead of withLabel: 'big_mem' use label: 'big_mem'
Maxime Garcia
@MaxUlysse
Nov 09 2018 08:32
@t-neumann Glad to be able to help you
I defintively like this community a lot :-D
Paolo Di Tommaso
@pditommaso
Nov 09 2018 08:33
Excellent Maxime ! :smile:
Also great your post on batch
Maxime Garcia
@MaxUlysse
Nov 09 2018 08:37
It was a simple post, but that was just the information this was missing from every blog post made about this subject, and I did click on the link from the custom AMI Nextflow doc, but didn't understand at the time
Paolo Di Tommaso
@pditommaso
Nov 09 2018 08:37
well done
can you remind me the link ?
Paolo Di Tommaso
@pditommaso
Nov 09 2018 08:39
that's me :)
I was meaning your post
I've added it to the awesome page
Maxime Garcia
@MaxUlysse
Nov 09 2018 08:43
Ahh sorry
:facepalm:
Paolo Di Tommaso
@pditommaso
Nov 09 2018 08:44
no pb
Maxime Garcia
@MaxUlysse
Nov 09 2018 08:44
There was this one also
A little old, but I liked the trouble shooting
Paolo Di Tommaso
@pditommaso
Nov 09 2018 08:45
I can't cope with all your post :)
Maxime Garcia
@MaxUlysse
Nov 09 2018 08:45
:-D
Anthony Underwood
@aunderwo
Nov 09 2018 09:45
@MaxUlysse thanks for your post. When you specify an increased volume size do you increase the 2nd volume that is there in the default launch screen or increase the root partition? In my experience the 2nd volume isn't auto-mounted
@MaxUlysse is there a risk that allow Docker access to 500gb that multiple processes running on the same instance could consume more storage than is available. e.g 500Gb volume but 2 docker processes each allowed to access 500Gb
Anthony Underwood
@aunderwo
Nov 09 2018 10:26
It appears that although the extra space is not recognised as mounted by the df command, docker sees it as available as shown by the docker info command
Maxime Garcia
@MaxUlysse
Nov 09 2018 10:49
Definitily, because, I had the space listed when I did docker info | grep -i data but not when I used docker info | grep -i base, so this --storage-opt dm.basesize=500GB was defintively the key
KochTobi
@KochTobi
Nov 09 2018 10:55
according to AWS support on ecs docker storage you can verify your storage size also using lsblk.
Maxime Garcia
@MaxUlysse
Nov 09 2018 10:56
Not sure on how to use that
KochTobi
@KochTobi
Nov 09 2018 10:58
I thought @aunderwo wasn't sure whether his attempt succeeded. Maybe I misread
Anthony Underwood
@aunderwo
Nov 09 2018 11:06
@KochTobi lsblk shows the extra storage but it isn't mounted as shown by df
@KochTobi however docker recognises this storage space
I tested by launching a couple of instances with different of volume sizes and docker info | grep -i data reflected this
Alexander Peltzer
@apeltzer
Nov 09 2018 12:23
Trying to do something like this here:
memory = { check_max( bam.size().toGiga() * 5.GB * task.attempt, 'memory')}
but it complains about Exception in thread "Task submitter" groovy.lang.MissingMethodException: No signature of method: java.lang.Long.toGiga() is applicable for argument types: () values: []
PhilPalmer
@PhilPalmer
Nov 09 2018 12:36
The error message also says Possible solutions: notify(), toLong(), toFloat(), toString(), wait(), toString() which makes me think theres a problem with the type of bam.size() that means you can't call toGiga() on it
Alexander Peltzer
@apeltzer
Nov 09 2018 12:40
This seems to work better, but scheduler rejects it
  memory = { check_max( ( bam.size()/1024/1024/1024)  * 5.GB  * task.attempt, 'memory')}
qsub: submit error (Illegal attribute or resource value for Resource_List.mem)
.command.runsays: #PBS -l mem=28.7gb
(which is correct for a file for ~5GB in size)
Paolo Di Tommaso
@pditommaso
Nov 09 2018 12:56
toGiga is not defined on plain numbers
but could nice to add it
Alexander Peltzer
@apeltzer
Nov 09 2018 12:56
aah there we go :-)
Alexander Peltzer
@apeltzer
Nov 09 2018 13:04
Thanks @pditommaso
Paolo Di Tommaso
@pditommaso
Nov 09 2018 13:06
:v:
Max Hargreaves
@DiDeoxy
Nov 09 2018 19:41
@apeltzer thanks for pointing me in the direction of that, i'd seen it before but hadn't given it enough time, full of lots of useful methods. The problem turned out to be a problem with alpine linux lacking any installed fonts, fontconfig thus couldn't find them when called by java to produce the fastqc report. Annoyingly this all happened quietly when running the nextflow script using the docker container, this problem did not cause a crash report, but failed quietly producing partial output. I had to manually run fastqc inside the container to find out what was going on. Thanks for the help though!
Alexander Peltzer
@apeltzer
Nov 09 2018 20:10
👍👍👍