These are chat archives for nextflow-io/nextflow

12th
Oct 2017
Paolo Di Tommaso
@pditommaso
Oct 12 2017 10:56
Just uploaded version 0.26.0-beta4 with a lot of changes
  • Added support for AWS Batch #475
  • Added support for AWS IAM role #422
  • Added Experimental support for Java 9 #462
  • Added transpose operator #440
  • Added join operator #460
  • Added html report (-with-report option)
  • Improved merge operator #458
  • Fixed HEAD is detached error #389
  • Fixed Include container name when calculating cache key #443
  • Fixed Invalidate process cache when a referenced external script is modified #413
  • Fixed Execution failure is not reported in the trace file when retry error strategy is used #404
  • Fixed "when" directive should not allow staging of input files for a process, if the criteria are not met #424
  • Fixed Failure during foreign files download cause execution to stop abruptly #295
  • Fixed The parameter groupName cannot be used with the parameter subnet #465
  • Fixed unable to download from gitlab.com public repository #457
  • Fixed Print work directory when a task fails #445
  • Fixed Misleading command line parameter type conversion #434
  • Removed support support for Java 7 #385
  • Removed deprecated Cirrus/ClusterK executor
  • Removed deprecated support for DNAnexus #386
Note: from this version Java 8 is required
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:01
Nice!
I'm having issue with what I believe is a latency problem with my cluster:
a process finishes, and nextflow crashes because it cannot find the produced files on disk (sometimes the work folder isn't present on disk). A few seconds later the files (or the work folder) is on disk. The cluster just took some time to write back on the local disk.
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:04
ugly
how is the error message ?
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:05
Has there been some changes between 0.22.0 and 0.25.6 ? It could be totaly irrelevant and due to the cluster getting slower lately
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:05
what batch scheduler are you using? slurm ?
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:05
yes SLURM
looking for the exact error message.. (argh not copied in my issue tracker)
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:07
well changes there are but hard to say if they are relevant
you should use the old version and verify that it works ..
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:08
Yep I was hesitating between roll back or adding "sleep 120" at the end of the very short processes (it seems to "fix" the problem)
[Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Missing output file(s) `blabla.pdf` expected by process `trio_excap_qc_join (1)`
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:12
mm, I don't think is a problem with the new version
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:13
The PDF file is an output of that proccess.
And it is on disk
At least now, when I check :)
Is this latency element considered in nextflow?
(not that it I'm saying should, I'm just curious)
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:18
yes, it is. NF waits for a .exitcode file, after that assumes the other must be available ..
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:28
I just stat both the .exitcode and my output file, and they show both the same time, 09:23:13.0000000.
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:31
that's the creation time, but I guess the problem is that the NF process, running on a different node, does not see it at that time
is your file system NFS ?
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:32
nope.
beeGFS
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:34
try to see with your admins if there's a way to mitigate this problem. I'm not an expert with beegfs
however keep me updated
Hugues Fontenelle
@huguesfontenelle
Oct 12 2017 11:35
Thanks for reading. I'll try to make a smaller script to see the modification/change times in stat.
Anthony Underwood
@aunderwo
Oct 12 2017 11:45
@pditommaso have you run into problems on AWS where hanging docker images take up all the disk space?
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:50
no, it could happen ..
actually yes, not that I'm thinking to that, but it was GCP
Anthony Underwood
@aunderwo
Oct 12 2017 11:53

GCP?

We have found that this has been an issue with our workflows requiring a 200Gb boot disk at /

Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:53
google cloud platform
Anthony Underwood
@aunderwo
Oct 12 2017 11:54
We would like to run a cronjob that cleans up using docker system prune --volumes
However this comes in in Docker 17.06.1. On the ami image it's 17.03.1
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:55
if they are long running instances, yes you need that
Anthony Underwood
@aunderwo
Oct 12 2017 11:55
docker system prune -a deleted images meaning that you have to keeping on pulling the images. Therefore --volumes would be required
How can we update the ami to have latest nextflow and updated Docker?
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:57
well NF is not a problem
curl get.nextflow.io | bash
Anthony Underwood
@aunderwo
Oct 12 2017 11:58
no I see that's at latest stable
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:58
?
Anthony Underwood
@aunderwo
Oct 12 2017 11:58
can you remind me how you specify a beta release
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:58
ahh
export NXF_VER=0.26.0-beta4
or any other version
to update docker frankly I don't know
Anthony Underwood
@aunderwo
Oct 12 2017 11:59
@pditommaso :thumbsup:
How did you build the ami?
Paolo Di Tommaso
@pditommaso
Oct 12 2017 11:59
however docker system prune is little more
alias docker-cleanup='X=$(docker ps -aq) && [[ $X ]] && docker rm $X; X=$(docker images -f dangling=true -q) && [[ $X ]] && docker rmi $X'
  1. launch a VM instance, 2. install your stuff, 3. Create image command in the dashboard, 4. done
Anthony Underwood
@aunderwo
Oct 12 2017 12:01
OK. Do you have a list of the commands you ran to install stuff for the AMI that works with Nextflow so we can try to fork from there please
Paolo Di Tommaso
@pditommaso
Oct 12 2017 12:02
I don't have sorry
but I install just java, docker and nfs-utils
eventually git
Anthony Underwood
@aunderwo
Oct 12 2017 12:02
and nextflow :)
how about the ignite stuff?
is that built in into nextflow?
Paolo Di Tommaso
@pditommaso
Oct 12 2017 12:03
that's self installed by NF cloud create
is that built in into nextflow?
yes
Anthony Underwood
@aunderwo
Oct 12 2017 12:04
Thanks we'll give it a go
Paolo Di Tommaso
@pditommaso
Oct 12 2017 12:04
:+1:
Rickard Hammarén
@Hammarn
Oct 12 2017 13:18
I got the question from a user of our pipeline if there is any compact documentation for someone how just wants to use Nextflow for executing pipelines and not develop anything themselves? I think the user would want basic info on how Nextflow submits job, ie. whether to just run it on a login node in the cluster or as an sbatch job.
I know this info is in the general docs but it's kinda hidden in the development documentation
Félix C. Morency
@fmorency
Oct 12 2017 13:28
@pditommaso fyi one thing I am not sure and couldn't test yesterday is if we can do docker ... --entrypoint /usr/bin/env bash ...
Paolo Di Tommaso
@pditommaso
Oct 12 2017 16:40
@Hammarn that would be useful, unfortunately at this time the only docs it the one available online. say him/her we will be happy to answer to any doubt
@fmorency you should build and try yourself
actually I'm a bit concerned about that
what's the bash location on FreeBSD ?
Félix C. Morency
@fmorency
Oct 12 2017 16:41
/usr/local/bin/bash iirc
Paolo Di Tommaso
@pditommaso
Oct 12 2017 16:41
!
Félix C. Morency
@fmorency
Oct 12 2017 16:41
I don't have my FreeBSD laptop here
I can check tonight
Félix C. Morency
@fmorency
Oct 12 2017 16:48
I just tested and --entrypoint /usr/bin/env bash doesn't work as I feared
Paolo Di Tommaso
@pditommaso
Oct 12 2017 16:48
yes, actually there makes no sense
Félix C. Morency
@fmorency
Oct 12 2017 16:49
Is there something wrong with calling --entrypoint bash instead?
Paolo Di Tommaso
@pditommaso
Oct 12 2017 16:49
because Docker images are supposed to be linux
yes, that should be remain as before
Félix C. Morency
@fmorency
Oct 12 2017 16:49
right
as before == /bin/bash. I would change it to bash
Paolo Di Tommaso
@pditommaso
Oct 12 2017 16:50
no, the docker stuff it's quite tricky
Félix C. Morency
@fmorency
Oct 12 2017 16:50
Oh okay. I'll revert my changes then for the --entrypoint stuff
Paolo Di Tommaso
@pditommaso
Oct 12 2017 16:51
:+1:
Bili Dong
@qobilidop
Oct 12 2017 18:23
Hi @pditommaso , I have a question about using singularity. When I’m looking at .commend.run, I see singularity exec xxx.img /bin/bash -c “cd $PWD; /bin/bash -ue /path/to/.command.sh”. So this means the path to workdir must be the same inside and outside the container, am I correct?
Paolo Di Tommaso
@pditommaso
Oct 12 2017 18:24
yes
Bili Dong
@qobilidop
Oct 12 2017 18:24
So I must bind paths this way in order for it to work
Okay, I got it
Paolo Di Tommaso
@pditommaso
Oct 12 2017 18:25
usually singularity mounts are defined by sysadmins
you can also use autoMounts = true
if you are allowed to bind custom paths
Bili Dong
@qobilidop
Oct 12 2017 18:26
thanks, I’ll try this if it doesn’t work automatically