These are chat archives for nextflow-io/nextflow

26th
Sep 2018
Paolo Di Tommaso
@pditommaso
Sep 26 2018 06:58
you may try something like afterScript 'cat spades.log' (test it locally!)
@rolivella @lucacozzuto the proposal of @ypriverol seems very interesting, you should be interested
Luca Cozzuto
@lucacozzuto
Sep 26 2018 07:51
@pditommaso @ypriverol @rolivella indeed we are interested. We are finishing a proteomics QC pipeline that is using some procedures from knime
orzechoj
@orzechoj
Sep 26 2018 08:13
I have problems installing nextflow on a cluster with no internet connection. I tried compiling it on another machine, with make pack, and then moving it to the cluster without internet connection. But I still get all kinds of errors, as it tries to download stuff over the internet:

First I got $ ./nextflow
Downloading nextflow dependencies. It may require a few seconds, please wait .. curl: (6) Could not resolve host: www.nextflow.io; Unknown error
ERROR: Cannot download nextflow required file -- make sure you can connect to the internet

Alternatively you can try to download this file:
https://www.nextflow.io/releases/v0.30.1/nextflow-0.30.1-one.jar

and save it as:
/home/jacke/.nextflow/framework/0.30.1/nextflow-0.30.1-one.jar

When I got this file, I got some other error messages e.g. CAPSULE: Downloading dependency org.multiverse:multiverse-core:pom:0.7.0 CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.multiverse:multiverse-core:pom:0.7.0 from/to central (https://repo1.maven.org/maven2/): repo1.maven.org: Name or service not known (for stack trace, run with -Dcapsule.log=verbose)
Hope this is the right place for these kinds of problems!
Roger Olivella
@rolivella
Sep 26 2018 08:24
Of course @pditommaso @ypriverol , we are very interested into join efforts regarding proteomics pipelines. As @lucacozzuto said, we almost finished a QC pipeline that we could share after being tested and fine-tuned.
Yasset Perez-Riverol
@ypriverol
Sep 26 2018 08:30
@rolivella is this pipeline @nexflow based?
Roger Olivella
@rolivella
Sep 26 2018 08:32
@ypriverol yes!
Maxime Garcia
@MaxUlysse
Sep 26 2018 08:33
@orzechoj I had this exact issue
I made this small tutorial on how to do it
Hoping it can help you
orzechoj
@orzechoj
Sep 26 2018 08:51
@MaxUlysse Thanks, I’ll try it right away!
Paolo Di Tommaso
@pditommaso
Sep 26 2018 08:54
if your system has not internet access, just download the -all package from GH
orzechoj
@orzechoj
Sep 26 2018 08:55
@pditommaso That seems even simpler!
Maxime Garcia
@MaxUlysse
Sep 26 2018 09:07
@pditommaso That seems like a good idea
Paolo Di Tommaso
@pditommaso
Sep 26 2018 09:08
A bit simpler :)
Maxime Garcia
@MaxUlysse
Sep 26 2018 09:08
@orzechoj I'm guessing you're on bianca as well, could you make a PR with this better system on the Sarek docs?
Luca Cozzuto
@lucacozzuto
Sep 26 2018 09:48
Hi @pditommaso I have some trouble with watchPath
besically it waits forever...
micans
@micans
Sep 26 2018 09:53
@pditommaso , we're trying kuberun in earnest now. Can I somehow specify that on the pvc (in our case /mnt/gluster) I want to run nextflow in /mnt/gluster/svd/tic-97?
I've tried a few things with -v and -w.
Paolo Di Tommaso
@pditommaso
Sep 26 2018 10:08
launch execution path of workflow execution path aka. workdir ?
micans
@micans
Sep 26 2018 10:13
My sample file is in /mnt/gluster/svd/tic-97/samples.txt; I'd like to set workflow.launchDir to /mnt/gluster/svd/tic-97; at the moment it seems to always be /mnt/gluster.
Luca Cozzuto
@lucacozzuto
Sep 26 2018 10:13
@pditommaso it looks like you need to use -bg
or it will wait endlessy EVEN when new files appear
Paolo Di Tommaso
@pditommaso
Sep 26 2018 10:15
@micans k8s.projectDir https://www.nextflow.io/docs/latest/config.html#scope-k8s (maybe it should be called launchDir for consistency)
micans
@micans
Sep 26 2018 10:19
Do you mean k8s.userDir? That desription seems to match
Paolo Di Tommaso
@pditommaso
Sep 26 2018 10:20
yes, it's confusing :)
micans
@micans
Sep 26 2018 10:20
Cool. I had another problem, which was that -c configfile did not seem to be picked up. I'll check that again.
Thanks!
userDir is that under the mount path? E.g. 'svd/tic-97'? No matter, I'll find out :-)
Paolo Di Tommaso
@pditommaso
Sep 26 2018 10:22
it's supposed to be an absolute path
micans
@micans
Sep 26 2018 10:22
ooooo it's running
Paolo Di Tommaso
@pditommaso
Sep 26 2018 10:22
lol
magic things happen
micans
@micans
Sep 26 2018 10:23
Current path   : /mnt/gluster/svd/tic-97
Script dir     : /mnt/gluster/projects/cellgeni/rnaseq-noqc
Config Profile : standard
=========================================
[warm up] executor > k8s
[60/dc9356] Submitted process > irods (auim_C)
[f4/108246] Submitted process > irods (auim_D)
[63/4bcd0c] Submitted process > irods (bubh_B)
[60/b24674] Submitted process > irods (bima_D)
I'm trying to weigh the differences between nextflow kuberun and logging into a nextflow pod and running nextflow from there. In both cases I need to do some directory+file management, and kuberun feels a bit like a fork with a very long handle. But it's early days. Are there fundamental differences?
Luca Cozzuto
@lucacozzuto
Sep 26 2018 10:28
So @pditommaso is -bg mandatory for watchPath?
Paolo Di Tommaso
@pditommaso
Sep 26 2018 10:28
the kube run it's just a shortcut to streamline a workflow execution, for advanced usage maybe better to manage directly the main pod allocation and execution
@lucacozzuto not aware of any difference
micans
@micans
Sep 26 2018 10:29
:+1:
Luca Cozzuto
@lucacozzuto
Sep 26 2018 10:30
@pditommaso let me test more...
Anthony Underwood
@aunderwo
Sep 26 2018 12:38
So I have been debugging my AWS batch issue a little. It appears to be the fact that Docker does not have access to enough memory. free -m suggests there is plenty 8Gb but perhaps the job definition specifies a memory limit for Docker . Is there a way of increasing the memory limit for a job in AWS batch via Nextflow
Paolo Di Tommaso
@pditommaso
Sep 26 2018 12:39
the usual memory directive is doing that
micans
@micans
Sep 26 2018 12:52

I run the exact same pipeline on LSF and k8s (almost identical command lines, using kuberun for the latter). On k8s I get this error:

Caused by:
  No signature of method: check_max() is applicable for argument types: (nextflow.util.MemoryUnit, java.lang.String) values: [4 GB, memory]
Possible solutions: check_log(java.lang.Object)

where check_max is a function defined in nextflow.config in the project directory. Has anyone seen something like this? I set k8s.userDir, and in that directory I find a new nextflow.config file that seems a machine dump of NF config settings; and in this file check_max is absent. It seems as if the function definition is not exported, is that possible?

Paolo Di Tommaso
@pditommaso
Sep 26 2018 12:55
interesting, this may be an issue on the config files made by kuberun, you may want to fill an issue
micans
@micans
Sep 26 2018 12:58
ok, have to run for meeting now, will get back to this.
misssoft
@misssoft
Sep 26 2018 14:02
Anthony Underwood
@aunderwo
Sep 26 2018 14:07

@pditommaso

the usual memory directive is doing that

Thanks so much for your patience. I had not used the memory directive previously since when I ran nextflow with docker on an EC2 instance with the same type as the AWS batch compute it 'just ran'.
However now that I have specified process.memory = '3.5G' AWS batch is working. Thanks a million!! I will blog post on a few extras I came across in the process but massively based on @apeltzer 's super useful post

@pditommaso would you recommend always using the memory directive
Luca Cozzuto
@lucacozzuto
Sep 26 2018 14:30
hi all. I cannot remember how to make the pipeline resilient to failing processes...
can someone help me?
found!
Alexander Peltzer
@apeltzer
Sep 26 2018 15:14
@aunderwo would be nice to hear about these yes :-)
Paolo Di Tommaso
@pditommaso
Sep 26 2018 16:35
@aunderwo yes, definitively to specify the mem when deploying over the cloud
@misssoft very nice and very useful, thanks a lot!