These are chat archives for nextflow-io/nextflow

14th
Mar 2019
Laurence E. Bernstein
@lebernstein
Mar 14 01:24
OK. Nevermind. I found away around all this.
Jonathan Manning
@pinin4fjords
Mar 14 09:00
Hi! Can I directly publish the output of a channel? e.g. I want to publish the results of a .collectFile() - do I need to feed it to a dummy process with a publishDir directive, or is there a more direct way?
Maxime Garcia
@MaxUlysse
Mar 14 09:01
yes
there's a direct way
Directly with the collectFile operator
look at the storeDir parameter
Jonathan Manning
@pinin4fjords
Mar 14 09:03
Ahh yes, I see it now - thanks. Sorry for the RTFM ;-)
Maxime Garcia
@MaxUlysse
Mar 14 09:03
Np, you were on the right track already
Vladimir Kiselev
@wikiselev
Mar 14 13:51
is there Nextflow integration with Spark?
Michael L Heuer
@heuermh
Mar 14 14:57
@wikiselev That question is music to my ears. :) What might that look like for you?
Vladimir Kiselev
@wikiselev
Mar 14 16:23
I have a general interest, not that there is any immediate application. Also, I heard that there is spark support for GATK cromwell pipeline
Anthony Underwood
@aunderwo
Mar 14 17:13
Hi all. Does anybody have any suggestions on best practice for auto clean up if a Nextflow run completes successfully. I know there is a nextflow clean but is there a way to run this from within nextflow that anybody has found?
Paolo Di Tommaso
@pditommaso
Mar 14 17:17
#!/bin/bash
set -e 
nextflow run .. etc
rm -rf work
Paolo Di Tommaso
@pditommaso
Mar 14 17:23
guess you were hoping for something different ..
Anthony Underwood
@aunderwo
Mar 14 17:27
Would you wrap that in a if block checking the return status of nextflow run??
Paolo Di Tommaso
@pditommaso
Mar 14 17:28
the set -e makes the script stop if NF return non-zero
Anthony Underwood
@aunderwo
Mar 14 17:29
Ah ha. Thanks
Vladimir Kiselev
@wikiselev
Mar 14 18:39

I am getting this:

Downloading nextflow dependencies. It may require a few seconds, please wait .. curl: (6) Could not resolve host: www.nextflow.io
ERROR: Cannot download nextflow required file -- make sure you can connect to the internet

Alternatively you can try to download this file:
    https://www.nextflow.io/releases/v19.03.0-SNAPSHOT/nextflow-19.03.0-SNAPSHOT-one.jar

and save it as:
    /.nextflow/framework/19.03.0-SNAPSHOT/nextflow-19.03.0-SNAPSHOT-one.jar

admiring-euclid:/workspace/vk6# ping www.nextflow.io
PING www.nextflow.io (54.239.164.108): 56 data bytes
64 bytes from 54.239.164.108: seq=0 ttl=237 time=4.466 ms
64 bytes from 54.239.164.108: seq=1 ttl=237 time=4.677 ms
64 bytes from 54.239.164.108: seq=2 ttl=237 time=4.668 ms
64 bytes from 54.239.164.108: seq=3 ttl=237 time=4.538 ms

am I doing something wrong?

ping works ok...
Paolo Di Tommaso
@pditommaso
Mar 14 18:43
NXF_VER=19.03.0-SNAPSHOT nextflow info
?
Vladimir Kiselev
@wikiselev
Mar 14 18:45
admiring-euclid:/workspace/vk6# NXF_VER=19.03.0-SNAPSHOT nextflow info
CAPSULE: Downloading dependency io.nextflow:nextflow:pom:19.03.0-20190313.124239-4
CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.nextflow:nextflow:pom:19.03.0-20190313.124239-4 in local (file:/root/.m2/repository) (for stack trace, run with -Dcapsule.log=verbose)
CAPSULE: Downloading dependency io.nextflow:nextflow:pom:19.03.0-20190313.124239-4
CAPSULE: Downloading dependency io.nextflow:nf-commons:pom:19.03.0-20190313.124311-4
CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.nextflow:nf-commons:pom:19.03.0-20190313.124311-4 in local (file:/root/.m2/repository) (for stack trace, run with -Dcapsule.log=verbose)
CAPSULE: Downloading dependency io.nextflow:nf-commons:pom:19.03.0-20190313.124311-4
CAPSULE: Downloading dependency io.nextflow:nf-httpfs:pom:19.03.0-20190313.124454-4
CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.nextflow:nf-httpfs:pom:19.03.0-20190313.124454-4 in local (file:/root/.m2/repository) (for stack trace, run with -Dcapsule.log=verbose)
CAPSULE: Downloading dependency io.nextflow:nf-httpfs:pom:19.03.0-20190313.124454-4
CAPSULE: Downloading dependency io.nextflow:nextflow:jar:19.03.0-20190313.124239-4
CAPSULE: Downloading dependency io.nextflow:nf-commons:jar:19.03.0-20190313.124311-4
CAPSULE: Downloading dependency io.nextflow:nf-httpfs:jar:19.03.0-20190313.124454-4
  Version: 19.03.0-SNAPSHOT build 5061
  Modified: 13-03-2019 12:41 UTC (12:41 GMT)
  System: Linux 4.4.0-131-generic
  Runtime: Groovy 2.5.6 on OpenJDK 64-Bit Server VM 1.8.0_151-b12
  Encoding: UTF-8 (UTF-8)
Paolo Di Tommaso
@pditommaso
Mar 14 18:46
looks better, no ?
Vladimir Kiselev
@wikiselev
Mar 14 18:47
yes, I had to run it multiple times though, before it worked...
Paolo Di Tommaso
@pditommaso
Mar 14 18:47
weird
Vladimir Kiselev
@wikiselev
Mar 14 18:48
now it’s starting but can't pull from GitHub…
admiring-euclid:/workspace/vk6# ./RUNbm2-cellgenisu small k8s
Running test small on platform k8s with profile docker
Samplefile: /workspace/svd/inputs/samples-4187.txt
Studyid: 4187
Source: cellgeni/rnaseq -r benchmark
N E X T F L O W  ~  version 19.03.0-SNAPSHOT
Pulling cellgeni/rnaseq ...
WARN: Cannot read project manifest -- Cause: api.github.com
Cannot find `cellgeni/rnaseq` -- Make sure exists a GitHub repository at this address `https://github.com/cellgeni/rnaseq`
Paolo Di Tommaso
@pditommaso
Mar 14 18:48
ummm
Vladimir Kiselev
@wikiselev
Mar 14 18:48
this is kubernetes in a NF pod
after login
maybe I am doing something wrong, but Stijn is away, can’t ask him ;-)
it may be a complicated network configuration we have there...
Paolo Di Tommaso
@pditommaso
Mar 14 18:52
it looks so to me
Vladimir Kiselev
@wikiselev
Mar 14 18:53
maybe it’s not allowing https...