These are chat archives for nextflow-io/nextflow

13th
Sep 2017
Paolo Di Tommaso
@pditommaso
Sep 13 2017 08:27
that's bad
I would suggest to first try with the Amazon linux AMI we provide, see here
if it is fine, I can help you to trouble shoot the problem with your custom ami
Venkat Malladi
@vsmalladi
Sep 13 2017 14:16
Are there any good examples of unit testing for each process?
Steve Marshall
@stevemmarshall
Sep 13 2017 14:24
ok thanks @pditommaso 
I assume you mean this ami? ami-d1f9a6aa
Steve Marshall
@stevemmarshall
Sep 13 2017 15:47
@pditommaso so i tried the nextflow ami and specified that ami in the nextflow.config and created a two node cluster...
I created this simple example..

params.query = "$baseDir/data/sample.fa"
params.out = 'result.txt'
/*

  • A channel emitting the fasta files specified as parameter
    /
    seq = Channel.fromPath(params.query)
    /
  • Execute a job for each chunk for the provided sequences
    */
    process run {
    tag { x.baseName }

    input:
    file x from seq

    output:
    file 'out' into result

    """
    echo "Host: \$(hostname) file: $x" > out
    """
    }

/*

  • Collect all the outputs produced by the process
  • executions to a single file, whose name is defined by
  • the params.out parameters
    */

result
.collectFile(name: file(params.out))
.println { "Result saved to file: $it" }

I ran it locally and gave me the same hostname
but I tried running with ignite option and it still ran on the local master node
Paolo Di Tommaso
@pditommaso
Sep 13 2017 16:41
also the master node is used to run job, so this could be fine
Steve Marshall
@stevemmarshall
Sep 13 2017 16:47
is there a way to not run on the master node?
Paolo Di Tommaso
@pditommaso
Sep 13 2017 16:48
use the option -cluster.master.maxCpus 0 on the launch command line
Steve Marshall
@stevemmarshall
Sep 13 2017 17:53
Ok so I tried that and it's seem to be hung up... looking at the log file, it mentions pending tasks... seems like it doesn't recognize the other machine... what ports need to be open? can I get more logging info?
Paolo Di Tommaso
@pditommaso
Sep 13 2017 17:54
make sure that all ports are open in the same security group
Steve Marshall
@stevemmarshall
Sep 13 2017 18:20
ya, they are in the same security groups and all ports are open
Paolo Di Tommaso
@pditommaso
Sep 13 2017 18:21
Are u able to ping the other instance?
Steve Marshall
@stevemmarshall
Sep 13 2017 18:21
yup
Paolo Di Tommaso
@pditommaso
Sep 13 2017 18:23
Can you post somewhere eg pastebin.com the NF log files both on master and the other node
Steve Marshall
@stevemmarshall
Sep 13 2017 18:26
i don't see log files on the worker node, are you referring to my other machine where i created the cluster?
Venkat Malladi
@vsmalladi
Sep 13 2017 19:35
@pditommaso I am trying to run unittests for each process, do you have a good example of this?
Mamana
@mypandos
Sep 13 2017 19:52

Hi, a colleague of mine is getting the error below when trying to run a nextflow workflow:

CAPSULE: Downloading dependency commons-codec:commons-codec:jar:1.9
CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact commons-codec:commons-codec:jar:1.9 from/to central (https://repo1.maven.org/maven2/): Range Not Satisfiable (416) (for stack trace, run with -Dcapsule.log=verbose)
CAPSULE EXCEPTION: Error resolving dependencies. while processing attribute Allow-Snapshots: false (for stack trace, run with -Dcapsule.log=verbose)
Unable to initialize nextflow environment

Suspecting a firewall issue :-) Anyone else experiencing this?

Steve Marshall
@stevemmarshall
Sep 13 2017 20:22
@pditommaso take a look at this https://pastebin.com/VQEtHV0t