These are chat archives for nextflow-io/nextflow

6th
Nov 2018
Riccardo Giannico
@giannicorik_twitter
Nov 06 2018 09:45
Hi guys, Is there a way for nextflow to print in stdout the full command line used to launch it with all the parameters included? It's for log purposes.
Paolo Di Tommaso
@pditommaso
Nov 06 2018 09:48
it's in the first lines of the .nextflow.log file
or you can just add the in the script log.info params
or something like this to make more readable log.info params.collect { k,v -> "--$k=$v" }.join(' ')
Tobias Neumann
@t-neumann
Nov 06 2018 09:51
sorry if I keep asking this, does anybody have experience with switching to a different AWS batch queue with increasing task.attempts in the config?
Paolo Di Tommaso
@pditommaso
Nov 06 2018 09:52
Tobias Neumann
@t-neumann
Nov 06 2018 09:54
perfect - thanks
Paolo Di Tommaso
@pditommaso
Nov 06 2018 09:54
welcome
Riccardo Giannico
@giannicorik_twitter
Nov 06 2018 10:04
@pditommaso thank you, but this is for the list of parameters only, what about the .nf file name? is there a way to catch it?
or just use nextflow run <something> -with-report
Riccardo Giannico
@giannicorik_twitter
Nov 06 2018 10:08
uuh! you always are 5 steps further :D
Paolo Di Tommaso
@pditommaso
Nov 06 2018 10:08
even a couple more :)
Riccardo Giannico
@giannicorik_twitter
Nov 06 2018 10:08
:D
Winni Kretzschmar
@winni2k
Nov 06 2018 13:52
Hi, quick question: Is it possible to access a file from an execute: block as part of a process?
Here is the code that I have:
process getNJunctions{
    input:
    set gid, file(json) from gid_traversal_jsons_ch

    output:
    set gid, file('g*.one_unitig') optional true into gid_one_unitig_ch
    set gid, file('g*.with_junctions') optional true into gid_several_unitigs_ch

    exec:
    n_junctions = new XmlSlurper().parse("$json")['n_junctions'] as Integer
    if (n_junctions == 0) {
    execute("touch g${gid}.one_unitig")
    } else if (params.max_junctions == null || n_junctions <= params.max_junctions) {
    execute("touch g${gid}.with_junctions")
    }
}
The error I get is this: Caused by: /private/var/folders/py/9zg6845j2gb664dqkw24wn4w0000gn/T/pytest-of-winni/pytest-117/test_traverses_cycle_into_no_t0/abeona/g0.traverse.ctx.json (No such file or directory)
Note the missing work/short_hash/long_hash part of the $json object
I saw workflow.workDir, but that gives me the /work directory.
Paolo Di Tommaso
@pditommaso
Nov 06 2018 13:54
Winni Kretzschmar
@winni2k
Nov 06 2018 13:54
cheers
Paolo Di Tommaso
@pditommaso
Nov 06 2018 13:54
you should use resolve files against task.workDir
what is execute ?
ah, just a touch ?
Winni Kretzschmar
@winni2k
Nov 06 2018 13:55
ya, I'm new to groovy...
Paolo Di Tommaso
@pditommaso
Nov 06 2018 13:55
:D
Winni Kretzschmar
@winni2k
Nov 06 2018 13:55
:D
Paolo Di Tommaso
@pditommaso
Nov 06 2018 13:56
I would refactor as gid_traversal_jsons_ch.map { /*your code */ }
of even
yourCode = { gid, file -> 
 // your stuff  
}
gid_traversal_jsons_ch.map(yourCode)
Winni Kretzschmar
@winni2k
Nov 06 2018 13:58
and then a .choice afterward?
Paolo Di Tommaso
@pditommaso
Nov 06 2018 13:59
yes
Winni Kretzschmar
@winni2k
Nov 06 2018 13:59
ok! Thanks for the advice.
Paolo Di Tommaso
@pditommaso
Nov 06 2018 13:59
:ok_hand:
Tobias Neumann
@t-neumann
Nov 06 2018 14:15

I defined the cloud scope in the config:

cloud {
    imageId = 'ami-0f99d00928be3a282'
    instanceType = 't2.micro'
    userName = 'ec2-user'
    keyFile = '~/aws/awsbatch.pem'
}

yet still Nextflow seems to be unable to recognize the credentials

nextflow -c nextflow.config cloud list
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/tmp
Missing AWS security credentials -- Provide access/security keys pair or define a IAM instance profile (suggested)

any ideas?

Paolo Di Tommaso
@pditommaso
Nov 06 2018 14:16
where aws credentials are stored ?
Tobias Neumann
@t-neumann
Nov 06 2018 14:22
$AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY
Paolo Di Tommaso
@pditommaso
Nov 06 2018 14:23
as env variables ?
Tobias Neumann
@t-neumann
Nov 06 2018 14:24
yepp
Paolo Di Tommaso
@pditommaso
Nov 06 2018 14:26
apparently NF is not recognising them
check the .nextflow.log there's come info where it's fetching aws credentials
Tobias Neumann
@t-neumann
Nov 06 2018 14:27
ok. it's just weird because I definitely already submitted to the AWS batch queue successfully
ach ok I ran it on a screen on an old node that didn't have the variables exported yet. thanks again
Tintest
@Tintest
Nov 06 2018 15:24
@pditommaso Thank you, this is exactly what I was looking for :) Are you considering to implement a spliTsv, or maybe there is a similar trick with splitText ?
Maxime Garcia
@MaxUlysse
Nov 06 2018 15:26
@Tintest You can use .splitCsv(sep: '\t')
Tintest
@Tintest
Nov 06 2018 15:28
@MaxUlysse :D Thank you
Winni Kretzschmar
@winni2k
Nov 06 2018 16:25
Is there a workaround for the use of mix with an empty channel as argument? See https://github.com/nextflow-io/nextflow/issues/702#issuecomment-390386241
Winni Kretzschmar
@winni2k
Nov 06 2018 16:32
Also, what's the definitive way to diagnose a hung pipline?
Winni Kretzschmar
@winni2k
Nov 06 2018 17:43
Ok, so it looks like my problem was caused by an erroneous redefinition of a channel: nextflow-io/nextflow#915