These are chat archives for nextflow-io/nextflow

8th
May 2018
Sven F.
@sven1103
May 08 2018 06:42
@stevekm also pull again , I made some changes ;)
Sven F.
@sven1103
May 08 2018 07:00
@pditommaso How can I get gradle to run only tests for one class? I tried gradle -Dtest.single=<ClassName> test but tests wont' run.
Paolo Di Tommaso
@pditommaso
May 08 2018 07:03
that's a popular question :)
Sven F.
@sven1103
May 08 2018 07:40
yes, sorry fot that, I googled before -.-
Paolo Di Tommaso
@pditommaso
May 08 2018 07:40
no problem at all, I've just linked because I wrote that
fixed ?
Sven F.
@sven1103
May 08 2018 07:41
I just came back from a meeting, will test now :)
ah cool, your post indeed :D
hm, gradle test --tests nextflow.trace.TraceRecordTest does not work
Paolo Di Tommaso
@pditommaso
May 08 2018 07:44
does not find the test ?
Sven F.
@sven1103
May 08 2018 07:44
no
Paolo Di Tommaso
@pditommaso
May 08 2018 07:45
what about
gradle :test --tests nextflow.trace.TraceRecordTest
Sven F.
@sven1103
May 08 2018 07:45
awesome
that works
Paolo Di Tommaso
@pditommaso
May 08 2018 07:45
:)
Sven F.
@sven1103
May 08 2018 07:45
wtf
Paolo Di Tommaso
@pditommaso
May 08 2018 07:45
yes, they have this strange syntax
Sven F.
@sven1103
May 08 2018 07:46
I googled a lot yesterday and felt ashamed not to get it running
:D
Paolo Di Tommaso
@pditommaso
May 08 2018 07:46
BTW you can use make test class=nextflow.trace.TraceRecordTest
Sven F.
@sven1103
May 08 2018 07:46
ah cool
will try to write tests for #681 now, but will be slow, as I am not used to groovy tests :D
Paolo Di Tommaso
@pditommaso
May 08 2018 07:48
have you experience with JUnit ?
Sven F.
@sven1103
May 08 2018 07:48
a little
Paolo Di Tommaso
@pditommaso
May 08 2018 07:49
ok, the concept is more or less the same
I'm using a library that's called Spock
Sven F.
@sven1103
May 08 2018 07:49
Ah yes, I saw that
Sven F.
@sven1103
May 08 2018 07:49
I like the name
googled it already
:D
"the enterprise ready specification framework"
Paolo Di Tommaso
@pditommaso
May 08 2018 07:50
it's kind of syntactic sugar over JUnit
Sven F.
@sven1103
May 08 2018 07:50
awesome
Paolo Di Tommaso
@pditommaso
May 08 2018 07:50
the main idiom is
given: 
def something 
when:
def result = some.callThisOrThat()
then:
result == expected
Sven F.
@sven1103
May 08 2018 07:51
It also comes with mocking functionality?
will need that for my simple http client
Paolo Di Tommaso
@pditommaso
May 08 2018 07:52
yes, and it's super cool
Sven F.
@sven1103
May 08 2018 07:52
super
Paolo Di Tommaso
@pditommaso
May 08 2018 07:52
for http stuff, I've already linked you an example
for mocking not related to http let me find an example
Give a look to the kubernetes http client
Sven F.
@sven1103
May 08 2018 07:53
ah, should I just use the K8s client for my requests directly?
sorry, thought you meant it as an example how to use native java URL connection libs
Paolo Di Tommaso
@pditommaso
May 08 2018 07:54
no
Sven F.
@sven1103
May 08 2018 07:54
kk
because I wrote a SimpleHttpClient for my stuff
Paolo Di Tommaso
@pditommaso
May 08 2018 07:54
yes, makes sense
Sven F.
@sven1103
May 08 2018 07:55
using HttpUrlConnection
it was fun
as you are more on the basis of how HTTP requests work
learned again some stuff
Paolo Di Tommaso
@pditommaso
May 08 2018 07:55
yes, I like to work with low-level stuff ;)
it's more fun
Sven F.
@sven1103
May 08 2018 07:56
yes, and you have less third-party dependencies
and for this, the implementation for the stuff I need was not rocket science
Zucco AG.
@adrigabzu
May 08 2018 07:57

Hi everybody, thanks for all the effort to develop Nextflow :) I'm a beginner trying to execute Nextflow in a HPC managed by Torque. I have the issue that I have only access to udocker in the HPC system I'm using so when I try to execute an existing pipeline I get the following error:

Command exit status:
  125

Command output:
  (empty)

Command error:
  /usr/bin/docker-current: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
  See '/usr/bin/docker-current run --help'.

I think that it might be something related to the path and the fact that I don't have docker available in my system but udocker. Any help would be really appreciated, Thanks :)

leiendeckerlu
@leiendeckerlu
May 08 2018 07:58

Hi Paolo, can you give me a hint what is going wrong here?

'''
process getUtilFiles {

output:
file 'bwa_idx_GRCh37d5.tar.gz' into index1
file 'core_ref_GRCh37d5.tar.gz' into reference1

"""
wget -c ftp://ftp.sanger.ac.uk/pub/cancer/dockstore/human/bwa_idx_GRCh37d5.tar.gz > bwa_idx_GRCh37d5.tar.gz
wget -c ftp://ftp.sanger.ac.uk/pub/cancer/dockstore/human/core_ref_GRCh37d5.tar.gz > core_ref_GRCh37d5.tar.gz
"""

}
'''

and it gives me the following output:

Command exit status:
127

Command output:
(empty)

Command error:
.command.sh: line 2: wget: command not found

Paolo Di Tommaso
@pditommaso
May 08 2018 07:59
sorry there's another question before
@adrigabzu not understanding, are you using the NF integration for udcoker ?
leiendeckerlu
@leiendeckerlu
May 08 2018 07:59
sure - i'll hang around :)
arontommi
@arontommi
May 08 2018 08:02
@leiendeckerlu it looks like nf cant access wget
leiendeckerlu
@leiendeckerlu
May 08 2018 08:03
yes - but this is very strange as it works perfectly fine by running wget directly on the command line
same machine
Paolo Di Tommaso
@pditommaso
May 08 2018 08:03
are you using a container ?
arontommi
@arontommi
May 08 2018 08:03
are you running a container ?
yes what he said
leiendeckerlu
@leiendeckerlu
May 08 2018 08:04
later in another process - yes
Zucco AG.
@adrigabzu
May 08 2018 08:05
@pditommaso sry I didn't know there was such integration, I couldn't find it in the documentation Where could I find that info to set it up? Thanks for the fast answer
arontommi
@arontommi
May 08 2018 08:05
@leiendeckerlu
i guess that you are running a container for the process that does not have wget
(take this with a grain of salt as i usually do not know what i am talking about )
leiendeckerlu
@leiendeckerlu
May 08 2018 08:08
@arontommi yes that sounds plausible! but why would that command be executed in a container even if the container is not associated with this process?
arontommi
@arontommi
May 08 2018 08:09
is it possible you are running the container for all processes ?

anyways, i am giving this another try;
I have a process like this:

`process testprocess{
    input:
        file bam
    output: 
        file bam, file(*.vcf) into nextstep
    script:
    """
        some process that creates a vcf file 
    """
     } `

So my problem is that the bam file is copied from NXF_SCRATCH when the process is finishing, and instead of getting the symlink i get the complete file into NXF_WORK

if i look into .command.run this is what is see:
cp -fRL bam.bam mypath/work/3e/b63f8aacb56b071f0d9c6d2e2e09bc || true

I need the bam for downstream processes, but i dont want to have massive workfiles. is there a simple trick that i can use to get the symlink for the bam instead of the bam ?

Paolo Di Tommaso
@pditommaso
May 08 2018 08:13
if you are using a scratch directory, the output files need to be copied into the shared storage
arontommi
@arontommi
May 08 2018 08:16
okay thanks! so by having scratch = false i would avoid this, and have everything as symlinks in NXF_WORK
Paolo Di Tommaso
@pditommaso
May 08 2018 08:17
yes
BTW process.scratch is false by default
leiendeckerlu
@leiendeckerlu
May 08 2018 08:18
@arontommi you are right. But I'm still wondering how the command can slip into the container if I defined them like this process {
withName:foo {
container = 'image_name_1'
}
withName:bar {
container = 'image_name_2'
}
}
singularity {
enabled = true
}
arontommi
@arontommi
May 08 2018 08:18
yes, i see where it was changed, can i have scratch = false for specific processes ?
Paolo Di Tommaso
@pditommaso
May 08 2018 08:19
no
arontommi
@arontommi
May 08 2018 08:19
okay i will test turning it off for the whole thing and see what i brake ;)
Paolo Di Tommaso
@pditommaso
May 08 2018 08:19
:+1:
@leiendeckerlu can you check the content of the .command.run and make sure there's a singularity command line ?
leiendeckerlu
@leiendeckerlu
May 08 2018 08:29
@pditommaso there is indeed a singularity command line...
Paolo Di Tommaso
@pditommaso
May 08 2018 08:30
then, there's something wrong in the config definition
Maxime Garcia
@MaxUlysse
May 08 2018 08:32
@arontommi Trying the same thing ;-)
Paolo Di Tommaso
@pditommaso
May 08 2018 08:34
bingo, I think there's a bug
could you please fill-out an issue ?
leiendeckerlu
@leiendeckerlu
May 08 2018 08:39
@pditommaso you mean me? sure can I fill-out an issue
Paolo Di Tommaso
@pditommaso
May 08 2018 08:39
any of you :)
leiendeckerlu
@leiendeckerlu
May 08 2018 08:45
issue-filled: nextflow-io/nextflow#690
Paolo Di Tommaso
@pditommaso
May 08 2018 08:50
:+1:
leiendeckerlu
@leiendeckerlu
May 08 2018 08:56
@pditommaso do you have a suggestion for a work-around?
Paolo Di Tommaso
@pditommaso
May 08 2018 08:56
let me check
found, is not a bug :smile:
you are suing a wrong syntax
it should be
Paolo Di Tommaso
@pditommaso
May 08 2018 09:01
process {
    withName:alignNormal {
        container = 'docker://quay.io/wtsicgp/dockstore-cgpmap:3.0.1'
    }
    withName:alignTumor {
        container = 'docker://quay.io/wtsicgp/dockstore-cgpwgs:1.1.3'
    }
}
Sven F.
@sven1103
May 08 2018 09:08
@pditommaso
class SimpleHttpClientTest extends Specification{

    @Shared def httpClient

    def setupSpec(){

        httpClient = new SimpleHttpClient()

    }

    def 'should set a http url successfully' () {

        given:
        def url = "http://localhost"

        when:
        httpClient.setUrl(url)

        then:
        noExceptionThrown()
        assert httpClient.getUrl() == url

    }


    def 'should raise a MalFormattedUrl exception' () {

        given:
        def url = "localhost"

        when:
        httpClient.setUrl(url)

        then:
        thrown(MalformedURLException)

    }

}
this is just great
Paolo Di Tommaso
@pditommaso
May 08 2018 09:08
:)
the funny part is this syntax
        given:
        def client = Spy(K8sClient)
        when:
        client.get('/foo/bar')
        then:
        1 * client.makeRequest('GET', '/foo/bar') >> null
did you get this
 1 * client.makeRequest('GET', '/foo/bar') >> null
?
Sven F.
@sven1103
May 08 2018 09:09
not really ^^
Paolo Di Tommaso
@pditommaso
May 08 2018 09:10
that means when you invoke client.get('/foo/bar')
Sven F.
@sven1103
May 08 2018 09:10
ahhh
nice
Paolo Di Tommaso
@pditommaso
May 08 2018 09:10
then you are expecting that client.makeRequest is invoked exactly 1 time
with the parameter specified
Sven F.
@sven1103
May 08 2018 09:11
oh this is very descriptive
and simple
Paolo Di Tommaso
@pditommaso
May 08 2018 09:11
and returns >> null
the >> something
allow you to intercept the method invocation and return any stub value
Sven F.
@sven1103
May 08 2018 09:12
Spy() is the mocking thingy I assume
yeah, this is great
will do that now for simulation of different server side events
leiendeckerlu
@leiendeckerlu
May 08 2018 09:12
@pditommaso excellent! Many thanks!
Paolo Di Tommaso
@pditommaso
May 08 2018 09:13
the different between Mock, Stub and Spy can be a but tricky
the baseline is Mock for a complete fake object
and Spy is a mock that behave like the real one except for the method that you overloads
Sven F.
@sven1103
May 08 2018 09:15
Cool, this is valuable to know
and I already see, why it is good to have tests, as my method is hard to test atm and needs to be refactored :joy:
Paolo Di Tommaso
@pditommaso
May 08 2018 09:19
exactly !
Sven F.
@sven1103
May 08 2018 09:19
For some projects, I did test-driven-development, and was amazed by the code resulted by that.
Paolo Di Tommaso
@pditommaso
May 08 2018 09:21
much moooooore decoupled and stable
Sven F.
@sven1103
May 08 2018 09:21
so true
"If you don't have the time to write tests, you certainly have no time for debugging" a quote I have once stumbled over
Paolo Di Tommaso
@pditommaso
May 08 2018 09:22
nice this, I need to remember
Sven F.
@sven1103
May 08 2018 09:22
go for lunch now, cu later
Paolo Di Tommaso
@pditommaso
May 08 2018 09:29
11.30 :clock1130:
this sounds so funny being in Spain .. :)
Maxime Garcia
@MaxUlysse
May 08 2018 09:30
Not an unsual time to go to lunch in Sweden ;-)
Sven F.
@sven1103
May 08 2018 10:20
the students are blocking everything if you go at 12-12:30 ...
Paolo Di Tommaso
@pditommaso
May 08 2018 12:15
@adrigabzu Regarding udocker, yes there's an undocumented support, to unable it you should add udocker.enabled = true in the nextflow file. However not sure it works, I've abandoned the development time ago
Rickard Hammarén
@Hammarn
May 08 2018 12:21
Hi! I have a channle with a Tuple of files.
How do I assign each of the to a different variable? my thought:
input:
     file foo '*.foo'; bar '*bar'; fam'*.fam'  from myChannel
Paolo Di Tommaso
@pditommaso
May 08 2018 12:23
input:
     set file(foo:'*.foo'), file(bar:'*bar'), file(fam:'*.fam')  from myChannel
Rickard Hammarén
@Hammarn
May 08 2018 12:23
thanks! :+1:
Maxime Garcia
@MaxUlysse
May 08 2018 12:23
I knew it was something simple
Rickard Hammarén
@Hammarn
May 08 2018 12:24
Yeah, kinda hard to guess though, I tried looking though the docs and example but couldn't find anything similar
Simone Baffelli
@baffelli
May 08 2018 13:11
Hello. Does using "toSortedList" slow up caching?
Maxime Garcia
@MaxUlysse
May 08 2018 14:56
Hi, I noticed that the phase operator is depreciated I guess I should have switched to the join one already
Paolo Di Tommaso
@pditommaso
May 08 2018 15:01
suggested
Maxime Garcia
@MaxUlysse
May 08 2018 15:04
not a high priority I guess
Paolo Di Tommaso
@pditommaso
May 08 2018 15:04
no, surely it won't be removed in the short term
Maxime Garcia
@MaxUlysse
May 08 2018 15:07
ok, good to know
Maxime Garcia
@MaxUlysse
May 08 2018 15:14
By the way, I managed to fix @arontommi issue with Sarek, the problem was caused by a file that was not touch during a process but still used as an output, and I sort it all out with join, that's how I notice that phase was depreciated
Simone Baffelli
@baffelli
May 08 2018 15:42
I may or not have an issue: should a process with a mix of native groovy and shell execute the groovy code when I specify storeDir? In my case, it seems it does, but I don't think it should be necessary
Mike Smoot
@mes5k
May 08 2018 17:51
Hi @pditommaso we've got a pipeline with a default docker container specified in nextflow.config and a process with container null specified for a specific situation where we don't want to use docker. As of version 0.29.0 of nextflow, instead of using no container, the process gets run in the default container. Is this a bug or a new feature?
Paolo Di Tommaso
@pditommaso
May 08 2018 17:57
um, maybe a side effect of the config handling
please, open an issue with an example to replicate it
Mike Smoot
@mes5k
May 08 2018 17:58
Sounds likely. I'll see if I can put together a small example and then open a ticket.
Paolo Di Tommaso
@pditommaso
May 08 2018 17:58
thanks