These are chat archives for nextflow-io/nextflow

8th
Jan 2019
David Mas-Ponte
@davidmasp
Jan 08 11:30
Hello all, I have a quick question about the optional output. (here)
could I change the true value for a condition?
like this task.exitStatus == 140
if yes, how should I do it?
Paolo Di Tommaso
@pditommaso
Jan 08 11:31
uhh
don't think it's possible
David Mas-Ponte
@davidmasp
Jan 08 11:32
Mmmh okay
Paolo Di Tommaso
@pditommaso
Jan 08 11:33
also generally 140 mean the cluster killed your job
looks a bit creepy that logic
David Mas-Ponte
@davidmasp
Jan 08 11:33
yes sorry, that was just the example I copied from here
in my code it would be a custom error message
I have some custom code that in a particular circumstance it returns an error. Outside nextflow this is fine and I don't want to create empty files, however, I am not sure how to handle this in the pipeline
Paolo Di Tommaso
@pditommaso
Jan 08 11:36
maybe using optional and declaring that exit code as valid https://www.nextflow.io/docs/latest/process.html#validexitstatus ?
David Mas-Ponte
@davidmasp
Jan 08 11:37
I'll just have to workaround that somehow. I think the optional true thing will work (testing now) but I just wanted to be more specific about in which situations we could do
oh true
I am actually already using this validExitStatus
of course, then any other thing will fail
okay, I should have seen this.
thanks Paolo!
Paolo Di Tommaso
@pditommaso
Jan 08 11:38
welcome!
micans
@micans
Jan 08 15:17
I have a docker container / NF process, where a location has to be added to PATH. I can do it in the script section, but is there a better way?
Paolo Di Tommaso
@pditommaso
Jan 08 15:18
env.PATH = 'something:$PATH'
in nextflow.config
micans
@micans
Jan 08 15:19
Thanks, that looks familiar :-P . that can be in a config imported with -c I assume?
Paolo Di Tommaso
@pditommaso
Jan 08 15:22
of course !
micans
@micans
Jan 08 15:22
hehe, sorry I'm rusty
Paolo Di Tommaso
@pditommaso
Jan 08 15:22
ahah
micans
@micans
Jan 08 15:22
Running it now ....
Paolo Di Tommaso
@pditommaso
Jan 08 15:22
go!
micans
@micans
Jan 08 15:22
Yes searge ... disappears into obstacle course
Paolo Di Tommaso
@pditommaso
Jan 08 15:23
wat? :satisfied:
micans
@micans
Jan 08 15:24
So env.path is exported into each docker container ... well it's certainly useful!
so far so good
Paolo Di Tommaso
@pditommaso
Jan 08 15:24
yes, it is :sunglasses:
micans
@micans
Jan 08 15:26
Awesome ... :+1:
Paolo Di Tommaso
@pditommaso
Jan 08 15:26
I'm improving the metrics, are you ready to test it :)
micans
@micans
Jan 08 15:27
Yes I saw that. Absolutely! Off to a meeting now ... speak later!
Good stuff
Paolo Di Tommaso
@pditommaso
Jan 08 15:28
it has been a bloodbath .. but it was time for it
Daniel E Cook
@danielecook
Jan 08 15:48
Has anyone played around with passing objects between processes? Would nextflow caching be able to account for this?
Paolo Di Tommaso
@pditommaso
Jan 08 15:49
yes, the only requirement is that it/they must implement the Serializable interface
Daniel E Cook
@danielecook
Jan 08 15:50
ok thanks! I'll look into it
Paolo Di Tommaso
@pditommaso
Jan 08 15:50
:ok_hand:
micans
@micans
Jan 08 17:08

@pditommaso on k8s I get this:

[22/adaaf1] Submitted process > crams_to_fastq (Immunodeficiency7132960)
[83/1f470a] Submitted process > irods (Immunodeficiency7112547)
WARN: K8s pod cannot be scheduled -- 0/19 nodes are available: 16 Insufficient cpu, 3 PodToleratesNodeTaints.
[0d/3d0f0f] Submitted process > bracer_assemble (Immunodeficiency7112048)
[95/dd136f] Submitted process > bracer_assemble (Immunodeficiency7111946)

(1) I assume the process affected by the WARN is staged/rescheduled as per nextflow-io/nextflow#824
(2) is it possible to have the e.g. the tag of that process in the warning message? Right now it is difficult to follow up.

This is with NXF_VER=18.12.0-edge
Paolo Di Tommaso
@pditommaso
Jan 08 17:10
Ouch, not sure, please open an issue
micans
@micans
Jan 08 17:11
OK will do. Not reopen 824?
I guess that's bad practice ..
Paolo Di Tommaso
@pditommaso
Jan 08 17:12
umm, better a new one, it's free :D
micans
@micans
Jan 08 17:12
:+1:
Stephen Kelly
@stevekm
Jan 08 17:17

@pditommaso is there a way to convert docker image in a writable singularity image with Nextflow?

This is outside the scope of Nextflow. However I have an old example of doing a direct Docker -> Singluarity conversion here: https://github.com/stevekm/docker2singularity-demo

However, the better suggestion is to simply make a Dockerfile and and an equivalent Singularity file, and build both independenty; I have lots of examples of that here: https://github.com/NYU-Molecular-Pathology/containers
@lucacozzuto

Stephen Kelly
@stevekm
Jan 08 17:23

Hey I have a quick question, what is the best way to implement a full combinatorial expansion of items in Nextflow?

For example, if I have [a, b, c], I want to run Nextflow tasks on a, ab, ac, abc, bc, b, c, etc.. Gets more complicated when you get to 4-5+ variables.

I have kinda a similar idea used here, but in that case I built up the combinations sequentially across multiple tasks. In this case, I want to run a single task with all possible combinations.

Use case for this is that I have a number of filter criteria for some variants that I want to evaluate, and I need to evaluate all combinations of all criteria.

I was thinking to use nested arrays either in an input Channel, or as a each repeater, such as:

[
['param1', '-arg 1'],
['param2', '-arg 2'],
['param3', '-arg 3'],
['param4', '-arg 4'],
]

and then I need to run a task on all combinations of these parameters

Paolo Di Tommaso
@pditommaso
Jan 08 17:27
there are many ways, many each results in a cartesian product
same with combine
the ideal would be nextflow-io/nextflow#168
Daniel E Cook
@danielecook
Jan 08 18:35
@pditommaso This is very relevant to my working on passing an object around...my plan was to use a method to output a slug containing the parameters...
Paolo Di Tommaso
@pditommaso
Jan 08 18:36
?
Daniel E Cook
@danielecook
Jan 08 18:36
import sun.nio.fs.UnixPath
class Info implements Serializable {
    /* Makes it easy to output the params as string for filenames */
    String ID, ST, RG, ID_RG // LTX ID, Status(BS, SU, etc.), TN=Tumor Number, RG=Tumor Region
    String HOSPITAL
    String basename
    Path fname, fname_bai
    Boolean is_tumor, is_germline
    HashMap p = [:] // parameter mapping
    private split

    Info(fname_in) {
        split = fname_in.baseName.split("[_-]+")
        HOSPITAL = split[0] /* Hospital Center */
        ID = split[1]
        //basename = fname.baseName
        /* Germline */
        if (split.size() == 4) {
            ST = split[2]
            RG = split[3]
        } else {
        /* Tumor */
            ST = split[2]
            RG = split[4]
        }
        ID_RG = ID + "-" + RG
        is_tumor = (RG ==~ /R[0-9]+/)
        is_germline = (RG == "GL")

        fname = fname_in
        fname_bai = fname_in + ".bai"
    }

    def params() {
        if (this.p.size() > 0) {
            return "-" + this.p.collect { "${it.key}_${it.value}" }.join(",")
        } else {
            return "-0_0" // No Params
        }
    }

    def fout() {
        return "${this.ID}-${this.RG}${this.params()}"
    }

    // Serialization method
    @Override
    public String toString() {
        return this.fout()
    }

}
Paolo Di Tommaso
@pditommaso
Jan 08 18:37
why not just an hash map ?
Daniel E Cook
@danielecook
Jan 08 18:37
sry thats a bit much - but basic idea is using a channel you can modify the object and pass in arguments for a function...
HashMap would work but this gives me more flexibility...for instance I have the fout method for generating a slug that brings together all the arguments for an output
And I can use that for outputting to a file...makes it easy to compare say 3 different sets of options for a command line tool without having to pass so many items between processes
I can't seem to get it to cache, but hopefully you see the idea.
Paolo Di Tommaso
@pditommaso
Jan 08 18:40
input files still needs to be declared as input:, same for outputs
Daniel E Cook
@danielecook
Jan 08 18:47
yes - it is a bit redundant I do wind up passing the fname to a file() input. Might just switch back to a HashMap as its simpler
Paolo Di Tommaso
@pditommaso
Jan 08 18:48
there's an example here
you can even extends an hashmap with your custom logic
Daniel E Cook
@danielecook
Jan 08 20:50
Thank you that is very helpful