by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Patrick Magee
    @patmagee
    I would agree that's this is more an engine configuration and not a specific language issue. The pr you linked also adds hints which is a way to specify metadata like this to engines which understand it. Imo I do not think the spec should have first class support for this but we shall see how the discussion goes!
    Matthew Pocock
    @drdozer
    Thanks
    Eric T. Dawson
    @edawson
    @drdozer If you still want examples, I have some in this repo and a few more (though some out of date) in this one
    Matthew Pocock
    @drdozer
    mm, getting an exception
    trying to call toUpper on an input variable, getting NoSuchMethodException for WdlStadnardLibraryFunction$$annon$1.toUpper(scala.collection.Seq)
    Matthew Pocock
    @drdozer
    weird - I can't actually see any definiton for toLower or toUpper in the cromwell source
    are these in the spec but not in cromwell?
    Patrick Magee
    @patmagee
    @drdozer they do not exist in the spec eitehr
    Jeff Gentry
    @geoffjentry
    That have to have been hand typed. WdlStadnard…, annon ….
    Matthew Pocock
    @drdozer
    @patmagee Ouch - I'd got the wdl docs muddled up with the ms workflow description language docs -- how embarasing
    Jeff Gentry
    @geoffjentry
    toUpper looks to be a scala string operator
    Chris
    @0x5f3759df_gitlab
    in the scatter gather example, you show how to iterate over an array of files... how do you iterate over a Map[String, File]?
    in a workflow block
    and get both elements
    Michael Franklin
    @illusional
    Chris
    @0x5f3759df_gitlab
    cool thanks.
    Matthew Pocock
    @drdozer
    can I ask cromwell to run a wdl file in an incremental update mode?
    Jeff Gentry
    @geoffjentry
    What do you mean?
    Matthew Pocock
    @drdozer
    @geoffjentry So during development I have a wdl workflow - I run it, then I add another task, then run it again. Ideally it would do the minimal extra work to re-run the workflow by stealing results from the previous run.
    Jeff Gentry
    @geoffjentry
    Gotcha. If you have a persistent database configured or if you’re using Cromwell in server mode w/ an in memory DB and haven’t turned it off in between, it’ll Just Magically Work like that via call caching
    Matthew Pocock
    @drdozer
    thanks - I'll look at that
    Evan Clark
    @djevo1_gitlab
    Anyone here use cromwell and cwl, having this issue with packed cwl Cannot find a tool or workflow with ID 'None'
    Patrick Magee
    @patmagee
    @djevo1_gitlab I woud suggest going over to the cwl channel or cromwell documentation
    Evan Clark
    @djevo1_gitlab
    @patmagee Thanks, is there by chance a cromwell channel?
    Jeff Gentry
    @geoffjentry
    No. There’s a slack but it’s unofficial and there’s no guarantee someone from the team will answer. If that didn’t dissuade you send me an email for you and I can send you an invite. Otherwise your best bet is to open a ticket on github
    Anton Kulaga
    @antonkulaga
    is it active? It is stil better than nothing
    It looks like that cromwell team is mostly focused on cloud API while needs of users with localbackend are often neglected
    Jeff Gentry
    @geoffjentry
    it == the slack channel? Not very, but there are some folks who post there
    The Cromwell team needs to focus on their own priorities first, that’s what pays the bills.
    vincent
    @vincent-octo
    Hi there, is it possible to specify a command like python myscript.py in WDL?
    When I try this I have the error: [First 300 bytes]:python: can't open file 'myscript.py': [Errno 2] No such file or directory
    The file myscript.py is right next to my wdl file.
    I run it with cromwell run test_failure.wdl. This file content:
    workflow TestFailure {
        call myScript
    }
    
    task myScript {
        command {
            python myscript.py
        }
        output {
            String out = read_string(stdout())
        }
    }
    Patrick Magee
    @patmagee
    Hello @vincent-octo . Regardless of whether you use cromwell, miniwdl or another execution tool, wdl runs tasks in containerized environments using docker or some other container technology.
    Running complex scripts with commands is 100%possible and in fact one of the things wdl excels at
    However the wdl in general knows nothing about the file system its stored in, or the files that are beside it
    In order to accomplish what you are trying to do above you have two options. You can either create a new docker image and back the python script directly into it. Or you can specify an input to your task that is a file, then pass the file in as an argument when you run the pipeline
    Patrick Magee
    @patmagee
    for example
    workflow TestFailure {
        call myScript
    }
    
    task myScript {
        input {
          File script
        }
        command {
            python ~{scrip}
        }
        output {
            String out = read_string(stdout())
        }
    }
    then when running it would be : cromwell run -i "TestFailure.myScript.script:./myscript" test_failure.wdl
    Michael Franklin
    @illusional
    Or within that command block you can write your file into the current execution directory (using echo > myfile.py) and then calling python myfile.py
    vincent
    @vincent-octo
    I see, that makes sense. Thanks for your answers!
    Andrew Birnberg
    @birnbera
    Hello, Are there any tools out there that allow you to build a workflow as an AST and then automatically serialize it to, e.g. WDL or CWL? It looks like the miniwdl python package allows you to parse a WDL file and then traverse the AST, but they don't seem to have an built-in methods to serialize a modified or new AST back to WDL.
    Michael Franklin
    @illusional
    Yeah not an AST and not super great either, I build python-wdlgen to intend to do that (but it doesn’t, just more abstracts some higher level concepts to better format the code) and there’s some conversation on miniwdl here: chanzuckerberg/miniwdl#216
    I’d be interested if you find anything though
    Andrew Birnberg
    @birnbera
    Cool thanks for the info. If I come across something (or prototype anything!!!) I'll definitely post here.
    Michael Franklin
    @illusional
    To help your searching, you’re looking for prettyprinting an AST
    Evan Clark
    @djevo1_gitlab
    Anyone ever see cromwell print this and then do nothing? CWL (Unspecified version) workflow 561a864b-2196-4d9c-ac6b-425e6ab81a64 submitted
    Patrick Magee
    @patmagee
    Hey @djevo1_gitlab, for cromwell or cwl related issues I suggest going to their support forums! You are more likely to find the answer
    Javad Noorbakhsh
    @javadnoorb

    Hi. I'm trying to give two arrays to a task in wdl and then loop through an index and access both arrays elementwise. Here's an example:

    task merge_all_beds {
    Array[File] input_bed_files
    Array[File] arxspan_ids
    command <<<
    
        set -euxo pipefail
        input_bed_files_=${sep=' ' input_bed_files}
        arxspan_ids_=${sep=' ' arxspan_ids}
    
        # add new columns corresponding to cell id
        for i in ${!input_bed_files_[*]}
        do
            input_bed_file_basename=$(basename ${input_bed_files_[$i]})
            sed -i "s/$/\t${arxspan_ids_[$i]}\t${input_bed_file_basename}/" ${input_bed_files_[$i]}
        done
    
        cat ${sep=' ' input_bed_files} > interval_read_counts.bed
    >>>

    ...

    Is this not possible in wdl?

    These are small tasks so I don't want to use scatter/gather and wait for new VMs to start.
    Ohad Rodeh
    @orodeh
    The short answer is that, for what you are trying to do here, the task you wrote is the best way to deal with it.
    You can zip the two arrays together and access the pairs in a scatter loop. However, that will have the undesired behavior you mentioned.