Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • May 23 21:50
    codecov[bot] commented #141
  • May 23 21:49
    codecov[bot] commented #141
  • May 23 21:49
    codecov[bot] commented #141
  • May 23 21:47
    tom-tan synchronize #141
  • May 23 21:42
    tom-tan opened #141
  • May 23 21:01
    tom-tan commented #139
  • May 23 20:20

    mr-c on main

    Update Python versions tested o… (compare)

  • May 23 20:20
    mr-c closed #140
  • May 23 20:20
    codecov[bot] commented #140
  • May 23 20:20
    mr-c synchronize #140
  • May 23 20:20
    mr-c closed #119
  • May 23 20:19

    mr-c on main

    Add `--exclude-tags` option (compare)

  • May 23 20:19
    mr-c closed #139
  • May 22 10:06
    codecov[bot] commented #140
  • May 22 10:06
    codecov[bot] commented #140
  • May 22 10:06
    codecov[bot] commented #140
  • May 22 10:06
    codecov[bot] commented #140
  • May 22 10:06
    codecov[bot] commented #140
  • May 22 10:04
    kinow edited #140
  • May 22 10:04
    kinow opened #140
Michael R. Crusoe
@mr-c:matrix.org
[m]
either drop the inputBinding and use $(inputs.sqlStatement) (instead of $4); or keep the inputBinding and drop the $4
bzeyben
@bzeyben:matrix.org
[m]

Cool.
I am using:
arguments: [ -c, sevenbridges, -q, $(inputs.sqlStatement) ]

inputs:
sqlStatement:
type: string

it worked!
I kept the $(inputs.sqlStatement), just looks cleaner. ;)
Thank you very much.
Michael R. Crusoe
@mr-c:matrix.org
[m]
You are welcome; enjoy!
bzeyben
@bzeyben:matrix.org
[m]
How can I eliminate the unnecessary parameters in my Docker Requirements.
I am getting additional info that is causing some problem with my logs.
I would like to eliminate the following in parameters:
--mount=type=bind,source=/private/tmp/docker_tmpioxvy1rg,target=/WXCNAB \
--mount=type=bind,source=/private/tmp/docker_tmpimf66_tj,target=/tmp \
--workdir=/WXCNAB \
--read-only=true \
--user=501:20 \
Michael R. Crusoe
@mr-c:matrix.org
[m]
There's no particular option to change that
bzeyben
@bzeyben:matrix.org
[m]
Why does it do that in the first?
If I need to mount anything I should be doing that explicitly.
Same with --user, --workdir, ...
Michael R. Crusoe
@mr-c:matrix.org
[m]
You need a working directory and temporary directory
bzeyben
@bzeyben:matrix.org
[m]

I have only:

cwlVersion: v1.2
class: CommandLineTool

requirements:
DockerRequirement:
dockerPull: sbg-snowflake
NetworkAccess:
networkAccess: true

baseCommand: []

arguments: [ -c, sevenbridges, -o, log_level=DEBUG, -q, $(inputs.sqlStatement) ]

inputs:
sqlStatement:
type: string

Michael R. Crusoe
@mr-c:matrix.org
[m]
It is the responsibility of the CWL runner to do the mounting
https://www.commonwl.org/v1.2/CommandLineTool.html#Runtime_environment
bzeyben
@bzeyben:matrix.org
[m]
my docker image has it's own /tmp dire and it's own user.
the docker image should be self contained and anything should be mapped explicitly
Michael R. Crusoe
@mr-c:matrix.org
[m]
This is the CWL model
bzeyben
@bzeyben:matrix.org
[m]
Overwriting the parameters is also not possible. :(
It really doesn't make sense though. It limits folks what kind of docker containers they can create.
Is this with all version of CWL versions?
Michael R. Crusoe
@mr-c:matrix.org
[m]

All versions of CWL

Use of implicit ENTRYPOINT or CMD are discouraged due to reproducibility concerns of the implicit hidden execution point (For further discussion, see https://doi.org/10.12688/f1000research.15140.1).

https://www.commonwl.org/v1.2/CommandLineTool.html#DockerRequirement

bzeyben
@bzeyben:matrix.org
[m]
:( My entrypoint is specific to the container and the CMDs are there to overwrite if I want to.
Thank you I will read the docs and see.
I will see if the CommandLine Tool would be better after all.
Michael R. Crusoe
@mr-c:matrix.org
[m]
It isn't forbidden, but just not recommend
bzeyben
@bzeyben:matrix.org
[m]
if it s not forbidden, how can I overwrite it?
Michael R. Crusoe
@mr-c:matrix.org
[m]
Hmmm. I thought you got it to work already?
bzeyben
@bzeyben:matrix.org
[m]
I have changed the command parameters in the entrypoint to reflect the real command. teh CLI is looking for an existing dir in the config.
switching this to a CommandLineTool and baseCammand as docker and teh rest as arguments shows a different error.
INFO [job sqlsect] /private/tmp/docker_tmpxls1zfbd$ docker \
run \
-it \
--rm \
sbg-snowflake \
-c \
sevenbridges \
-q \
'SELECT 1 AS seq, CURRENT_VERSION() as version, CURRENT_TIMESTAMP() as timestamp;'
the input device is not a TTY
Never mind CommandLineTool worked. I changes -it to just -i.

cwlVersion: v1.2
class: CommandLineTool

baseCommand: docker

arguments: [run, -i, --rm, sbg-snowflake, -c, sevenbridges, -q, $(inputs.sqlStatement)]

inputs:
sqlStatement:
type: string

outputs: []

I think the DockerRequirement would be cleaner.
Bruno P. Kinoshita
@kinow:matrix.org
[m]
Have the permissions of the github repositories or organizations changed today/yesterday? I seem to not be able to assign issues to myself on the common-workflow-language project, nor submit branches to the cwl-1.2.1 repository anymore (did both this week, pushed a branch to 1.2.1 last night).
1 reply
Michael R. Crusoe
@mr-c:matrix.org
[m]
Huh, I might have made a mistake. I'll take a look.
Bruno P. Kinoshita
@kinow:matrix.org
[m]
All good now Michael R. Crusoe , thanks!!!!
Bruno P. Kinoshita
@kinow:matrix.org
[m]
Same problem with the cwlviewer repository now 🙏
kinow@ranma:~/Development/java/workspace/cwlviewer$ git push -u upstream update-dependencies 
ERROR: Permission to common-workflow-language/cwlviewer.git denied to kinow.
...
Michael R. Crusoe
@mr-c:matrix.org
[m]
Can you use a fork for now? We don't really collaborate on branches there, so there isn't much value
Bruno P. Kinoshita
@kinow:matrix.org
[m]
Sure thing 👍️
Michael R. Crusoe
@mr-c:matrix.org
[m]
Today's CWL EMEA-Americas video chat starts in less than 5 minutes
Agenda & notes: https://w3id.org/cwl/meeting_minutes
Direct link to video room: https://matrix.to/#/#cwl_conf:matrix.org
Bart Nijsse
@bart_n:matrix.org
[m]
Hi, how to only execute a conditional step when a specific (optional) input parameter is defined?
I tried : when: $(inputs.in2 !== null)
and different variations but cannot seem to get the right result
Michael R. Crusoe
@mr-c:matrix.org
[m]
Do you get any error messages?
1 reply
Michael R. Crusoe
@mr-c:matrix.org
[m]
Yeah, that is tricky; glad you got it working!
Michael R. Crusoe
@mr-c:matrix.org
[m]
Today's CWL video chat starts now!
Agenda & notes: https://w3id.org/cwl/meeting_minutes
Direct link to video room: https://matrix.to/#/#cwl_conf:matrix.org
Michael R. Crusoe
@mr-c:matrix.org
[m]
Douglas Lowe
@douglowe
Is there any way in the CWL ExpressionTool to read a directory (not the working directory) of a docker container, pull the filelist, and (selectively) copy these files into a directory object to pass back to my workflow?
the last 2 parts of this (I think) I have working - but I can't get the script to read a directory which is not the working directory
Michael R. Crusoe
@mr-c:matrix.org
[m]
A CWL ExpressionTool can take CWL objects and rearrange the data structures; this includes combining files and directories into a new directory with a particular layout. None of that is done using a docker container (just javascript).
Douglas Lowe
@douglowe

that looks good - but, can I take files that pre-exist in a docker container, rather than passing them into the workflow as an input, and use them in my workflow?

This is the expression tool I'm working on: https://github.com/UoMResearchIT/wrf_emep_cwl_workflow/blob/master/cwl/create_run_dir.cwl - the directory I'm interested in is /usr/local/WRF/run, which is included with the software as a template running directory

Michael R. Crusoe
@mr-c:matrix.org
[m]
Ah, you want to extract files that are baked into a docker container?
That is not really the CWL way, but we can hack something together
Make a CommandLineTool that copies the files you want into the current working directory (. or $(runtime.outdir)); then glob them up like any other outputs
Douglas Lowe
@douglowe

Yes, it's definitely a bit of a hack - but it fits with the general working paradigm of the model itself (which can be a bit archaic in places).

I'll switch to a commandline tool. I'll try outputEval as well as glob, for the exclusion of the few files I don't need.

Bruno P. Kinoshita
@kinow:matrix.org
[m]
Great to see WRF running with CWL @douglowe! Some time ago I tried comparing it to the UM+Cylc but I couldn't find an easy/reproducible example to play with 🙂 Will check out and try to run that workflow later.