In one of those cases where my personal life intersects with my work life: https://www.apotelyt.com/photo-lens/mft-compatibility
What has this got to do with anything? You ask. Reminds me that adhering to a standard constrains somethings, but not others and does not guarantee a uniform user experience.
BeeCWL https://sc19.supercomputing.org/proceedings/tech_poster/tech_poster_pages/rpost235.html (includes links to the poster PDF and extended abstract)
Have no idea why. However, I'm surprised there is no CWL mentioned
I've copied the design document over to the CWL v1.2 repo. I think the discussion in the pull request is very valuable. So I'm going to copy the discussion over. I apologize for some spam that discussion participants will get as a result of this - I will tag them for their comments
Gitter just deleted my comment. Maybe I should simplify it anyway - should the doc generator handle
doc: attributes that are defined as lists (e.g. on field definitions or record type definitions) - or is this only valid for type: documentation? If it is a bug, I can fix it - I just wanted to verify it is a bug
What does the metaschemas say?
Hi all, can someone please help me understand the usage of inputBinding in CommandInputRecordSchema and interaction with inputBinding in the fields? E.g. what should a simple record defined like this produce?
id: input
type:
inputBinding:
position: 3
valueFrom: c
type: record
fields:
- name: input_field
type: string?
inputBinding:
position: 2
valueFrom: b
name: input
I feel this kind of scenario can be better addressed in both the spec and conformance tests. I'll put up an issue on the github repo anyway, but thought I ask here first just to make sure I am not missing anything
CWL video chat is starting https://docs.google.com/document/d/1iUCSUtkd9xBYfjAdDukSyZQqGhnw1FEBhevpxt4jss0/edit?usp=sharing
Hi everyone, just wondering if there's a way to slightly rename an output file after the tool is run. Essentially, GATK MergeSamFiles is generating the index as ^.bai, but I need it as .bai on input, and then on the glob as well.I could manually brute force some mv statements in there with && and shell=True, but less keen if there's a better way.
The more tools I explore, the more confused I am that there seems to be no consistency or standard in this respect.
Hi @mr-c , I've been playing with rewriting the secondaryFiles, but inside the output glob it looks like there isn't a secondaryFiles property.
cwlVersion: v1.0
class: CommandLineTool
baseCommand: "ls"
inputs:
bam:
type: File
secondaryFiles: ["^.bai"]
outputs:
std: stdout
out:
type: File
secondaryFiles: ["^.bai"]
outputBinding:
glob: $(inputs.bam.basename)
outputEval: |
${
console.log(self)
self[0].secondaryFiles[0].basename=".bai"
}
requirements:
InitialWorkDirRequirement:
listing:
- $(inputs.bam)
InlineJavascriptRequirement: {}(Formatted error)
('Error collecting output for parameter \'out\':Expression evaluation error:Expecting value: line 1 column 1 (char 0)script was:
01 "use strict";
02 var inputs = {
03 "bam": {
04 "class": "File",
05 "location": "file:///Users/franklinmichael/Desktop/tmp/bamsplit/BRCA1.bam",
06 "size": 2997846,
07 "basename": "BRCA1.bam",
08 "nameroot": "BRCA1",
09 "nameext": ".bam",
10 "secondaryFiles": [
11 {
12 "location": "file:///Users/franklinmichael/Desktop/tmp/bamsplit/BRCA1.bai",
13 "basename": "BRCA1.bai",
14 "class": "File",
15 "nameroot": "BRCA1",
16 "nameext": ".bai",
17 "path": "/private/tmp/docker_tmplo7vt97z/BRCA1.bai",
18 "dirname": "/private/tmp/docker_tmplo7vt97z"
19 }
20 ],
21 "path": "/private/tmp/docker_tmplo7vt97z/BRCA1.bam",
22 "dirname": "/private/tmp/docker_tmplo7vt97z"
23 }
24 };
25 var self = [
26 {
27 "location": "file:///private/tmp/docker_tmplo7vt97z/BRCA1.bam",
28 "path": "/private/tmp/docker_tmplo7vt97z/BRCA1.bam",
29 "basename": "BRCA1.bam",
30 "nameroot": "BRCA1",
31 "nameext": ".bam",
32 "class": "File",
33 "checksum": "sha1$8a38a4e8d58c91d9aaca8cdc739e05eddf7dec1d",
34 "size": 2997846
35 }
36 ];
37 var runtime = {
38 "cores": 1,
39 "ram": 1024,
40 "tmpdirSize": 1024,
41 "outdirSize": 1024,
42 "tmpdir": "/private/var/folders/jz/y9gqxt_s7jxcjkc26gr71ywr7zs5yz/T/tmpoer80xbz",
43 "outdir": "/private/tmp/docker_tmplo7vt97z"
44 };
45 (function(){
46 console.log(self)
47 self[0].secondaryFiles[0].basename=".bai"
48 })()stdout was: \'\'stderr was: \'evalmachine.<anonymous>:
47 self[0].secondaryFiles[0].basename=".bai" ^TypeError: Cannot read property \'0\' of undefined at evalmachine.<anonymous>:47:
27 at evalmachine.<anonymous>:48:3 at Script.runInContext (vm.js:107:20) at Script.runInNewContext (vm.js:113:17) at Object.runInNewContext (vm.js:296:38) at Socket.<anonymous> ([eval]:11:57) at Socket.emit (events.js:182:13) at addChunk (_stream_readable.js:283:12) at readableAddChunk (_stream_readable.js:260:13) at Socket.Readable.push (_stream_readable.js:219:10)\'', {})My guess is this is because the "outputEval" block is before the secondaryFiles block
I checked out the source code and moved the outputEval block below the secondary files, and this works.
@illusional Alas, the order of operations you want to change is set in the standard itself: https://www.commonwl.org/v1.0/CommandLineTool.html#CommandOutputBinding
Perhaps you can use a generous glob that catches all the files you need, assemble & rename them in the
outputEval, including attaching the index to the primary file in a synthesized secondaryFiles property. Then hopefully this full object will be passed without problems through the secondaryFiles: .bai processing stage.
That may or may not work with
cwltool and I'd have to think on it more to see if it is compliant with the standard. Maybe @tetron has an idea.
It may just be easier, and less surprising to CWL engines, to put an expression in the
secondaryFiles field of your output and not use outputEval, as you are allowed to return a File object and thus can do the regular renaming trick
I'm happy to use a CWL expression to do this, but I don't quite understand what I should return if:
- The primary file (correctly globbed) is called
myfile.bam - A file called
myfile.baiis present in the output directory - The returned secondary file should have the extension
myfile.bam.bai
This would ensure it's correctly localised in future steps. But maybe I don't quite understand how secondary files are passed around in CWL especially CWLTool
Thanks @keiranmraine, I'll ask around at work tomorrow to see if it's something we can convince people to use. Looks like it's only been in htsjdk since earlier this year and idk how quickly tools tend to update. Have you got any good reading stuff about how well it's adopted?
Handling bam + bai|csi and cram + crai for a single tool gets complicated. We ended up having to define multiple CWLs:
https://github.com/cancerit/dockstore-cgpmap/tree/develop/cwls
More elegant solutions would be preferable, but we have to ensure they are also compatible (or will be compatible) with the Dockstore registry.
We are likely to migrate fully to CRAM for new data quite soon, so
csi is not going to be that much of an issue for us. CRAM with modified split size (1k, instead of 10k) is as fast and sometimes faster than BAM to parse now (with samtools/htslib). If I was tied to BAM (legacy tools can be a pain), and setting up something new I'd certainly go with csi.