common-workflow-language/common-workflow-language

https://cwl.discourse.group

bogdang989

@bogdang989

Am I missing something or is the CWL v1.1 spec page missing an option to have strings in secondaryFiles? I am looking at https://www.commonwl.org/v1.1/CommandLineTool.html#CommandInputParameter which says it can be SecondaryFileSchema or a list of SecondaryFileSchema but not that elements of the list can also be string/expression. I would expect that there was something in https://www.commonwl.org/v1.1/CommandLineTool.html#SecondaryFileSchema that would mention that use case, but there isn't as well.
I know that the resolving of cwlv1.1 apps works fine for strings in secondaryFiles, but someone looking for the first time would not expect strings to be allowed there

Michael R. Crusoe

@mr-c

@bogdang989 did you check the v1.1 schema salad docs?

I know that isn't the best place for this particular piece of information...

bogdang989

@bogdang989

Thanks a lot @mr-c , this (https://www.commonwl.org/v1.1/SchemaSalad.html#Domain_Specific_Language_for_secondary_files) is exactly the info I was hoping to see in the spec. Would it make sense to somehow add this to https://www.commonwl.org/v1.1/CommandLineTool.html#SecondaryFileSchema (at least a link to the SchemaSalad spec)

Michael R. Crusoe

@mr-c

@bogdang989 you're welcome. I agree, can you open an issue or send a PR?

bogdang989

@bogdang989

:thumbsup:

Peter Amstutz

@tetron

happy new year everyone!

Denis Yuen

@denis-yuen

:+1:

Michael R. Crusoe

@mr-c

w00t!

Ian

@ionox0

:fireworks:

Matthew Garvin

@mattgarvin1

@tetron , @mr-c I really wish we could go back in time and change all the places in the spec where we allow a map and list to just map. It makes writing code so much more cumbersome ...

I see that this issue has been mentioned - I was wondering if there was any plan or initiative to go one way or another with the schema, as far as uniformly requiring the list data structure or uniformly requiring the map structure?

additionally, for all the places in the schema where the option is given for the user to provide either type <T> or an array of type <T> - this seems like another instance where, why not just define the schema such that the type is always an array, and if the user has only one value to put there, then it's a list with one item in it, and that would be fine

Michael R. Crusoe

@mr-c

Goodbye Python 2! https://github.com/common-workflow-language/cwltool/pull/1231#merged-event

Peter Amstutz

@tetron

@mattgarvin1 so technically, the underlying data model only allows the list data structure, the complexity comes because there's a pre-processing layer on top that accepts maps and turns them into lists

@mattgarvin1 and so if you're dealing with CWL as its actually written and not the normalized (i.e. after processing) version then it is more complex

Matthew Garvin

@mattgarvin1

@tetron I guess my question is then, why is it "advertised" to users of CWL that they can choose either data structure? why not just say "this is a list - be sure to use a list."

Peter Amstutz

@tetron

the map version is easier to read

Matthew Garvin

@mattgarvin1

alright, then why not enforce the map data structure for writing CWL, and then have whatever processing tools expect the map and just turn it into a list for further processing

Peter Amstutz

@tetron

well, except then the normalized version would no longer be valid and it becomes an incompatible dialect, there's tradeoffs each way

the next question should be, "so why is the normalized version using lists?" and the answer is "so a normalized CWL document would be valid JSON-LD and convertable to a knowledge graph" which is a cool feature that most users don't care about

Matthew Garvin

@mattgarvin1

I maintain that, regardless of the details, picking one method of representation and sticking to it for writing and processing CWL in this case might be a valuable action to take

re: the maps vs. list issue, as well as the <T> vs. array of <T> issue

allowing for those different representations makes processing CWL preventably clumsier and more painful than it would otherwise be

as other folks have also already pointed out

Kaushik Ghose

@kaushik-work

@tetron I don't know why the map form can't be used for a knowledge graph, but the map form is convertible to a list (and poses no possibility for error, unlike the list, which maybe missing an id field), so I don't understand why this is a blocker.

Also, are we meeting today? Or is it too early in the year?

Michael R. Crusoe

@mr-c

CWL video chat starting soon

https://docs.google.com/document/d/1gbYHY52fuhaLsQWjTz3WUnUThn1wehBMEaFObq0DR8s/edit

https://meet.jit.si/cwl

Michael R. Crusoe

@mr-c

@mattgarvin1 we're discussing map vs. list form right now at https://meet.jit.si/cwl

@mattgarvin1 notes at https://docs.google.com/document/d/1gbYHY52fuhaLsQWjTz3WUnUThn1wehBMEaFObq0DR8s/edit

Michael R. Crusoe

@mr-c

@mattgarvin1 common-workflow-language/common-workflow-language#892

Evan Clark

@djevo1_gitlab

I have created a poll to see what dates work best for an official NYC CWL Meetup. Please fill in which two dates (Thursday & Friday) work best for you. https://doodle.com/poll/e4dw4ccetg9x536t

Matthew Garvin

@mattgarvin1

sorry, just checking this now, is the video chat still going on?

@mr-c thanks for the notes and sharing the proposal - looks great!

Peter Amstutz

@tetron

@mattgarvin1 the meeting just ended, but I think we addressed your concerns, it won't be a short-term change but could happen in CWL 2.0

Matthew Garvin

@mattgarvin1

@tetron @mr-c sounds good - thanks so much for taking action so promptly and discussing the issue

Peter Amstutz

@tetron

Lyft made a workflow engine https://eng.lyft.com/introducing-flyte-cloud-native-machine-learning-and-data-processing-platform-fb2bb3046a59

Andrey Kartashov

@portah

Looks cool especially designed for ML

And Lyft still using Apache Airflow a lot. So, looks like for ML was not enough

Peter Amstutz

@tetron

@mr-c I have a conflicting meeting, I might be late to the CWL video chat today

Michael R. Crusoe

@mr-c

@tetron okay

Michael R. Crusoe

@mr-c

Speaking of :-) https://docs.google.com/document/d/142vLA5t2AdHG-fp5cxjzOLnWEic2zeVWvUaHZ_kZ2MI/edit

https://meet.jit.si/cwl

Might just be me today...

Tom Morris

@tfmorris

Quotes from Gingko Bioworks on CWL, WDL, & worfklow engines (to consider when deciding how to promote/counter) https://youtu.be/arDI64ja6KA?t=2121

Michael R. Crusoe

@mr-c

I think only one of those statements is about CWL directly

(perceived) weaknesses in the ecosystem do challenge the perception of the whole

Peter Amstutz

@tetron

the CWL chat yesterday we talked about communications/marketing and dividing the web site into different sections for different audiences

Michael R. Crusoe

@mr-c

Microsoft is looking for a Senior Scientist (Bioinformatics) with CWL experience https://www.linkedin.com/jobs/view/1683549207/

Where communities thrive

People

Repo info

Activity