the schema is used to validate the workflow, but gxformat2 contains functions for the conversion that don't depend on the schema
the schema is an attempt to model the format - I wouldn't describe it as the format :|
Unlike CWL say where it is pretty explicitly the format. I'm just not good enough at JSON LD or schema salad to make that perfect. Maybe someday
that question is a bit too philosophical for me 😅, if I was forced to give a definition of the format I'd make it an empirical one based on what Galaxy runs.
yes that's kind of what I meant... so where is the schema actually used?
aha, so it should co-evolve with the code for linting purposes... nod
Carole just proposed we should be renaming to "Intergalactic Workflow Constellation" :)
Question: does the
attributes: {} in IWC tests for collection outputs have any purpose?
Apart from causing tests to fail if it's not included, I mean 😛
Yeah, but I think we fixed that in a more recent galaxy-tool-util release ?!
Is that still failing in CI ?
this one's newer and doesn't specify attributes: https://github.com/galaxyproject/iwc/blob/1d69747e43ec0db6bc09c87a9e70389482b52690/workflows/data-fetching/parallel-accession-download/parallel-accession-download-tests.yml
Hello, I'm currently encountering the following Planemo error for the last step of my workflow:
Traceback (most recent call last):
File "/home/simon/GitRepos/planemo/planemo/galaxy/activity.py", line 50, in execute
return _execute(ctx, config, runnable, job_path, **kwds)
File "/home/simon/GitRepos/planemo/planemo/galaxy/activity.py", line 219, in _execute
run_response.collect_outputs(ctx, output_directory)
File "/home/simon/GitRepos/planemo/planemo/galaxy/activity.py", line 366, in collect_outputs
output_src = self.output_src(runnable_output)
File "/home/simon/GitRepos/planemo/planemo/galaxy/activity.py", line 567, in output_src
raise Exception("Failed to find output [%s] in invocation outputs [%s]" % (output_name, invocation["outputs"]))
There is a corresponding message in the Galaxy logs:
galaxy.workflow.run DEBUG 2021-10-27 17:46:24,479 [pN:main.web.1,p:11036,w:1,m:0,tN:WorkflowRequestMonitor.monitor_thread] Failed to find expected workflow output [outp_col] in step outputs [{'outp': <galaxy.model.HistoryDatasetAssociation(2228) at 0x7f4a0e35a668>, 'outp_frict': <galaxy.model.HistoryDatasetAssociation(2229) at 0x7f4a0e35aa20>}]
yes
there's a bunch of different possible causes, like filtered out outputs or bugs
fwiw, maybe it's related, but whenever I open the workflow in the editor, it always tries to delete the final connection (to the problematic output) if I try and modify the tool
anyway, I was planning to make an IWC PR, so I will push it there
I can also try and create a more minimal case
it seems it isn't possible to add citations to a workflow, as it is for tools?
would be a useful feature imo
I'm prepping to look into the minimal workflow you posted Simon Bray , one thing I noticed is that gmx_sim isn't at the latest version. Might be totally unrelated of course
arg, I can't read, I just didn't install the right version it looks like
<collection name="outp_col" type="list" label="Free energy data">
<discover_datasets pattern="(?P<designation>^cluster\d+)_outp\.txt$" ext="tabular"/>
<filter>
json
</filter>
</collection>
<collection name="outp_frict_col" type="list" label="Friction data">
<discover_datasets pattern="(?P<designation>^cluster\d+)_ofrict\.txt$" ext="tabular"/>
<filter>
json
</filter>
</collection>
<data name="outp" from_work_dir="outp.txt" format="tabular" label="Free energy data">
<filter>
not json
</filter>
</data>
<data name="outp_frict" from_work_dir="ofrict.txt" format="tabular" label="Friction data">
<filter>
not json
</filter>
</data>that seems to be the problem, the workflow defines a workflow output outp_col, but it seems that output is being filtered out because there's no input connected to Ensemble subgroups (the json parameter). Does that make sense ? I think this might be one of the cases where the workflow editor cannot accurately predict the result of the filter
Since otherwise you'd expect to have labelled
outp, correct ?
Thanks for having a look Marius!
I will test it tomorrow on the real workflow, but I'm sure that's it
Could I have a review of galaxyproject/iwc#68, when someone has time, from a technical, IWC point of view?
Chris Barnett agreed to review it from the scientific side
If you want to get the tests passing you could take a look into either installing the tools on usegalaxy.org via https://github.com/galaxyproject/usegalaxy-tools or seeing if we can remove
--no_shed_install from https://github.com/galaxyproject/planemo-ci-action/blob/main/planemo_ci_actions.sh#L6