in any old venv where
planemo is installed
(but I'm 80% sure that's not quite the correct command, trying to find a reference for something that might work.)
it's definitely with planemo
yeah, most tool tests will at least give you a better feeling for is it a tool problem, or, is it a cvmfs problem somehow
Oh, ok, I'm wrong. The command is in another library :| You need to install ephemeris. https://ephemeris.readthedocs.io/
Then given a file like:
$ cat test.galaxy.yml
tools:
- name: rna_starsolo
owner: iuc
tool_panel_section_label: Mapping
tool_shed_url: toolshed.g2.bx.psu.eduYou can run:
shed-tools test -a $GALAXY_API_KEY -g $GALAXY_URL --parallel_tests 10 -t test.galaxy.yml
(thanks martenson for having a repo that does it.)
And it'll look something like
$ shed-tools test -a $GALAXY_API_KEY -g https://usegalaxy.eu --parallel_tests 1 -t test.galaxy.yml
Storing log file in: /tmp/ephemeris_raqm6rsc
Executing test 'toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.8a-0'
good luck @martinwolst
John Chilton: I don't know how but when I configured a Pulsar destination for Singularity and set it to local dependency resolution (since this particular destination has CVMFS mounted), Galaxy generated an actual
tool_script.sh containing the correct paths for the remote, and sent it over with the job.
Which is awesome.
Tool and job output streams are still merged but this fully isolates the tool execution environment from the job.
(which admittedly is the fault of element's threading behaviour more than anything, but, it's a bit nicer reference too, for future people who encounter the same problem)
the error displayed on the interface is
Error generating display_link: No conversion path found for data param: bgzip_file
I may have lost/changed something. It seems to be related to the function
find_conversion_destination_for_dataset_by_extensions in lib/galaxy/datatypes/registry.py
After taking the last 3 months off, PaperCuts is back. The September Papercuts is this Thursday. PaperCuts is an online, community wide event to address relatively easy to fix friction points in any part of Galaxy.
We will be on Gitter for chat all day long, and on 3 calls spread across the day. Please take advantage of both to communicate with your collaborators around the world.
- Call 1: Oceania, Australia, Asia
- 14:00 Australia Eastern time. See your time.
- Zoom Link
- Call 2: Middle East, Europe, Africa
- 10:00 Central European Summer time. See your time.
- Zoom Link
- Call 3: Americas
- 12:00 US Eastern time. See your time.
- Zoom Link
Need help for a question -- moving from a year old Galaxy to the latest release. Advice? Incrementally updating/migrating? Any special tips since the database is non-functional (in some way, could clarify that). https://help.galaxyproject.org/t/how-to-migrate-the-database-created-from-old-version/6684
due to some kind of install errors I have a Galaxy that has tools installed but dependencies missing... is there a way to get Ephemeris to (re)install things using the dependency manager (conda)? I see a
--skip_install_resolver_dependencies but no "force install_resolver_dependencies"...
You can make ephemeris test the tools and if you have conda auto set then it should install the deps as the tests run. Just make sure you don't set parallel_tests
shed-tools test
That's how I force install conda deps on all of my pulsar servers.
If I have half installed conda envs then I delete them first
@cat-bro and I have become ok at it due to necessity..
So, I'm running into a strange issue with a custom "data_source" tool I have. It's using json_collect_data_source to parse some json and pull in multiple files. It has been working great, but recently started giving me trouble. In an attempt to fix, I reinstalled the conda environment for it, and now, it's not activating that environment. It reports that it's using the environment, but the only thing it does is set the path to the envs bin directory. Any insights?
Other tools work fine, it's just that one
data_source tool that's not activating it's enviornment.
I'm fairly sure this was working, but I'm now wondering if I'm just going to have to get the thing running in the default (base galaxy) env?
@lparsons: does the command line generated for the job include the env activation command?
Most jobs output a whole block of activation stuff in the log, but this doesn't do that.
However, I don't think it did that before and I swear it was working before.
It would be in
tool_script.sh in the job's job directory, and the activation output (if any) would be logged to working/conda_activate.log in the job's job directory.
Example from a job on usegalaxy.org:
[ "$(basename "$CONDA_DEFAULT_ENV")" = "$(basename '/cvmfs/main.galaxyproject.org/deps/_conda/envs/mulled-v1-3ccdf3c9d403e3bad7ed8274d1f4d30fe8a1fcba83ce9681a646d5b010a03c00')" ] || {
MAX_TRIES=3
COUNT=0
while [ $COUNT -lt $MAX_TRIES ]; do
. '/cvmfs/main.galaxyproject.org/deps/_conda/bin/activate' '/cvmfs/main.galaxyproject.org/deps/_conda/envs/mulled-v1-3ccdf3c9d403e3bad7ed8274d1f4d30fe8a1fcba83ce9681a646d5b010a03c00' > conda_activate.log 2>&1
if [ $? -eq 0 ];then
break
else
let COUNT=COUNT+1
if [ $COUNT -eq $MAX_TRIES ];then
echo "Failed to activate conda environment! Error was:"
cat conda_activate.log
exit 1
fi
sleep 10s
fi
done
}
Yeah, that is not happening for the data source tool, but it does happen for others (at least in the debug logs)
Do the tool's
<requirement> tag(s) match the necessary env? You can check that the dependency resolver thinks so by checking the "Tool Dependencies" page under the Admin UI in Galaxy
Yeah, the resolver thinks so, and I see this:
galaxy.tool_util.deps DEBUG 2021-09-14 15:11:50,811 [p:22933,w:0,m:2] [SlurmRunner.work_thread-1] Using dependency json_collect_data_source version 1.0.1 of type conda
galaxy.tool_util.deps DEBUG 2021-09-14 15:11:50,812 [p:22933,w:0,m:2] [SlurmRunner.work_thread-1] Using dependency json_collect_data_source version 1.0.1 of type conda
galaxy.jobs.command_factory INFO 2021-09-14 15:11:50,930 [p:22933,w:0,m:2] [SlurmRunner.work_thread-1] Built script [/galaxy/galaxy-app/database/jobs_directory/000/312/312099/tool_script.sh] for tool command [export PATH=$PATH:'/galaxy/galaxy-app/tool-dependencies/_conda/envs/__json_collect_data_source@1.0.1/bin' ; json_collect_data_source.py '0' --json_param_file '/galaxy/files/000/571/dataset_571741.dat' --path '.' --appdata 'tmp']
Weird, that's like a
galaxy_package dependency where it just adds the bin dir to $PATH.
I wonder if it's missing
conda/activate/deactivate symlinks in the env? Although newer versions of conda shouldn't care, but maybe our resolver still does.
Is there somewhere it's looking for old style deps?
It shouldn't be looking for galaxy pacakges in
_conda/envs unless you have a non-standard resolver config (dependency_resolvers_conf.xml) (and even then it wouldn't look for __json_collect_data_source@1.0.1, it'd look for json_collect_data_source/1.0.1)