Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
Marius van den Beek
@mvdbeek
I think https://bugs.python.org/issue21161 might be a problem for cheetah templates that use list comprehensions on python 3 :(
or at least for list comprehensions that reference variables from the surrounding scope, like [i for i in $var1 if i in $var2]
Vahid
@VJalili
@nsoranzo when I remove a pinned required from dependencies/pipfiles/default/pinned-requirements.txt and run make update-dependencies, the command then empties the dependencies/pipfiles/default/pinned-requirements.txt file. Is this an expected behavior?
Vahid
@VJalili
I think it is due to some conflicts in the dependencies.
Nicola Soranzo
@nsoranzo
It is surely not expected. There should be quite a bit of verbose output from pipenv though.
tomsuntom
@tomsuntom
I am trying to run Galaxy on my mac but when I type in "sh run.sh" it says "activating virtualenv at .venv
Collecting pip>=8.1
Could not find a version that satisfies the requirement pip>=8.1 (from versions: )
No matching distribution found for pip>=8.1". Anyone know how to fix it?
Nicola Soranzo
@nsoranzo
What system is it?
Marius van den Beek
@mvdbeek
@VJalili @nsoranzo I've been having the same issues. The main problem is the genomospace cleint which pins an old boto. If this ends up in a conflict I also get an empty dependencies/pipfiles/default/pinned-requirements.txt
Christophe Antoniewski
@drosofff

Hi, I did not notice immediately, but I think that since we have updated to release_18.05, the files that are ftp-uploaded by our instance's users are not seen anymore in the "Choose FTP files" panel (warning "Your FTP directory does not contain any files."). We are still using the .ini format for the galaxy config files; the galaxy log says

galaxy.webapps.galaxy.api.remote_files WARNING 2018-08-22 13:02:33,773 [p:31389,w:2,m:0] [uWSGIWorker2Core4] The directory "'/home/galaxy/galaxy-dist/database/ftp/drosofff@gmail.com'" does not exist.

which is not true !

Any new variable to adjust with release_18.05 ?

Marius van den Beek
@mvdbeek
Any chance that '/home/galaxy/galaxy-dist/database/ftp/drosofff@gmail.com' has single quotes in the galaxy.ini file ?
The string template is "The directory \"%s\" does not exist.", if you fill that with a path you should get 'The directory "/some/path" does not exist.'
This is dynamically generated anyway, so I guess that shouldn't be the issue
Marius van den Beek
@mvdbeek
do you see directory 'home/galaxy/galaxy-dist/database/ftp/drosofff@gmail.com' is a symlink to a location not on the whitelist' in the logs ?
Marcel Kempenaar
@mkempenaar
Hi guys, question; I've deleted + purged a large number of users from our Galaxy server. which of the cleanup scripts do I need to run to delete all their data from disk? (spoiler: none worked)
Nicola Soranzo
@nsoranzo
@mkempenaar See galaxyproject/galaxy#6603
@mvdbeek I have a fix for the pipenv issue, PR coming soon
Marius van den Beek
@mvdbeek
awesome, thanks @nsoranzo !
Nicola Soranzo
@nsoranzo
I should say a workaround
Marcel Kempenaar
@mkempenaar
@nsoranzo concluding that there is no easy method of actual deleting the files that are still assigned to deleted + purged users?
Instead of un-deleting + impersonating?
side question: there is about 300GB in database/tmp, are they save to remove manually?
Marius van den Beek
@mvdbeek
if nothing is using files in database/tmp , yes
we need to also delete histories when deleting a user, then the cleanup scripts should work, I was going to look into this for this release cycle
Marcel Kempenaar
@mkempenaar
Thanks.
Is there a way of performing @AjitPS latest comment in the database? That would be a preferable solution for now.
Marius van den Beek
@mvdbeek
Sure, mark histories owned by the deleted user as deleted
you don't need to set them to userless, deleted is better
Marcel Kempenaar
@mkempenaar
Thanks, going to give that a try.
Marcel Kempenaar
@mkempenaar
What value would I use for user_id in the history table to set it to userless?
Marius van den Beek
@mvdbeek
why do you want to do that, does deleting not work ?
for real difficult post-mortem cases it would be good to know who produced a history, even if you can't restore the data
which you can't if you set it to userless
Nicola Soranzo
@nsoranzo
@mkempenaar You can use tmpreaper (or tmpwatch) to clean database/tmp
Marcel Kempenaar
@mkempenaar
For one user, all his histories are already flagged as deleted, however he still has 15GB in the disk_usage column.
Marius van den Beek
@mvdbeek
ignore this column, it is often not accurate and needs to be updated out of band, or by explicitly logging out the user
the important question is wether the datasets have been deleted
Christophe Antoniewski
@drosofff

@mvdbeek yes, I have

ftp_upload_dir_template = '${ftp_upload_dir}/${ftp_upload_dir_identifier}'

with single quote in the galaxy.ini. Under this setting, I don't see directory 'home/galaxy/galaxy-dist/database/ftp/drosofff@gmail.com' is a symlink to a location not on the whitelist' in the log.

On the other hand, if I remove the single quotes for the ftp_upload_dir_template variable, I can see now the error in the log:

galaxy.webapps.galaxy.api.remote_files WARNING 2018-08-22 14:48:23,956 [p:35985,w:2,m:0] [uWSGIWorker2Core1] Could not get ftp files: directory (/home/galaxy/galaxy-dist/database/ftp/drosofff@gmail.com) is a symlink to a location not on the whitelist
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 103, in index
    response = self.__load_all_filenames(user_ftp_dir, trans.app.config.user_library_import_symlink_whitelist)
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 118, in __load_all_filenames
    if self.__safe_directory(directory, whitelist=whitelist):
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 169, in __safe_directory
    raise exceptions.ConfigDoesNotAllowException('directory (%s) is a symlink to a location not on the whitelist' % directory)
ConfigDoesNotAllowException: directory (/home/galaxy/galaxy-dist/database/ftp/drosofff@gmail.com) is a symlink to a location not on the whitelist

but this does not show the uploaded file in the ftp panel anyway.

Marius van den Beek
@mvdbeek
good, that is the issue
let me think, we had an option for that
(it is there to prevent users symlinking to confidential data accessible by the galaxy user)
so the easiest would be to change the ftp_upload_dir to wherever the /home/galaxy/galaxy-dist/database/ftp symlink points to
I think we had this as a symlink to /var/storage/...
Christophe Antoniewski
@drosofff
AH yes ! I get it, it makes sense !
Marius van den Beek
@mvdbeek
to be complete, we have a user_library_import_symlink_whitelist setting for user imports, it seems we don't have the equivalent for the ftp dir, but you may be able to use that setting for ftp uploads as well (i didn't check that)
Christophe Antoniewski
@drosofff

Well it turns out not to be so easy to fix. With ftp_upload_dir to /var/storage/galaxy/ftp, we still have

galaxy.webapps.galaxy.api.remote_files WARNING 2018-08-22 16:54:25,955 [p:21145,w:3,m:0] [uWSGIWorker3Core0] Could not get ftp files: directory (/var/storage/galaxy/ftp/drosofff@gmail.com) is a symlink to a location not on the whitelist
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 103, in index
    response = self.__load_all_filenames(user_ftp_dir, trans.app.config.user_library_import_symlink_whitelist)
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 118, in __load_all_filenames
    if self.__safe_directory(directory, whitelist=whitelist):
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 169, in __safe_directory
    raise exceptions.ConfigDoesNotAllowException('directory (%s) is a symlink to a location not on the whitelist' % directory)
ConfigDoesNotAllowException: directory (/var/storage/galaxy/ftp/drosofff@gmail.com) is a symlink to a location not on the whitelist

although /var/storage/galaxy/ftp/drosofff@gmail.com is not a symlink.

In addition, I tried to fill a list of true locations in user_library_import_symlink_whitelist but it has no effect

Marius van den Beek
@mvdbeek
is it a hardlink ?
Christophe Antoniewski
@drosofff
I don't think so lrwxrwxrwx 1 galaxy galaxy 23 Aug 22 16:21 ftp -> /var/storage/galaxy/ftp/
Marius van den Beek
@mvdbeek
can you do python -c "import os; print(os.path.realpath('/var/storage/galaxy/ftp/drosofff@gmail.com'))" ?
ah, that's not how it works -- sorry
Christophe Antoniewski
@drosofff
but it looks interesting:
(.venv)galaxy@GED-Server:~/galaxy-dist/database$ python -c "import os; print(os.path.realpath('/var/storage/galaxy/ftp/drosofff@gmail.com'))"
/var/storage_new/galaxy/ftp/drosofff@gmail.com