Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Alexandru Mahmoud
    @almahmoud
    Do you have a ReadWriteMany storage class? also I'm not sure if we've tested the chart with 1.20, there might be some api versions out of date or such things... Let me know how it goes though, i'd be happy to help start pushing to make it compatible with 1.20 if things don't work
    Gulsum Gudukbay Akbulut
    @gulsumgudukbay

    @almahmoud I checked the PVC using kubectl describe pvc and I got "FailedBinding 2m55s (x17605 over 3d1h) persistentvolume-controller no persistent volumes available for this claim and no storage class is set" (I am not sure if this is the correct way to check it).
    Also I tried the tutorial which is in the link you sent, however I am getting "Error: rendered manifests contain a resource that already exists. Unable to continue with install: StorageClass "cvmfs-gxy-data" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "my-galaxy-release": current value is "cvmfs"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "default": current value is "cvmfs""
    Error after I try to execute "helm install my-galaxy-release cloudve/galaxy --set cvmfs.enabled=true --set cvmfs.deploy=true"

    I tried to delete every pod, every namespace and every helm deployment to do a clean start, however, I still get this error.

    Alexandru Mahmoud
    @almahmoud
    Regarding the first error, you need a storage class to dynamically provision volumes for you, or you need to provision volumes yourself for persistence (this would be for persistent volumes for data, not the node disks that hold the ephemeral storage). If you're planning on running all of the Galaxy pods on one node, any volume should be enough. If you want to run across multiple nodes, you will need some sort of shared filesystem. We have used NFS and CephFS in the past.
    Regarding the second issue, it seems you have resources from a previous helm release that were not deleted. This is likely due to an uninstall that has not properly ended in the past. From that error, it seems you still have leftover resources from a CVMFS installation under cvmfs namespace. This is only an issue because some resources, like the storage class, are not namespaced. So while the new release installed all the namespaced resources in default namespace for the new release, it conflicted with existing resources when trying to also create the cluster-wide resources such as the storage class. If it's not too hard, it might be worth just starting from a new cluster, adding the storage, and then trying again, rather than trying to clean up the existing one
    Gulsum Gudukbay Akbulut
    @gulsumgudukbay
    I want to run Galaxy across multiple nodes, so I will need a shared filesystem.
    I will try to setup the k8s cluster again from the beginning. Thanks for the advice. I will let you know if it works fine or not. Thanks!
    Alexandru Mahmoud
    @almahmoud
    Hope it helps, let me know if you encounter other issues
    Po Foster
    @pofoster_gitlab
    I am running galaxy from AWS EC2 using cloudlaunch (https://launch.usegalaxy.org/appliances) and the GVL appliance. it seems that I need a .pem file to ssh into the ec2 after galaxy was setup. There is a keypair in my AWS console for cloudlaunch, but I don't have the .pem file (no way to download it) and password authentication seems to be disabled. IS THERE A WAY TO SSH INTO THE MACHINE?
    Nuwan Goonasekera
    @nuwang
    The .pem file is available for download for an hour after launch through the cloudlaunch interface. If you didn’t download it, and you need to ssh in, the easiest thing is to relaunch a fresh instance. It’s often easier to just create a keypair yourself in AWS, and select that keypair at launch time.
    If you really need access to this instance itself, then it’s a bit more of an involved process with something like: https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2reset.html or https://sennovate.com/how-to-reset-the-forgotten-root-password-in-aws-ec2-instance/
    Po Foster
    @pofoster_gitlab
    @nuwang thanks. that works. I created a keypair on AWS, then used the advanced deployment option on cloudlaunch to select the keypair.
    EnasBasset
    @EnasBasset

    Hi, we are installing Galaxy on Kubernetes cluster, we followed the instructions here: https://training.galaxyproject.org/training-material/topics/admin/tutorials/k8s-deploying-galaxy/tutorial.html

    pods get stuck in state init0/1

    $ kubectl get pods 
    NAME                                    READY   STATUS      RESTARTS   AGE
    galaxy-init-mounts-n1oce-dfnbn          0/4     Completed   0          39m
    galaxy-galaxy-postgres-c9949849-hv7gk   1/1     Running     0          34m
    galaxy-web-9567fdfc-2gckm               0/1     Init:0/1    0          34m
    galaxy-nginx-7cbb5495f5-chqf5           1/1     Running     0          34m
    galaxy-workflow-777945c9f-g9f49         0/1     Init:0/1    0          34m
    galaxy-job-0-6cf859b489-6xbrc           0/1     Init:0/1    0          34m
    galaxy-galaxy-galaxy-postgres-0         1/1     Running     0          34m

    logs for pod "galaxy-galaxy-postgres-c9949849-hv7gk":

    time="2021-08-29T11:28:22Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: pq: password authentication failed for user \"postgres\"" cluster-name=default/galaxy-galaxy-galaxy-postgres pkg=cluster
    time="2021-08-29T11:28:22Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: password authentication failed for user \"postgres\"" cluster-name=default/galaxy-galaxy-galaxy-postgres pkg=controller worker=0
    Nuwan Goonasekera
    @nuwang
    @EnasBasset Did you set postgresql.galaxyDatabasePassword when installing? If you didn’t, this can occur on restart (there’s a note in the readme about this). Also, what is your helm version?
    EnasBasset
    @EnasBasset
    @nuwang Thank you for your answer, after I set postgresql.galaxyDatabasePassword it worked fine
    EnasBasset
    @EnasBasset
    Hello, I 'm installing galaxy on kubernetes, some pods are failing with this error Error: failed to prepare subPath for volumeMount "extra-files-probe-script" of container "galaxy-job-0" and the logs from pod galaxy-job ConfigurationError("Unable to create missing directory: {}\n{}".format(path, unicodify(e))) galaxy.exceptions.ConfigurationError: Unable to create missing directory: /galaxy/server/database/tmp [Errno 13] Permission denied: '/galaxy/server/database/tmp'
    I've installed galaxy before on default namespace and didn't face these errors, the errors above are appearing when installing new galaxy instance on new namespace called galaxy-new
    Nuwan Goonasekera
    @nuwang
    @EnasBasset This seems like a permissions issue? What's the storageclass for the galaxy-pvc?
    It looks like the Galaxy container doesn’t have permissions to create the tmp folder on the pvc, and that’s probably tied to what sort of storage you are using. Is this an existing NFS? Can user “101”, which is the default fsGroup for the chart, access and create folders there?
    The namespace is unlikely to be an issue since the chart is often used in non-default namespaces
    EnasBasset
    @EnasBasset

    @nuwang the storageclass of galaxy-pvc is local-path, I changed the permissions on the path so user "101" can write on it. permission denied error is fixed now but I had new error

    OSError: Error reading file '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml': failed to load external entity "/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml"

    Nuwan Goonasekera
    @nuwang
    @EnasBasset Try installing the chart with cvmfs.enabled=False. Also, what version of k8s are you using?
    EnasBasset
    @EnasBasset
    @nuwang Thank you, it's working now.
    I'm running multiple instances of Galaxy on my cluster, Is cvmfs necessary for production or not?
    Nuwan Goonasekera
    @nuwang
    If you need reference data and all the tools to be preinstalled, then having cvmfs enabled is useful. The cvmfs-csi driver for kubernetes is not particularly well maintained, and is incompatible with k8s versions above v1.19. We are working on migrating away from it towards S3 as an alternative.
    Pablo Moreno
    @pcm32
    Hi guys, is the deployment of multiple galaxy instances within a single k8s cluster (maybe on different namespaces) reliable currently? I’m sizing some openstack tenancies (where we would run k8s) for a project and I would like to know if I can use the same k8s cluster reliably for a dev and a prod galaxy instances. Thanks!
    Nuwan Goonasekera
    @nuwang
    Afaik, nothing has been introduced that breaks this. However, I haven’t tried this in the recent past. Let me check with Keith, I think he tried this recently
    Keith Suderman
    @ksuderman
    I haven’t tried two Galaxy instances in the same cluster, but it is discussed in the readme: https://github.com/galaxyproject/galaxy-helm#example-installation-for-multiple-galaxy-instances-on-the-same-cluster
    Pablo Moreno
    @pcm32
    Thanks guys!
    fortneyjm
    @fortneyjm
    Hi all. I am new to the community.. Does galaxy support the idea of moving compute capability to the data? Ex. We have a main galaxy Server A. We have a “remote” DRM Server B (We will say it is Pulsar for simplicities sake). We have a very large file on Server B. We can not put this file on a shared file system (I think this rules out all DRM options except Pulsar?). Can a Galaxy Tool running on Server B access the large file as part of a workflow without loading it into Server A first and allowing Server B to copy it back over to Server B?
    bgruening
    @bgruening:matrix.org
    [m]
    This is one of the more complicated setups, but yes this could be done now, with filesystems like iRODS etc ... bit the remote files work that is currently in PR state this will be easier from the summer onwards we assume.
    fortneyjm
    @fortneyjm
    Thank you. We are looking at iRODS now. Do you have a link to the PR you are referring too?
    Pablo Moreno
    @pcm32
    Hi guys, is the Galaxy helm chart expected to work in Kubernetes 1.18 (which is the version I get in our local openstacks through magnum). I know that the readme says 1.18 as minimum, but maybe that is outdated? On what versions of k8s are you currently deploying? Thanks
    Keith Suderman
    @ksuderman
    We are currently deploying on 1.19. Are you seeing errors with 1.18?
    Gulsum Gudukbay
    @gulsumgudukbayakbulut:matrix.org
    [m]
    Hi everyone. Is there any resource to find out usage statistics for some popular tools in Galaxy? For example how much different category of different tools are used?
    Also is there any resource that shows other metrics about the popular tools (such as cloud (container) statistics, OS, app, HW statistics etc)?
    bgruening
    @bgruening:matrix.org
    [m]
    @gulsumgudukbay: I can give it to you for one galaxy sever
    1 reply
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Hey all - @almahmoud set up a cluster for me on JS2 for running GxITs, and it's working but I'm not able to scale - with autoscaling enabled, I am not seeing nodes starting up for waiting pods, and when I try to manually add a node, I get:
    FAILURE
    Create appliance task failed: CloudBridgeBaseException: Image None is not active. (HTTP 400) (Request-ID: req-59807f7d-22a9-4e46-8bb3-ea391de0e95b) from exception type: <class 'novaclient.exceptions.BadRequest'>
    Nuwan Goonasekera
    @nuwan_ag:matrix.org
    [m]
    It sure looks like the relevant VM Image may have been deleted in JS2? How was this cluster launched? Through cloudlaunch?
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Yeah, Alex said in another channel that was the issue - JS2 rotates images pretty frequently and I guess the ID is what is stored.
    I am not sure how he launched it though, isn't CloudLaunch decommissioned?
    Nuwan Goonasekera
    @nuwan_ag:matrix.org
    [m]
    CloudLaunch can still be run locally, plus there's a dev environment etc. but I was just trying to figure out when/how it was launched... I guess the issue is now resolved?
    Nate Coraor
    @natefoo:matrix.org
    [m]
    He's at a conference until next week and I'm trying to get a better understanding of it myself. He did fix it, but it'll break again the next time JS2 admins push a new image.
    Nuwan Goonasekera
    @nuwan_ag:matrix.org
    [m]
    You can update the image in the cloudman django admin console. Usually available at: https://<host>/cloudman/cloudlaunch/admin
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Ah, thanks! I had not thought to look there.
    Pablo Moreno
    @pcm32
    Anyone using the Galaxy Helm chart on k8s on OpenStack? What are you using to provision shared file system? Thanks!
    Keith Suderman
    @ksuderman
    @pcm32 it has been some time since I launched the Galaxy Helm chart on OpenStack, but I used to frequently and used NFS. In particular ganesha-nfs (https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner_
    1 reply
    Pablo Moreno
    @pcm32
    Hi! how is the k8s setup controlling the tools that get installed? I see some of our single cell tools at some of the versions, but not all of them (and not all the versions). For other instances we have some setup with bioblend/ephemeris which install tools for us, but I suspect that this is currently reading tool definitions from some s3 and I’m guessing not all tools are updated there? Is there somewhere I can see which tool are already there and check for the available versions (to see if it is a lack of files in the repo or galaxy not seeing them locally)? Thanks!
    Keith Suderman
    @ksuderman
    They are downloaded by an init job in jobs-init.yaml https://github.com/galaxyproject/galaxy-helm/blob/a08bbae28a3dbd991489fdfa9cf1b839cc9357a7/galaxy/templates/jobs-init.yaml#L194
    I will have to check, but those archives should be update regularly.
    If needed, you could define your own extraInitContainer to update/install extra tools that are missing: https://github.com/galaxyproject/galaxy-helm/blob/a08bbae28a3dbd991489fdfa9cf1b839cc9357a7/galaxy/values.yaml#L213
    Pablo Moreno
    @pcm32
    Thanks! Cool, so if I pack my XMLs like those tar.gz and put them somewhere public I could do it like that. Isn’t it problematic that more than one .tar.gz with tools and confs that will reuse the same partial paths exists or this is somehow gracefully handled (ie. if I have a subsequent tar.gz somewhere, this would also put files within the toolshed.g2.bx.psu.edu/repos directory).
    Also, is there somewhere some logic to generate this tar.gz? I guess it is a matter of installing them on some running instance and then extracting the shed_tool_conf and tool XMLs from there?
    What I see from that .tar.gz is that there seems to be only a couple of versions for some our tools (ebi-gxa), maybe triggered by them being used on some specific workflow?
    Pablo Moreno
    @pcm32
    Are you able to trigger those jobs independently for a currently running instance (to update the toolset), or you are expected to restart the setup so that those init jobs run?