Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Helena Rasche
    @hexylena:matrix.org
    [m]

    The project will try to hire a DevOps or a system administrator person

    devops makes me think they'll be a k8s person, I hope systems-wg can have a say in that so we can get someone who could maybe accomplish the goals of our WG (like pulsar, remote data, etc.)

    Nate Coraor
    @natefoo:matrix.org
    [m]
    I don't think it means everything, it means if you're spending significant amount of effort developing/maintaining something that would be of wider usefulness to other Galaxies, and that could be incorporated upstream, let's do it.
    If it's only applicable to your deployment then no
    Helena Rasche
    @hexylena:matrix.org
    [m]
    gotcha
    so things like vortex where we're already been working on trying to standardise
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Yes, good example
    Helena Rasche
    @hexylena:matrix.org
    [m]
    yep, putting that in a spreadsheet somewhere will definitely help.
    TIL @jmchilton reads this. Welcome!
    Nate Coraor
    @natefoo:matrix.org
    [m]
    I hope we can agree that we don't need a single deployment strategy but we should unify efforts so that the amount of engineering required to support diverse deployments is minimized.
    Helena Rasche
    @hexylena:matrix.org
    [m]
    yeah, that may be the common ground that's possible.
    and it would be incredibly fantastic to unify that, there's lots of cool stuff being engineered and it'd be great for everyone to have access to that by default
    Anton Nekrutenko
    @nekrut
    /@all: is there a gitter channel for deployment group?
    Helena Rasche
    @hexylena:matrix.org
    [m]
    Anton Nekrutenko
    @nekrut
    Tx!
    Anton Nekrutenko
    @nekrut
    /@all: Here is the logic for merging admin and deployment. This was crafted by Björn and Enis:

    As you know, there are currently two main distinct approaches for deploying Galaxy: Ansible/Terraform and Kubernetes. An issue here is that much of the work is replicated in developing and maintaining these two approaches. We recognize the administration of our deployments is understaffed, regardless if it’s one of the usegalaxy.* servers or AnVIL for example. We are looking at hiring more help but the reality is that there will always be more work than available effort. To help this situation, we’d like to see more synchronization between the deployment approaches and consolidation on parts of the deployment stack.

    With that, we’d like to see the Admin and Deployment working groups merge into a new Systems working group. The aim here is to find synergies between the deployment approaches. All of you as members of these working groups have complementary experience and expertise that can be leveraged to create more robust solutions. We would like to encourage open-minded and strategic thinking on how Galaxy instances are deployed, starting by creating an inventory of the components that make up Galaxy installations. Then, deciding if a single deployment strategy for any individual component is achievable, realizing it, and deploying it across different environments. Even if the number of such components is minimal, engagement within one group will raise the level of understanding between the members for different deployments, strategies, and use cases.

    Furthermore, problems encountered during running instances should not be hacked around in deployments, but rather fixed upstream. For this it is essential the Systems working group engage closely with the backend working group. Every problem that is fixed upstream will make your deployments and the ones by the community easier. This is also where long-term strategic thinking comes into play where Galaxy architectural decisions can, and should, be made to ease our deployments.

    Let us know what you think or if you have alternative suggestions.

    Helena Rasche
    @hexylena:matrix.org
    [m]

    problems encountered during running instances should not be hacked around in deployments, but rather fixed upstream

    I really love this in theory, but as an ex-admin, I never had time to do that :(

    I filed lots of issues during my term with EU, but we ran SQL on cron to work around a number of issues that I simply wasn't knowledgeable enough to fix in the codebase. As far as solutions, is there any chance that some devs could be made available to the admin group? I think we're very heavy on the admin side and light on the people who can upstream solutions

    Anton Nekrutenko
    @nekrut
    is there any chance that some devs could be made available to the admin group?
    Nate Coraor
    @natefoo:matrix.org
    [m]
    I don't think it's necessarily saying that the admins should fix upstream, it's that the admins should (as you did) file issues to have the problem addressed upstream and then the devs should fix them. The disconnect in the past has been getting devs to do that, but I think the WG structure (the admin WG lead should communicate with the backend WG lead to transmit these needs) is supposed to address it. If admin needs are not being addressed by the other WGs then that's a breakdown in the functioning of the WG model. But PIs correct me if I have this wrong.
    Anton Nekrutenko
    @nekrut
    yes. @kxk302 will be joining the new systems group
    Helena Rasche
    @hexylena:matrix.org
    [m]
    🎉
    But for e.g. pulsar issues, I think we filed (not sure about communicated) these issues, but, until we tried to fix it badly ourselves, maybe it was hard for others to feel the urgency or justify the time from their side? which makes total sense, everyone is super busy, and no one was turning off .* just because pulsar wasn't working for our use case yet.
    Helena Rasche
    @hexylena:matrix.org
    [m]
    Nate points out to me that filing issues is insufficient. Ok we can raise things at relevant WG meetings
    Maybe I had the wrong idea of WGs from earlier discussions with Björn, I thought they were more or less silos in which projects happened from start to finish and interested people from each discipline were in a single wg to help things along
    Anton Nekrutenko
    @nekrut
    Not silos (they are silos now and that does not work so well). One reason for this is that we simply do not have enough people
    3 replies
    Helena Rasche
    @hexylena:matrix.org
    [m]
    sounds like an improvement!
    slugger70
    @slugger70:matrix.org
    [m]
    Us Aussies love the idea and we really want to participate and the progress we've made will allow that. But we're always 12 hours late to the party. Bloody timezones. The reason the last hackathon made so much progress was because we were all awake at the same time. Looking forward to all working together to be honest.
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Regular meetings are tough for us due to time differences but maybe we can figure out a rotating meeting schedule that involves some pain for everyone?
    1 reply
    Also I mentioned over on wg-deployment, I'm going to use wg-admin for the systems WG for the moment until we decide what to do with these channels (rename, create a new one, etc)
    Everyone should have received the "Systems groups: Initial goals" email from Anton, if you didn't, please let me know.

    I guess I could just paste it here as well...

    Dear Admin + Deployment people:

    We have two main deployment approaches: (1) Ansible and (2) Helm driven. The group responsible for each approach (the first is critical for usegalaxy.*; the second is for AnVIL). Both groups are understaffed in terms of system administration and ideally need two new sysadmins each. The reality is that we do not have resources to afford two additional sysadmins considering that we have serious deficiencies in UI and so on. On top of this the PIs are increasingly removed from fine technical details due to a number of factors related to day-to-day needs.

    So, here is what we are asking: can you first develop a sufficiently detailed "map" of what is involved in each type of deployment (usegalaxy.* versus AnVIL). Once such a map exists can you find common components or other ways in which two approaches can be engineered and maintained in parallel as much as technically possible.

    Finally, we need to reexamine our previous technical choices and see if they still serve our needs. For example, why do we complicate the tool installation on usegalaxy.org in such a way that we need to have CVMFS for tools, is that a shortcoming in our backend that makes large scale deployments harder than it should be? Why does Anvil design is an "Airbus A380" deployment for a
    single-user-use case? Is that over-architectured for that specific use-case?

    Have you decided on your meeting schedule yet?

    Thanks A + B + E

    Nate Coraor
    @natefoo:matrix.org
    [m]
    Yeah, unfortunately, I don't think much gets prioritized/done as it is.
    Keith Suderman
    @ksuderman
    I didn’t receive the email from Anton...

    But I would suggest that a better distinction between deployment methods would be (1) Bare-Metal, and (2) Kubernetes. Anvil is a bit of an odd duck in that it is a singe user Kubernetes deployment, but at heart it is just a Kubernetes deployment. As to which is the Airbus; for comparison, the Ansible playbooks to setup Galaxy contain 4K lines of YAML while the Galaxy Helm chart contains 2.4k lines of YAML.

    As for the question of what to do with the Gitter channels, I would vote for archiving the wg-deployment and wg-admin channels and starting a wg-systems channel just to make it clear for posterity what happened. Although renaming one or the other works for me as well.

    Keith Suderman
    @ksuderman
    I saw this on the Jetstream Slack channel. Given the discussion on egress costs at todays meeting it seemed apropos:
    https://blog.cloudflare.com/introducing-r2-object-storage/
    Nuwan Goonasekera
    @nuwang
    When should we plan for the merged meeting? It would be great to get some feedback on the work done so far on the dtd enhancement work, and have a general discussion about it.
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Helena Rasche
    @hexylena:matrix.org
    [m]
    I can't make it but vortex gets my vote.
    Nuwan Goonasekera
    @nuwang
    Thanks @hexylena:matrix.org. I’ll send along the powerpoint in case you have some feedback.
    Helena Rasche
    @hexylena:matrix.org
    [m]
    please do!
    Nuwan Goonasekera
    @nuwang
    On a separate note, is there a preferred UID for galaxy? The galaxy playbook is fairly agnostic on the matter, but it would be a bit easier in containerized environments if we had a standard one?
    Helena Rasche
    @hexylena:matrix.org
    [m]
    I don't think there's an answer to that, everyone uses a different number depending on environments. you're welcome to pick something container standard I guess?
    Nuwan Goonasekera
    @nuwang
    Ok, that makes sense.
    Nuwan Goonasekera
    @nuwan_ag:matrix.org
    [m]
    Looks like this Wednesday, Thursday or Friday could work at 12pm GMT
    Keith Suderman
    @ksuderman
    Thursday at noon is another roundtable, so maybe Wednesday?
    Keith Suderman
    @ksuderman
    I’ve creaated a “Systems” folder in the “Working Groups” folder in the shared Google Drive https://drive.google.com/drive/folders/1jj9M2wpH-j5fRtEXOXUPF2R55W4AAHDT
    I’ve started on a “Charter” and agenda for the first meeting, but they are both basically just empty docs, so feel free to add material!
    Nate Coraor
    @natefoo:matrix.org
    [m]
    Wednesday at 12 PM UTC seems like the winner.
    Nate Coraor
    @natefoo:matrix.org
    [m]
    I sent an email to everyone who I believe is a member of the Systems WG, if you didn't get it or you did get it and know of anyone I missed, please let me know.
    Nate Coraor
    @natefoo:matrix.org
    [m]
    The Systems WG call begins in 10 minutes