Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Marius
    @mvdbeek:matrix.org
    [m]
    :point_up: Edit: I did have to remove all occurences of SafeStringWrapper from annotations.json (probably the custom dynamic name is at fault?), but otherwise this looks to be spot on and should make adding type annotations a lot faster
    Marius
    @mvdbeek:matrix.org
    [m]
    with galaxyproject/galaxy#14675 we can use pyannotate out of the box
    Helena
    @hexylena:matrix.org
    [m]

    John Davis: I'll admit I'm struggling a bit with this new migration system, and I'm feeling a bit stuck with the docs. My coworker upgraded us to 22.05 and we're not downgrading back to 22.01 but it's unclear how to downgrade the database to the expected version.

    1. sh manage_db.sh --help just returns an error, not help
    2. Eventually I did sh manage_db.sh --version which showed me some internal command / other commands I could call

    I can't really figure out what other versions are available or how to work with them

    I've found run_alembic.sh which at least shows me some very opaque branch identifiers

    sh run_alembic.sh current
    Activating virtualenv at .venv
    INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
    INFO:alembic.runtime.migration:Will assume transactional DDL.
    d4a650f47a3c (head)
    186d4835587b (head)
    INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
    INFO:alembic.runtime.migration:Will assume transactional DDL.
    d4a650f47a3c (head)
    186d4835587b (head)

    ahh one of them is for galaxy, and one for tsi which I assume is toolshed but the i makes me think tool install db?

    $ sh run_alembic.sh heads
    Activating virtualenv at .venv
    186d4835587b (gxy) (head)
    d4a650f47a3c (tsi) (head)
    is there a specific hash that we should know as the "downgrade to 22.01" hash?
    Marius
    @mvdbeek:matrix.org
    [m]
    There's a a doc in lib/galaxy/model/migrations/README.md that should explain most of this.
    Helena
    @hexylena:matrix.org
    [m]
    yeah that was the one I was looking at
    but it only mentions versions like abc123 and xyz789, and wasn't sure where those came from.
    Or gxy@ which also was unclear (and produced "no revisions")
    Marius
    @mvdbeek:matrix.org
    [m]
    So now you're looking for a version to downgrade to ?
    Helena
    @hexylena:matrix.org
    [m]
    We were, yes. In the end we've restored a database backup given time constraints
    but it would be really useful to know how downgrading back to 22.01 should work
    Marius
    @mvdbeek:matrix.org
    [m]
    22.01 is still sqlalchemy-migrate, so I guess you'd go to the first revision
    Helena
    @hexylena:matrix.org
    [m]
    ahh how do we find out a list of revisions?
    Marius
    @mvdbeek:matrix.org
    [m]
    manage_db.sh history
    Helena
    @hexylena:matrix.org
    [m]
    $ sh manage_db.sh history
    Activating virtualenv at .venv
    Traceback (most recent call last):
      File "/home/hxr/arbeit/galaxy/galaxy/./scripts/manage_db_adapter.py", line 38, in <module>
        run()
      File "/home/hxr/arbeit/galaxy/galaxy/./scripts/manage_db_adapter.py", line 33, in run
        ls.run()
      File "/home/hxr/arbeit/galaxy/galaxy/lib/galaxy/model/migrations/scripts.py", line 130, in run
        self.convert_args()
      File "/home/hxr/arbeit/galaxy/galaxy/lib/galaxy/model/migrations/scripts.py", line 143, in convert_args
        self.convert_version_argument()
      File "/home/hxr/arbeit/galaxy/galaxy/lib/galaxy/model/migrations/scripts.py", line 189, in convert_version_argument
        raise LegacyScriptsException("If no `--version` argument supplied, `upgrade` argument is requried")
    galaxy.model.migrations.scripts.LegacyScriptsException: If no `--version` argument supplied, `upgrade` argument is requried
    $ sh run_alembic.sh history
    Activating virtualenv at .venv
    6a67bf27e6a6 -> 186d4835587b (gxy) (head), drop job_state_history.update_time column
    b182f655505f -> 6a67bf27e6a6 (gxy), deferred data tables
    e7b6dcb09efd -> b182f655505f (gxy), add workflow.source_metadata column
    <base> -> e7b6dcb09efd (gxy), create gxy branch
    <base> -> d4a650f47a3c (tsi) (head), create tsi branch
    so from there I should understand that <base> is it?
    Do I downgrade to <base> or to e7b6...
    Marius
    @mvdbeek:matrix.org
    [m]
    No, I think you need to go to e7b6dcb09efd
    Helena
    @hexylena:matrix.org
    [m]
    Ok, good to know
    Cheers!
    Nicola Soranzo
    @nsoranzo:matrix.org
    [m]
    Please let me know if you're happy with the direction I'm taking there!
    John Davis
    @ic4f:matrix.org
    [m]
    Helena: Sorry about that! There's a new version of manage_db.sh with extensive documentation under lib/galaxy/model/migrations/README.md - I've just merged it. Marius - thanks again for the extensive review!
    Helena
    @hexylena:matrix.org
    [m]
    Thanks John Davis !
    I'll have a look at it
    John Davis
    @ic4f:matrix.org
    [m]
    Now you can do sh manage_db.sh --help - and expect a list of basic commands, and, overall, normal CLI behavior
    There's another one - in /scripts/ - it's db_dev.sh - that one has a more extensive list of commands, including --history and such. Again, you can sh db_dev.sh --help, or check README for a more detailed overview with examples.
    Marius
    @mvdbeek:matrix.org
    [m]
    We still don't have a way to easily go back to previous releases ... not that we used to have it, so not really a regression. Maybe we can somehow bookmark the release hash for the database, so admins could do manage_db.sh downgrade release_22.05 ?
    1 reply
    John Davis
    @ic4f:matrix.org
    [m]
    Then there's run_alembic.sh (in scripts/) - that one is just a wrapper around alembic's own CLI runner - so it provides access to everything
    and yes - as Marius just said - the next step is automated movement across the 22.01-22.05 boundary.
    Marius
    @mvdbeek:matrix.org
    [m]
    I don't know if that's necessary
    John Davis
    @ic4f:matrix.org
    [m]
    about bookmarking - we might be able to do better -
    so, the thing is that sqlalchemy migrate treated versions as a linear sequence and nothing else - hence, ascending integers as version numbers hardcoded into the tool. Alembic treats version relationships as a DAG: branches, merges - hence, hashes instead of integers.
    However, in our use case, it's still a linear sequence of versions. So, what we can do, is simply assign version identifiers instead of using the default hash.
    Helena
    @hexylena:matrix.org
    [m]
    nice
    John Davis
    @ic4f:matrix.org
    [m]
    So we could go back to ints and have alembic start at 182 (I think). Or at some other meaningful number. However, I haven't thought this through, and there might be complications or edge cases. But, in general, that might be an option (if and only if we really need it)
    By the way, you do NOT need to type in the full hash - see the new docs - a prefix uniquely identifying the revision is enough
    Marius
    @mvdbeek:matrix.org
    [m]
    Yes, that was certainly easier, but if we can have named revisions that's even better
    cause it's still not obvious that 211 is the revision you need for 23.05 ;)
    John Davis
    @ic4f:matrix.org
    [m]
    Marius - it's the container-based solution we discussed. You suggested it, I think, with regard to the move from 22.01 to 22.05, not the reverse. But I think if we have a tool that takes you one direction, it could take you back.
    we can have any revision identifier we like
    Marius
    @mvdbeek:matrix.org
    [m]
    I don't think the container solution was meant for doing anything automatically, it's just a worst case thing we may need if people want to upgrade their multi-year lagging Galaxy database
    The important thing being that you can go to the last sqlalachemy-migrate revision
    John Davis
    @ic4f:matrix.org
    [m]

    I don't think the container solution was meant for doing anything automatically, it's just a worst case thing we may need if people want to upgrade their multi-year lagging Galaxy database

    automatically meaning by manually launching a script, not automatically at startup

    Marius
    @mvdbeek:matrix.org
    [m]
    I understand, but in that scenario it doens't make sense to go back to sqlalchemy-migrate from a alembic revision
    so, if that's a piece of cake, sure, go for it, but if it adds more than 2 or 3 lines of code I don't think it's worth it
    John Davis
    @ic4f:matrix.org
    [m]
    OK. I think of it as a "nice to have" feature, but non-essential. So yes, agreed.
    Helena: the updated scripts and the README, since you may be using it some of that functionality, please do let me know if anything is off/unclear/wrong/stupid/etc. - either here galaxyproject/galaxy#14537 or wherever - I'll fix/redo right away. The migrations are complex, it bothers me to no end and I very much want to make it better/simpler, etc....
    John Chilton
    @jmchilton:matrix.org
    [m]
    I would like move the schema from galaxy-data into a new galaxy-schema package that only depends on galaxy-util and pydantic so that I can use the pydantic models in testing and such. Perhaps a more correct thing to do is generate clients from the schema - but I've never been happy with the models I've gotten out of our API in any language I've tried.
    Any objections?