At the moment, we have a couple of runners and mounting a mirror of CTAN is indeed one of the options on the table.
However, it's less reproducible than doing it inside docker, so that's that. Also does not scale as well to more runners.
If anyone is interested: https://hub.docker.com/r/texlive/texlive/tags, the
TL<YEAR>-historic-tree* tags are those that contain only the TeX Live trees (for Linux x86-64). We have not publicly announced the names yet but probably they will stay as is.
We discussed it with Ben Frank two weeks ago: https://gitlab.com/islandoftex/images/texlive/-/issues/5#note_959539914
Apparently, we have everything in place. With !22 merged, it's just a matter of carving out a couple of hours to make an MR.
1 reply
It still seems to me that incremental updates are a more pressing issue from the standpoint of a user.
Yes, but incremental updates combined with incremental split into schemes would be very bad layering resulting in explosion of pulled image size.
I guess incremental updates are quite a strong argument for the independent scheme images. Probably users will not switch that often between schemes so it probably makes sense.
It would seem that way. We cannot be incremental both across time and across schemes.
And it seems to me that most users will be pulling a single scheme repeatedly.
So each scheme would be its own set of layers corresponding to incremental updates in time, independent on other schemes.
What we could do is this: we could build the root image for each year from the smaller schemes.
3 replies
But they would be separate from then on.
This would likely solve most of the duplication issues, since the diff over a year will be a drop in a bucked t compared to what we already have.
Not sure what we would name them, probably tree-something, but there would be several, one for each scheme. They could share layers.
3 replies
On top of them would be layers with weekly updates.
islandoftex/images/texlive#11 is what I had in mind after all. Simply build the images per scheme which would be one
matrix entry more in the CI file. And as I said, the update strategy as per the other issue would be orthogonal to that split.
And the tree-* images, which we would build at the beginning of TL's annual cycle could be build on top of each other (such as tree-full using tree-minimal).
The latest-* images would not care and could be just a matrix entry as you say.
This way, we get rid of most of the duplication, get incremental updates in time and we get Debian's security updates.
Definitely, let's break these changes into small bits. It's the only way any of this gets done. đď¸
1 reply
Not a problem. Can make a tikz version too if we'd like to have it in the TUG 2022 paper.
Oh, you should ask Paulo Cereda about that one, he is our chief of papers and presentations. But maybe he'd like it in the presentation as well đ
1 reply
What I don't entirely understand is this, because I never cared much about schemes: is there a nice division of packages into schemes without much duplication? Because is seems to me that schemes overlap and for example Debian packagers have to completely reorder things to get rid of duplication.
1 reply
Ah.
Would it make sense to provide images for collections rather than schemes?
No. Collections are not really usable. Nobody needs a collection that has only packages but does not provide any engine. Schemes are more self-contained.
Understood, but is there a way to assemble schemes out of smaller shared units?
Oh, definitely, but packages seem too fine-grained.
Of course, it would be best if each TL package would be its own âlayerâ. That would have the best composition. But then we would be at Nix level (package manager), not at Docker level (distribution layer).
It's not even clear that we would not get a massive slowdown from overlayfs with this many layers.
2 replies
It's imaginable that we would cut at boundaries of both schemes and collections to get small enough pieces that we can use to compose any scheme. Not clear how to implement this practically, though.
1 reply
This can be automated, but still seems as a major hassle to implement and maintain.
1 reply
Does this mean that we have no clue on how to deduplicate the scheme images? Because I am ok with that, makes the implementation easier for the moment.
1 reply
I did, but then I also posited that we can deduplicate the first image for a TL and then keep the incremental updates separate for every scheme.
And promised to draw that.
It's just that I wasn't clear on how many of our schemes can be nicely based on another smaller scheme.
Will have to read up on that.
Oh, the patch that Henri proposed shows the proper order and a sensible selection of schemes.
Should have read that first.
I will shut up until I have the drawing. đ
On another note: Perhaps we can merge the IoT Docker room with this one sometime in the future, so that we are all on the same page.
1 reply
Actually, we should have far more of these conversation in a structured way in issues so that they are accessible in the project. But on the other hand, quick questions and responses are still better in messengers like Matrix.