It's only on DockerHub, though. Pushing to GitLab failed.
That is unfortunate, but also much appreciated. I will create a TL2021 release of witiko/markdown right away.
Hi friends, I just want to say thank you to Ben and Vít for taking the time to look at the Docker images! ❤️
:point_up: Edit: That is unfortunate, but also much appreciated. I will create a TL2021 release of witiko/markdown right away.
https://github.com/Witiko/markdown/actions/runs/2289199925
https://github.com/Witiko/markdown/actions/runs/2289199925
Hi guys, do you have someone to host gitlab runner already or not? If you don't, I could provide a VM on my desktop (VM params: 4 cores, 16GB ram, 60gb nvme) right now to build what is needed. I know it is a bad solution (since it would be up only 18 hours per day), but this is the quickest help I can offer...
1 reply
I've recently read on the TeX Live list that you are having problems with being time-limited by GitLab CI for building the images. Have you considered inverting the build for the historic images? Since the entire TeX Live tree is frozen for them it would make sense to “install” TeX Live (potentially in portable mode) outside of Docker into a directory, then start
FROM scratch, copy that directory into the image and only as a last step layer the operating system on top. The initial layer containing nothing but the historic TeX Live will be perfectly deterministic (by virtue of being frozen) so you get perfect layer reuse and OS updates on top can be done in a matter of seconds.
1 reply
At the moment, we have a couple of runners and mounting a mirror of CTAN is indeed one of the options on the table.
However, it's less reproducible than doing it inside docker, so that's that. Also does not scale as well to more runners.
If anyone is interested: https://hub.docker.com/r/texlive/texlive/tags, the
TL<YEAR>-historic-tree* tags are those that contain only the TeX Live trees (for Linux x86-64). We have not publicly announced the names yet but probably they will stay as is.
We discussed it with Ben Frank two weeks ago: https://gitlab.com/islandoftex/images/texlive/-/issues/5#note_959539914
Apparently, we have everything in place. With !22 merged, it's just a matter of carving out a couple of hours to make an MR.
1 reply
It still seems to me that incremental updates are a more pressing issue from the standpoint of a user.
Yes, but incremental updates combined with incremental split into schemes would be very bad layering resulting in explosion of pulled image size.
I guess incremental updates are quite a strong argument for the independent scheme images. Probably users will not switch that often between schemes so it probably makes sense.
It would seem that way. We cannot be incremental both across time and across schemes.
And it seems to me that most users will be pulling a single scheme repeatedly.
So each scheme would be its own set of layers corresponding to incremental updates in time, independent on other schemes.
What we could do is this: we could build the root image for each year from the smaller schemes.
3 replies
But they would be separate from then on.
This would likely solve most of the duplication issues, since the diff over a year will be a drop in a bucked t compared to what we already have.
Not sure what we would name them, probably tree-something, but there would be several, one for each scheme. They could share layers.
3 replies
On top of them would be layers with weekly updates.
islandoftex/images/texlive#11 is what I had in mind after all. Simply build the images per scheme which would be one
matrix entry more in the CI file. And as I said, the update strategy as per the other issue would be orthogonal to that split.
And the tree-* images, which we would build at the beginning of TL's annual cycle could be build on top of each other (such as tree-full using tree-minimal).
The latest-* images would not care and could be just a matrix entry as you say.
This way, we get rid of most of the duplication, get incremental updates in time and we get Debian's security updates.
Definitely, let's break these changes into small bits. It's the only way any of this gets done. 👍️
1 reply
Not a problem. Can make a tikz version too if we'd like to have it in the TUG 2022 paper.
Oh, you should ask Paulo Cereda about that one, he is our chief of papers and presentations. But maybe he'd like it in the presentation as well 😄
1 reply
What I don't entirely understand is this, because I never cared much about schemes: is there a nice division of packages into schemes without much duplication? Because is seems to me that schemes overlap and for example Debian packagers have to completely reorder things to get rid of duplication.
1 reply
Ah.
Would it make sense to provide images for collections rather than schemes?
No. Collections are not really usable. Nobody needs a collection that has only packages but does not provide any engine. Schemes are more self-contained.
Understood, but is there a way to assemble schemes out of smaller shared units?
Oh, definitely, but packages seem too fine-grained.
Of course, it would be best if each TL package would be its own “layer”. That would have the best composition. But then we would be at Nix level (package manager), not at Docker level (distribution layer).
It's not even clear that we would not get a massive slowdown from overlayfs with this many layers.
2 replies
It's imaginable that we would cut at boundaries of both schemes and collections to get small enough pieces that we can use to compose any scheme. Not clear how to implement this practically, though.
1 reply
This can be automated, but still seems as a major hassle to implement and maintain.
1 reply
Does this mean that we have no clue on how to deduplicate the scheme images? Because I am ok with that, makes the implementation easier for the moment.
1 reply
I did, but then I also posited that we can deduplicate the first image for a TL and then keep the incremental updates separate for every scheme.
And promised to draw that.
It's just that I wasn't clear on how many of our schemes can be nicely based on another smaller scheme.
Will have to read up on that.
Oh, the patch that Henri proposed shows the proper order and a sensible selection of schemes.