docker buildmore aggressive about the layers it keeps when building. Please, let me know if it helps. 🙂
TL2021-historic(no doc, no src).
It's only on DockerHub, though. Pushing to GitLab failed.
FROM scratch, copy that directory into the image and only as a last step layer the operating system on top. The initial layer containing nothing but the historic TeX Live will be perfectly deterministic (by virtue of being frozen) so you get perfect layer reuse and OS updates on top can be done in a matter of seconds.