skelbins are not appropriate to be distributable as is, as a directories with bunch of files. That could be fragile due to network filesystem limitations. That is slow, because some skelbins already contains tens of thousands of files. And additional metadata has to be supplied with the skelbin. Your build steps are not aware about the exact $hsh values of the package and it would be insane to hardcode and repeatedly update after each BASS/skel’s change. And skelbin can depend on another skelbin to work (runtime dependency).
That is why, we have to use some kind of distribution format for solving the issues above. "skelpkg" is a packed skelbin with additional metadata. Similarly to Arch Linux and Gentoo, skelpkg is a single file, uncompressed POSIX pax archive with following entries:
Full name of the skelbin directory, $NAME-$hsh. With an optional checksum file.
Just a textual information how that skelbin/skelpkg was built. Currently just a current BASS’es commit revision.
Compressed POSIX pax archive containing the skelbin ($NAME-$hsh/ directory hierarchy).
POSIX ustar archive format can not hold more than 8GiB of data and (very)
long filenames. Forced pax usage guarantees compatibility with variety
of OSes. GNU tar’s format (also not having limitations above) easily
could be unreadable on non-GNU systems. BASS uses
build/contrib/detpax archiver for creating pax archives in
deterministic bit-to-bit reproducible way.
As pax/tar does not have any kind of index, as ZIP does, it is crucial to place the largest bin file at the very end of the archive. And that is why the outer archive is not compressed – to easily seek among its entries.
Metalink4 (RFC 5854) XML-based format is used to keep integrity checksums for files. It is well supported format by various tools and it is capable of storing multiple checksums simultaneously. That allows us to keep both Streebog hashes and much more faster ones.
Nothing prevents you from extending it with additional files, for example holding cryptographic signatures.
skelpkg’s name is whatever you want. As a rule it should be just skel’s
$NAME. But what if you do not care about exact skel’s version
and just want to install whatever perl (for example)? You can
always just create a (sym)link to it with a short name.
bin inner archive is compressed by default with
Zstandard. Being much faster
than venerable gzip, it achieves much better compression
ratio. But the main issues is its ultimate decompression speed, where
hardly your CPU will be the bottleneck. Reducing amount of data transfer
between disks/network and you system results in considerable decrease in
transfer/installation time. That is why so many package managers and
distributions already moved to its usage by default.
But you can override and use any kind of compressor in the skelpkg (with
$COMPRESSOR when using build/lib/mk-pkg). That is
required for example for zstd skelpkg itself, that can not be
decompressed without already having zstd installed.