Personally I've always considered it bad hygiene to commit generated outputs, but this article notes that this takes on a new significance in the light of supply chain security concerns. Good changes from PostgreSQL here.
Generated output, vendored source trees, etc. aren't, or can't be, meaningfully audited as part of a code review process, so they're basically merged without real audit or verification.
My personal preference is never to include generated output in a repository or tarball, including e.g. autoconf/automake scripts. This is directly contrary to the advice of the autotools documentation, which wants people to ship these unauditably gargantuan and obtuse generated scripts as part of tarballs... an approach which created an ideal space for things like the XZ backdoor.
My take is that they should always be committed, but never generated by the dev, instead generated and pushed when necessary by CI. The problem with generating those files yourself is that, in many cases, it makes the output nondeterministic and nonreproducible. In the ideal world those tools would just generate those files deterministically, but until then for me committing them from CI is an acceptable stopgap
My preference is to do both. Have them generated by a dev, committed, and also generated in CI. The latter gets compared with the checked in contents to ensure the results match the expected value.
This speeds up CI (the generation path can be done in parallel) and most local development.
The one catch is that it relies on mostly trusting whoever has a commit bit. But if you don’t have that and any part of the build involves scripts that are part of the repo itself, then you’ve already lost.
My preference is to do both. Have them generated by a dev, committed, and also generated in CI. The latter gets compared with the checked in contents to ensure the results match the expected value.
Bingo. This is what I am working towards convincing people to adopt at my current job. It's a long road.
The generation routine bits would be highly specific to the project, but the final check in CI is as simple checking the git diff/status of the generated targets to see if they match the ref. Any deviance indicates that it’s been missed by the patch submitter (likely inadvertently in the case of honest actors).
The real work is being able to transform the generation task into a reproducible step that be run consistently anywhere. Containerizing those steps can help but it’s not strictly required nor is it enough if the “inputs” are a non-seeded random or the current time.
That's not the case for autotools output, or flex and bison output.
If the generated files are what you say? Well, just embed the generation step into the build system. A simple approach like that is easily made reproducible, and we avoid introducing noise into the repository.
> an approach which created an ideal space for things like the XZ backdoor.
That's not entirely correct. Indeed there was a part of the xz backdoor that lived in the configure script. However, that part was also included in the sources of the configure script as found in the tarball (and not in the git archive).
Thus regenerating the configure script didn't help, but regenerating the tarball did.
They are not and never did commit generated files (as far as I can tell). Their release process used to generate some files and place that into a distribution file, but that file was never committed anywhere.
If you make a large but simple refactoring, like renaming a frequently-used function across a large repo, nobody is going to audit that diff and check for extra changes.
Things don't have to be this way, Google's source control systems apparently has tools that can do such refactorings for you in a centralized fashion, and one could make something like that for git.
Going to the extreme of this though, I really really hate getting an autoconf project with no generated configure file. I don’t want to install the full autotools suite to do build!
On the other hand, keeping tarballs close to the git tree makes it easy to reuse git archive and related GitHub features, provided the repo properly includes some kind of versioning information in tree.
Linux software sources are in a weird spot between users and developers.
I, as a developer, organize sources in a way that make it easy to work for another developer. My software will never be compiled by any user. All my users use build artifacts.
I might consider adding autogenerated code, but only when I'm like 99% sure that this code won't ever change. For example that's the case for integration with many organizations where WSDLs are agreed upon once and then never touched. Having Java sources regenerated every build just adds few seconds to every build time without noticeable advantages.
The fact that some Linux users prefer to build software from the sources and at the same time do not want to install necessary build tools is a bit strange situation.
May be containers should be better utilized for this workflow. Like developer supplies Dockerfile which builds a software and then copies it to some directory. You're running `docker build .` and they copying binary files from the container to the host.
Including autoconf outputs servers to avoid having to have autoconf installed. Because autoconf installs historically lagged behind what autoconf-using projects wanted, this used to be a problem. Nowadays it's not that big a deal.
As u/nrabulinski says, you can have the CI system generate and commit (with signed commits) autoconf artifacts.
Nix and Guix have their issues, but it is hard to read something like this and not wonder why you would migrate to them when facing issues like this
There is a learning curve for either Nix or Guix that puts many off. However its not that steep, certainly it is many orders of magnitude easier than maintaining PostgreSQL, and once you are over that you no longer need to do things like keeping a dedicated clean machine just to pack a tarball. Write the derivation and anyone, anywhere, on any machine can generate the exact same tarball with a one liner
The barrier caused by the initial steps of learning Nix/Guix is a shame because once you are over it, it is difficult to see why software is built any other way (the same may apply to bazel, but i have no experience with that).
Docker isn't reproducible. The one thing it can give you is a consistent set of mystery meat binaries, but that's an even worse starting point than the old problem of mystery meat source code.
They just aren't by default (because they include a timestamp) and you need to jump through multiple hoops to get them there, consistently. (And things like "apk add" or "apt install" can't be used unless you're installing pinned versions)
Reproducible docker images are almost useless for the things you want reproducible for. Sure you can reproduce the image for all of the future, but that image is useless in a few years when the certificates expire. Those expired certificates mean you cannot use the image for whatever you wanted it for.
A variation of the above is reproducible builds are not that useful - sure you can prove the build is the same, but in the end you want the latest security fixes applies and so by the time you create the replacement build and verify it the build is obsolete.
Don't get me wrong, reproducible builds are important and do good things - but there are severe limits to what you can/should do with them and so while it is important to demand them, they are not important to use yourself.
Are you pinning your base image? Where did that come from? Are you pinning your packages? What about their dependencies? Are you locking down the hashes or just hoping that your distro won't replace a package in-place?
And that's before you get into crap like OpenShift certification that blanket requires a `dnf update` statement.
Why would you not be pinning versions in a Dockerfile? The entire point is “if it works on my machine, it works on yours,” and that goes out the window if you can’t be assured that every program in the release is at the same version you had.
That "entire point" is already accomplished by the built binary container image, which has a unique identifier in the form of its SHA-256 hash, and can be shared with others easily.
A reproducible build is grand, but somewhat tangential to that goal, and hard to obtain in practice. Besides the timestamp problem already mentioned, you can't always pin the versions of system libraries and other distribution-provided software. The large long-term cost of hosting and geographically distributing content leads to many distributions, and especially their externally provided package mirrors, discarding stale versions from repositories. Often, the only available versions are the one included in the release plus the latest N, with N sometimes as small as 1.
If you're building a no-frills image for production deployment of a single piece of software, this problem can be bypassed thanks to distroless and other stripped-down base images, but "batteries included" images can't go this route.
Because if you pin versions you are pinning to some version with a security flaw that you are not allowing yourself to get. Often a flaw is fixed by a developer who realizes something is wrong with the code but it hasn't been exploited yet so anyone who keeps up to date cannot be exploited by that flaw, while anyone who doesn't keep up doesn't even know they are vulnerable.
Of course there is a balance here, there is a reason to pin versions. I'm stating why you shouldn't do that, but I cannot figure out all the pros and cons and how they should work out for your needs.
Nix is very unobtrusive on non-nixos installs, but I put together a flake that builds a CNPG compatible image, it has postgres, barman, pgmq, pl/python, pl/lua, pl/pgsql, pl/v8, pg_squeeze, pg_jsonschema, pg_graphql, pg_analytics, pg_safeupdate, pg_cron, pg_similarity, pgaudit, pgrouting, postgis and timescaledb. Weighs in heavy on about 700mb container, but it literally has everything-ish. And as long as postgres and barman is in $PATH, shadow files are configured and some folders are created CNPG just goes with the flow.
I can't imagine building such a monstrosity with anything else. And since the plugins are dependant on postgresql but not eachother I can add and remove them at a whim. Nix will create layers for me automatically.
And when I upgrade postgres I know I all packages will be built against the new postgres because Nix.
I think Nix could use list/dict comprehensions and some more devcandy sure, but it's really really great.
And at the end of the day, if you just go look at the source it's all there available to you, you don't have to wonder how Debian or RedHat built their golden postgres, there's no golden anything in Nix because if their hashes don't match mine I won't be pulling from their cache.
I think Nix biggest issue is that it doesn't attract promo skiddies the same way an imperative dirtbag like Salt or Ansible would, and most people can't even comprehend the things that open up when you can trust your shit.
Wanna write the hackiest perl script ever that'll never keep working? That's what activates most people's new NixOS generation still (there is a rewrite undergoing).
But back to point, Nix on Ubuntu patches /etc/{bash,fish,zsh}rc, creates the /nix top folder and that's it. It doesn't eat your system.
Yes, it has warts and they're big. But it's the only way forwards
I suspect you're linking to this without having read it recently. This section now explains that there used to be problems on macOS and that they're now resolved + some optional extra information about what the installer is doing. And to be fair, the mac installation issue was quite bad for a few years.
Does this make the downstream packagers’ jobs harder — they must now presumably have Perl, Bison, Flex and DocBook installed on the packaging machines?
The build server now needs the actual build dependencies instead of relying on pre-built intermediate build artifacts. This is a good thing, and should be expected from anything that claims to "build from source".
It's going to be a trivial amount of extra work. Packaging build systems (well, the mainstream ones: Debian, RPMs, ebuilds etc.) have the concept of runtime and build-time dependencies, so the maintainers will just need to add a few more packages to the build deps list.
Many distributions have a policy of not using generated files, and more are implementing it after the xz hack. Most package systems have (not just Debian) have the concept of build and deploy dependencies. It is of course more work, and package systems tend to not be well documented and so it is hard to figure out, but it is possible: if you ask the experts they will help
This doesn't seem like a problem though? As long as it's open source with packages available, that's just a CI process. At some point those machines are building those packages anyway, so the content is also already cached.
As I understood it, they used to generate the flex/bison stuff because they relied on very specific versions of those. But now those versions are commonly available, they rely on the user to generate these.
Historically in autoconf codebases (which PostgreSQL is) `make dist` is done: a) after `./configure` (so you have a Makefile, naturally), which is b) after `autoreconf -fi` (so you have a `./configure`). This allows the dist archive to contain the outputs of `autoreconf -fi` so that users need not have autoconf installed and they can just run `./configure`.
The configure generated from configure.ac has always been checked into Git for PostgreSQL. So with either the old or the new make dist approach, the configure in the tarball matches the one checked into the source code repository. So this is outside of what this article is discussing.
> Currently, the Git version used to produce the release tarballs (on the above-mentioned “clean” box) is too old to create reproducible .tar.gz tarballs, but it will create reproducible .tar.bz2 tarballs.
What is different about gzip and bzip2 that causes this?
Gzip just isn't specified well enough to be fully reproduceable.
Gzip is technically only a file format specification, with multiple compression algorithms allowed.
Also Gzip includes metadata about the compression like a timestamp for when the file was compressed, and bits for which OS was in use when the file was compressed, etc. so the out put is never 100% the same, though it ought to be easy to work around that part.
Git 2.38.0 is the version where git archive uses an internal gzip implementation instead of calling the actual external gzip. This internal implementation has two improvements for this purpose: First, it doesn't store the timestamp. You could also get that with gzip -n. (But the old git archive didn't do that, so you have to run the gzip as a separate step after git archive.) Second, it stores the platform identification bits as "UNIX" on all platforms, so the output is identical on all platforms. There is no gzip command-line option for that, unfortunately.
Ah right. My suggestion would have been to use a small Perl or Python script or something, assuming there would be a zip library that would let you set the proper flags.
Flex and Bison are simple enough tools (probably, you never know with GNU stuff) that you should be able to just vendor & compile them as part of the build process if you actually care about reproducible builds.
Generated output, vendored source trees, etc. aren't, or can't be, meaningfully audited as part of a code review process, so they're basically merged without real audit or verification.
My personal preference is never to include generated output in a repository or tarball, including e.g. autoconf/automake scripts. This is directly contrary to the advice of the autotools documentation, which wants people to ship these unauditably gargantuan and obtuse generated scripts as part of tarballs... an approach which created an ideal space for things like the XZ backdoor.