One common piece of advice you will hear from cybersecurity experts is that you should always keep your software up-to-date, so that vulnerabilities can be corrected. We like to believe that consumers are the biggest offenders in terms of keeping old software around. After all, Grandpa doesn’t always know how to upgrade his iPhone.
This isn’t so easy for professionals. Let’s take a web site. Your average web site is composed of numerous servers, each running all manner of code, from the operating system to standard applications like Apache to web support code such as Python, PHP, and back end services, like MongoDB. ALL of it has to be updated.
What does the inventory look like? This has been the work of a group of people who have created something known as software bills of materials or SBOMs. One could think of an SBOM as the ingredient list on the back of a can of soup, only there’s software inside.
So how does one generate the inventory? And this is where things can get a bit interesting. On any reasonable UNIX machine there exists a package manager. In the case of Ubuntu, that would be apt, which is built on top of Debian’s dpkg. One can simply type “apt list –installed” to get the installed packages on a system, right?
WRONG.
Of course that will get you software that apt has installed, but if your site runs with python, then you might need to get a software list by using Python’s package manager, pip. In fact, “pip freeze” provides this information, and that will get you most, but likely not all, python packages. Now repeat for node.js’s npm, and others (assuming you can find all of them).
Now you might want to update those packages. That was the problem statement, after all. This should be simple enough, right? Well, let’s start with apt. It generally is simple enough, at least to start with: one simply runs, “apt upgrade -y” and software package upgrades happen. Of course, you have to test all of your code after this, as apt won’t do that for you. Your CI/CD process is hopefully doing this for you.
Next we go to Python. pip’s upgrade command requires a list of packages. Conveniently, we have one that we froze from above. However, it turns out that not all python packages are managed by pip; and it knows this, and pip will happily blow chunks when you try to upgrade a package it does own that requires an upgrade for something it doesn’t own. In the Ubuntu world, there are a few good examples of this, Cairo and MongoDB to name a few. Instead, these packages are managed by apt. Well that’s all good, since we can use the apt upgrade process, right?
WRONG.
Often times these packages are installed have or meet dependencies of their won that might go unmet as part of an upgrade.
Alpine Linux has applied an interesting solution to all of this: provide common python packages in their own ‘apk’ package manager. By using apk, you assuredly will not get the latest version of a package, but you will get consistent upgrade behavior. But this presents its own set of problems:
What you are looking at is a publication hierarchy. Now we have software in multiple distributions, deriving from a repository, probably from different branches. It’s possible that there are even multiple repositories for the same code. If we are lucky, the repo owner is doing the integration to push updates to Ubuntu, Pypi, and Alpine. But we are almost never going to be that lucky, if for no other reason, a lack of authorization for all of them. And so we’ll end up with a mix of push and pull. It’s the pull that causes the version skew, and this is just a single distribution.
Now, I’ve picked on Python in this post, but pip and pypi actually do a pretty reasonable job of managing what packages it installed, so long as there aren’t dependencies to stuff it didn’t install.
So what does this mean in terms of managing your product or package?
I’m not ready to make any sweeping recommendations about what to do. One option is to just deal, and upgrade when you can. The problem is that your velocity will be slower than if you pull all source from their repos directly. On the other hand, doing the latter is a whole lot of work, requiring a whole lot of expertise when builds blow up.
This is a good example of where OSS needs to mature a bit more.
Chicken Noodle Soup by By Willis Lam – Campbell’s Chicken Noodle Soup, CC BY-SA 2.0.