Debian Linux and Ubuntu Linux systems, like most modern computer operating systems, require regular patching to stay secure -- and the pace of updates required seems to keep increasing.

One way to stay on top of this is to use the package unattended-upgrades to automatically install the security updates every day. There are two steps required to enable it:

sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -p low unattended-upgrades      # Choose Yes

The latter step sets:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

in /etc/apt/apt.conf.d/20auto-upgrades (they default to "0", which leaves the package effectively inactive unless you run unattended-upgrades via some other mechanism); if you use a configuration automation system you can also just write a file with that contents.

That apt configuration file, and other related config files, are parsed by /etc/cron.daily/apt, and will run unattended-upgrades as required. Which will download and install the upgrades needed, from the repositories that are permitted for automatic upgrade (configured in /etc/apt/apt.conf.d/50unattended-upgrades, along with several other unattended-upgrades related settings).

If one enables unattended-upgrades on an Unbutu Linux system and leaves it alone for long enough, one eventually discovers a hidden flaw: over time your /boot partition fills up, and (hopefully!) your monitoring starts telling you it is almost full. This happens because (a) Ubuntu Linux release (ETA 2015-06-22: almost -- see below) all kernel updates as new packages (cf Debian which at least sometimes reuses the same package names), (b) there are kernel upgrades at least every month, if not more often, and (c) there is nothing removing the older kernel packages. So the kernel packages build up over time.

If one checks the Ubuntu Linux Automatic Security Updates page, one finds that there seems to be a helpful option to automatically remove unneeded packages while running unattended-upgrades:

// Do automatic removal of new unused dependencies after the upgrade
// (equivalent to apt-get autoremove)
//Unattended-Upgrade::Remove-Unused-Dependencies "false";

(also in /etc/apt/apt.conf.d/50unattended-upgrades; defaulting to "false").

Which seems like the ideal solution to clean things up. Sadly it does not actually work as described, or really at all, due to Ubuntu bug #1267059 -- a combination of programming errors mean that especially on Ubuntu 14.04 it does not actually do anything :-(

The ideal behaviour would be:

  1. Keep the currently running kernel, to avoid breaking the system

  2. Keep the latest kernel, so we can reboot onto it later to upgrade

  3. If the currently running kernel and the latest kernel are the same, keep one other older kernel (probably the next oldest would do, but optimally would be the previously running kernel, if we could track that).

As Mostly Unixish points out, in modern Unbuntu Linux, you can get nearly this behaviour (first two) just by using:

sudo apt-get --purge autoremove

since it already knows not to remove the currently running kernel. However if the current and latest kernels are the same that is the only one you will be left with, which may be acceptable. And if you manually install a kernel, making it a selected package not just an unneeded dependency, it will be retained.

From some experimentation:

  • Do NOT use on Ubuntu Linux 10.04 LTS: it appears there is no special case for the currently running kernel, so that command might remove either the current kernel or the headers of the current kernel, if it is not also the latest.

  • Unclear if it works reliably on Ubuntu Linux 12.04 LTS: the systems I have left on Ubuntu 12.04 LTS do not seem to have enough kernels installed to test reliably. (Possibly it does not do anything there.) (ETA, 2015-03-02: Comments on markmcb.com's post suggest that Ubuntu 13.04, and Debian Unstable as of 2013-07-19 or earlier, has this fix; which means it may not be safe on Unbutu Linux 12.04 LTS either.)

  • On Ubuntu 14.04 LTS appears to work as advertised: both the current and the latest kernel images are protected from deletion, even if they are different, including the kernel headers.

You can test with:

sudo apt-get --purge autoremove --dry-run | grep $(uname -r)

if you get any output, then on an Ubuntu system that means it is going to remove your currently running kernel, or the headers for it, so beware!

Several people have attempted to build a manual system to do similar things, including:

  • Mostly Unixish, as mentioned above, which has a Python script to determine the packages to remove. It is the most promising solution, but unfortunately does not always protect the latest kernel version (unless it is also the current kernel version), if there are kernels with multiple different major/minor versions (eg, 2.6.32, 3.2, 3.13, etc) -- which appears to be due to the choice of lexiographical sorting combined with 3.2 versus 3.13. (It also does not work on Python 2.6, due to the use of check_output, which was added in Python 2.7; and Ubuntu 10.04 LTS has Python 2.6.)

  • chr4.org has a well explained shell script version, but the use of sort -nr means that it too does not handle multiple kernel versions with different major/minor versions (especially 3.2 versus 3.13 kernels).

  • ubuntugenius and markmcb.com have a similar, frankly terrifying, regex solution to omit the current kernel -- but does not even try to protect the latest kernel, just assuming the current kernel is the latest kernel (ie, that you have already rebooted). So it is not suitable for unattended use.

But none of them was completely suitable for my purposes, with a mix of Ubuntu 10.04 LTS, Ubuntu 12.04 LTS and Ubuntu 14.04 LTS servers, several of which had been upgraded from 10.04 LTS to 12.04 LTS to 14.04 LTS (and thus accumulated various kernel image cruft along the way). In particular I needed to properly handle the kernel version sorting between 2.6.32/3.2/3.13, and to work on Ubuntu 10.04 LTS.

So I have extensively edited the Python script from Mostly Unixish, to produce obsolete-kernel-pkgs, which attempts to enforce the above three rules, listing only packages to be removed which are not the current kernel, latest kernel, or would leave fewer than two kernel versions installed. It appears to run on Ubuntu 10.04 LTS, Ubuntu 12.04 LTS and Ubuntu 14.04 LTS.

Usage would be:

PURGE_KERNELS=$(obsolete-kernel-pkgs)
if [ -n "${PURGE_KERNELS}" ]; then
   sudo apt-get purge --dry-run -y ${PURGE_KERNELS}
fi

to see what it would do (--dry-run). When you're happy it will do the right thing, remove the --dry-run, and run for real. (I would strongly suggest testing it with --dry-run a few times, and manually from the command line a few times, before automating it.)

ubuntu-cleanup-kernels is a short (Bourne) shell script that does the above, assuming obsolete-kernel-pkgs is installed in a location on the PATH it sets -- I put mine in /usr/local/sbin. By default it will run with --dry-run to show what to remove; if you pass it an argument (any argument) it will omit the --dry-run parameter and run for real, eg ubuntu-cleanup-kernels go.

As written, that will produce quite a bit of output (eg, to be emailed via cron), when it is running. Which is probably desirable until you are sure that it is working robustly. Running it, eg, once a week seems reasonable, since you only need to be removing the kernel packages quicker than they accumulate!

You may also with to enable the unattended-upgrades flag to remove unused dependencies in case it gets fixed in future versions, and starts working properly. One way to do that would be:

UNATTENDED_CONF=/etc/apt/apt.conf.d/50unattended-upgrades
if grep -q "Remove-Unused-Dependencies.*true" "${UNATTENDED_CONF}"; then
   : # Already enabled
else
   # In-place the file with an enabled version
   sudo perl -i.bak -ne 'if (/Remove-Unused-Dependencies/) { print <<EOF;
Unattended-Upgrade::Remove-Unused-Dependencies "true";
EOF
   } else { print; }' "${UNATTENDED_CONF}"
fi

ETA, 2015-03-02: Another script, in Python, which I only found after writing mine; it appears to try to only remove kernels older than the running version, which may not be sufficient if you do not reboot often.

ETA, 2015-04-13: On later Ubuntu versions (eg, Ubuntu 12.04 LTS and Ubuntu 14.04 LTS, but not Ubuntu 10.04 LTS), there is /etc/kernel/postinst.d/apt-auto-removal which maintains flags for whether or not specific kernel packages are auto-removable or not (which is what allows them to be removed with apt-get --purge autoremove). From the comments it appears to save (1) the running kernel, (2) the kernel just installed, (3) the latest kernel, and (4) the second latest kernel -- so might save up to four of them. The status file is written to /etc/apt/apt.conf.d/01autoremove-kernels, ie in the apt configuration area, and there is some configuration of what is considered a kernel package in /etc/apt/apt.conf.d/01autoremove.

ETA 2015-06-22: It turns out that occassionally, Ubuntu will re-use the same package name, eg 3.13.0-55.92 and 3.13.0-55.94, but it seems to happen only to fix regressions in an earlier fix (see Ubuntu bug #1465998 for the regression fix in that case).