Debian Bullseye: i386 to amd64 crossgrade

Introduction

The (original) Linux i386 architecture is pretty close to being removed, especially from Debian Linux (but also from the Linux kernel). The last few years have brought several announcements of "this is the last release with i386", only for there to be one more "last release" with i386 as a supported architecture. And various third party software has dropped support for i386, or is in the process of doing so.

Most relevant to me, SaltStack announced a switch to "onedir" packaging about a year ago, from Salt 3005 and especially Salt 3006. The "onedir" packaging means that Salt becomes a binary distribution, bundling its own Python interpreter -- which is obviously architecture specific. This means that my work around of using the "amd64" Salt packages (for older Salt releases) with an i386 Python (and some helper back ported Python modules) will no longer work. So that prompted me to finally convert the last few i386 VMs that were managed by Salt across to amd64 VMs.

The i386 cross grade approach I followed was fairly similar to my earlier test Debian i386 to amd64 crossgrade, but required some tweaks to get it working for Debian Bullseye. Especially, as the Debian CrossGrading Wiki page points out, Debian Bullseye requires perl-base cross graded even earlier in the process, before the rest of the perl packages. And there are some twisty perl / Debian packaging tool dependencies to resolve early on.

This post documents the approach I used on three Debian Bullseye i386 systems to cross grade them to Debian Bullseye amd64. (At the time of this writing Debian Bookworm is the stable Debian Linux; but I decided to do the cross grade on these system before upgrading them to Debian Bookworm, as the Debian Bullseye cross grade challenges were better documented.)

If you are considering attempting this cross grade approach yourself:

know that you will end up with a broken system in the middle of the cross grade, and it will take care and experience to move forward into a working system again. Key packages you rely on might be auto-removed, or require data or configuration to be recreated to continue working.
make backups before you start. More than one. Be very sure you can restore from backups. If you are cross grading a VM, consider if you can minimise other writes to disk during the cross grade (eg, firewall the VM off from incoming network connections) and snapshot the VM disk image before you start, so you have an undo option of rolling back to the snapshot and pretending you did not try the risky crossgrade. But either way make sure you have a backup handy you can look at to see "how it was before" to guide you in putting the system back together.
especially if it is a VM, consider making a clone of the system in a virtual machine, and doing a few dry runs of the upgrade process in a VM you are happy to throw away. That should let you encounter most (but maybe not all) of the problems in a safer environment where you can roll back and try again.
at least twice during the cross grade you will have to tell the Debian packaging tools that you know what you are asking is a terrible idea, but you want to do it anyway (Yes, do as I say! literally entered). Neither Debian nor I take any responsibility for your broken system as a result of attempting to cross grade it from i386 to amd64 following this or any other guide. You need to assure yourself that you have the experience to carry out this complicated risky procedure through to a safe conclusion.
be aware that some packages (especially databases) have architecture dependent file formats on disk, which will break horribly if you change architecture. Make sure you have a plan to recover a working system from that cross grade (eg, in the case of databases make sure you know how to dump and reload the database before/after the crossgrade).

My first few trial attempts (in a VM clone) of the cross grade with Debian Bullseye took 1-2 hours to figure out all the required steps. By the time I had reduced each VM clone upgrade to a runsheet of steps specific to that VM install, the production crossgrade took about 30 minutes. But it took at least half a dozen attempts to get a "do these things, in this order" run sheet, that I was happy to attempt in production. Per VM.

If you have some other way to reinstall the system (especially automatically) on a new VM that is 64-bit from the start, that would be a safer option. This risky crossgrade procedure is probably only relevant to long standing pets which are difficult to replace. (All of mine were originally installed 15+ years ago, and have been through many Debian hand upgrades of Debian Linux releases since the install; two of them had been through a physical to virtual conversion too.)

Crossgrade preparation

Make sure your system is capable of running 64-bit code (especially if it is a VM, make sure the VM "hardware" is ready for 64-bit code). The lm flag ("Long Mode") needs to be visible in the CPU flags, eg:

ewen@debian:~$ grep ' lm ' /proc/cpuinfo
flags       : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
ewen@debian:~$

Or check with:

lscpu | grep 64-bit

if you have lscpu installed.

Make sure you have access to the system console (eg, via a serial console or a virtual machine console, or a remote KVM switch -- or maybe by being local to the system). At one point in the upgrade you need to select a specific kernel version, from the grub boot menu, which requires console access.

Finish any package installs, and upgrades on the Debian Linux i386 system, and make sure any earlier upgrades have been tidied up. Eg, dpkg --audit should be clean and dpkg -l | grep -v 'ii' shouldn't show anything left. Any obsolete packages that you cannot reinstall again should be removed. Make sure the system boots cleanly on the Debian Linux i386 install.

Make a backup. That you are confident you can restore from and get a working system again.

Record the package status of the i386 system, for cross reference later on (eg, confirming the same packages got reinstalled, and the same package are marked as "automatic dependencies"):

apt-mark showauto >i386-auto-packages
dpkg --get-selections >i386-package-selections

Switch to dual architectures, and switch to a 64-bit Linux Kernel

Enable the amd64 architecture as a foreign architecture:

dpkg --print-architecture
sudo dpkg --add-architecture amd64
dpkg --print-foreign-architectures
sudo apt-get update

The base architecture at this point should be i386, but you should be able to install amd64 architecture packages by explicitly specifying them at this point.

Then install the 64-bit amd64 kernel, and boot into that:

sudo apt-get install linux-image-amd64:amd64
sudo update-grub
sudo shutdown -r now

For this reboot (and only this reboot) you will have to interrupt the grub automatic boot, and pick a different kernel from the menu. Specifically you need to go into the Advanced options for Debian GNU/Linux menu option (second menu) and pick the amd64 kernel entry which is named something like Debian GNU/Linux, with Linux 5.10.0-26-amd64 and will be the third entry in that submenu (because the existing i386 kernel sorts before the amd64 kernel in the generated list).

Make sure the system booted into a 64-bit kernel:

uname -m

should return x86_64 if you are running a 64-bit kernel. If not, try rebooting again and choosing the correct kernel in the grub boot menu.

Once you have the 64-bit kernel booting, remove the equivalent i386 (32-bit) kernel, and reboot again to make sure the 64-bit kernel is booted automatically from now on:

sudo apt-get purge linux-image-686-pae linux-image-5.10.0-26-686-pae
sudo update-grub
sudo shutdown -r now

If you wish you can pause at this point indefinitely -- the Linux i386 user land should run on the Linux amd64 kernel without any substantial problems, as this was used for a long period in the early x86_64 kernel development.

Change the primary architecture to amd64

This is one of the more complicated steps. At this point we are changing dpkg / apt to the amd64, which changes the primary architecture from i386 to amd64.

On Debian Bullseye, perl-base:i386 conflicts (indirectly, I think) with perl-base:amd64. So we have to force installing perl-base:amd64 at this point. The conflict is sufficiently bad that we cannot even use apt to download packages once dpkg / apt have been changed to amd64. So we need to pre-download several amd64 packages first, then install them "all at once". And then let apt fix up the conflicts any way it can, after we have installed a few more perl packages. (We cannot install those additional packages at the same time as perl-base:amd64 as the conflicts are too complicated to resolve, so apt gives up.)

This combination of steps worked for me on my Debian Bullseye systems (although apt-get -f install removed some packages important to me on each system, which had to be put back later on):

sudo -s
apt-get clean
apt-get --download-only install dpkg:amd64 tar:amd64 apt:amd64 apt-utils:amd64
(cd /var/cache/apt/archives/ && apt download perl-base:amd64)
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --print-architecture              # Should show: amd64
dpkg --print-foreign-architectures     # Should show: i386

Note the two runs of dpkg --install ... for the same packages; this is required because some packages will not install on the first run due to missing amd64 dependencies; but they can install the second time around. (There might be a topological sort of those dependencies to install them in "the right order". But for these critical packages, there are lots of circular dependencies, so simply running the install of these key packages twice is the most pragmatic approach.)

Once perl-base:amd64 is installed, a bunch of other Perl dependencies used for Debian package management will be broken, so some additional perl packages need to be upgraded at this step, including libtext-csv-xs-perl and libencode-perl:

(cd /var/cache/apt/archives/ && apt download libperl5.32:amd64 libgdbm-compat4:amd64 libgdbm6:amd64)
(cd /var/cache/apt/archives/ && apt download perl:amd64)
(cd /var/cache/apt/archives/ && apt download libencode-perl:amd64)
(cd /var/cache/apt/archives/ && apt download libtext-csv-xs-perl:amd64)
(cd /var/cache/apt/archives/ && apt download libdebconfclient0:amd64)
dpkg --install /var/cache/apt/archives/*_amd64.deb

At this point it should be possible to get apt back to a point where it is "happy" automatically, but do keep track of which packages it is removing as some of them will be important to you and need to be reinstalled later on. Those packages need to be removed at this point to break dependency loops:

apt-get -f install
dpkg --configure -a

Switch the shell to `amd64`

Install the amd64 bash and dash, and run them instead of the 32-bit versions. This will pull in several other key dependencies, and also force-remove the i386 versions, so you will have to agree that you know this can break your system and you want to do it anyway (Yes, do as I say!):

apt-get install dash bash
exec /bin/bash

Install the remaining `amd64` replacements for installed `i386` packages

Download the packages which are needed to install amd64 replacements:

apt-get --download-only -y --no-install-recommends install \
   `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 | uniq |
    grep -v 'linux-image.*686-pae' |
    sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'`

Then install as many of the amd64 libraries as possible (since these can mostly be installed in parallel with the i386 packages, and fix dependency issues):

dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb
apt-get -f install
dpkg --configure -a

(we also install the outstanding perl packages at this point for certainty).

Then we can install the majority of the remaining amd64 packages:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit | cut -f 1 -d : |
    xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' | uniq |
    tee /tmp/packages-to-install

dpkg --install `cat /tmp/packages-to-install`

which might take a second pass to be able to install all the packages due to dependency issues:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit | cut -f 1 -d : |
    grep -v 'linux-image.*686-pae' |
    xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' | uniq |
    tee /tmp/packages-to-install

test -s /tmp/packages-to-install && dpkg --install `cat /tmp/packages-to-install`

Tidy up the `amd64` system

Check what else is left to be converted from i386 to amd64. Some packages will still have i386 versions installed, but will also have an amd64 version -- packages like gcc-9-base and gcc-10-base are effectively library packages and permit parallel installs of both i386 and amd64 versions.

dpkg --get-selections | grep :i386 | grep -v lib | grep -v deinstall | wc -l
dpkg --get-selections | grep :i386 | grep -v lib | grep -v deinstall
dpkg --get-selections | grep :i386 | grep -v lib | awk '$2 ~ /^install$/ { print $1; }' | grep -v 'linux-image.*686-pae' | sed 's/:i386/:amd64/;' | grep -vf /tmp/64-bit

There may still be conflicts at this point, in which case you will need to figure out some way to resolve them for your combination of packages; it might involve removing a package important to you, cross grading other packages, and then reinstalling the packages you wanted on the system once the dependencies have been resolved properly. Inevitably if apt cannot figure out a solution, it will be a complicated problem, so assess carefully what you can live without temporarily to be able to make forward progress.

Once that is done, run a normal upgrade of the amd64 system to make sure apt is happy, and reboot the system to make sure all the running programs are the amd64 versions:

apt-get update
apt-get upgrade
apt-get dist-upgrade
sudo update-grub
sudo shutdown -r now

Assuming the system came back up as a 64-bit system:

uname -m               # Should show: x86_64
getconf LONG_BIT       # Should show: 64

then clean up most of the i386 libraries next:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get --purge autoremove

Then I found if gnuplot was installed, I had to temporarily remove it to make progress on the remaining cleanup (and install it again later):

HAVE_GNUPLOT=$(dpkg -l gnuplot >/dev/null 2>&1 && echo "true")
if [ -n "${HAVE_GNUPLOT}" ]; then sudo apt-get remove gnuplot; fi

before I could remove the essential i386 libraries, which hopefully by this point your system does not need (just the amd64 versions of those libraries). But apt / dpkg does not know this so you will have to confirm you really want to remove these essential libraries (Yes, do as I say!):

dpkg --get-selections | grep :i386 | grep -v deinstall | wc -l
sudo apt-get purge $(dpkg --get-selections | grep :i386 | grep -v deinstall | awk '{print $1;}')

Then you should be able to reinstall any packages that were removed to break dependency loops:

dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/'
sudo apt-get install --no-install-recommends $(dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/')
sudo apt-get -f install
sudo dpkg --configure -a

dpkg --get-selections | grep deinstall
sudo apt-get install $(dpkg --get-selections | awk '/deinstall/ { print $1; }')

Including reinstalling gnuplot again if you had to remove it above:

if [ -n "${HAVE_GNUPLOT}" ]; then sudo apt-get install gnuplot; fi

Then confirm there is nothing else that needs to be reinstalled:

dpkg --get-selections | grep deinstall
dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/'

and no i386 packages still installed:

dpkg --get-selections | grep i386
dpkg -l | grep "^i.*:386"

Cleaning up from the crossgrade

Do a regular package update to make sure there are no complaints, and then reboot to make sure you have a working system:

sudo apt-get update
sudo apt-get dist-upgrade
sudo dpkg --configure -a
sudo update-grub
sudo shutdown -r now

Assuming the system booted okay, purge the i386 packages that were removed, and remove the i386 architecture:

sudo apt-get --purge autoremove

sudo dpkg --remove-architecture i386
dpkg --print-architecture               # Should show: amd64
dpkg --print-foreign-architectures      # Should return nothing

Then make sure that dpkg / apt are happy:

dpkg --audit
dpkg -l | grep -v "^ii"

and if the package state is tidy reboot again as a plain amd64 system:

sudo update-grub
sudo shutdown -r now

Ensure your production system is production ready again

Check for any other packages that might have been removed as dependencies and need to be installed, by comparing the i386 package list to the amd64 package list:

dpkg --get-selections >/tmp/amd64-package-selections
diff -bw <(sed 's/:i386/:amd64/;' i386-package-selections | sort) <(sort </tmp/amd64-package-selections) | grep "<" | grep -v "686-pae" | awk '{print $2;}' | grep -v "^lib" | grep -v pkgmonkey-client | tee /tmp/missing-packages
sudo apt-get install $(cat /tmp/missing-packages)

dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep "^>"
sudo apt-get install $(diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep -v "pkgmonkey-client" | awk '/^>/ {print $2;}')

dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep -v "pkgmonkey-client" | grep "^>"

Then check for any extra packages that got dragged in as dependencies during the upgrade which you did not have installed before and do not want, and consider if you want to remove them:

dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep "^<" | grep -v binutils | grep -v 'lib.san' | grep -v 'libunwind8'

Beware that some of these additional packages will be new amd64 dependencies. Eg, strace on amd64 requires libunwind8, and gcc on amd64 indirectly requires liblsan0 and libtsan0. It will take some systems administration experience, and some investigation, to determine what is still needed for production and what was temporarily installed to meet dependencies during the cross grade.

Once you are happy that the production packages are installed, and no unnecessary packages are installed, restore the "automatically installed" marks so that future upgrades will work better:

sed 's/:i386/:amd64/' i386-auto-packages | grep -v 686-pae | xargs sudo apt-mark auto

(which uses the list of i386 automatically installed packages, created at the beginning).

Then remove any packages which are now no longer required:

sudo apt-get -f install
sudo apt-get --purge autoremove

and confirm you do not have any 386 or 686 packages left:

dpkg -l | grep '[36]86'

Architecture upgrades for specific tools

Some packages have their own architecture dependent files on disk. Particularly databases, but also other programs. You will need to figure out, before committing to the production upgrade, how to handle these situations.

On my mail servers, I found that (a) postfix got removed during the crossgrade (replaced by exim) and had to be reinstalled, and (b) the spamassassin tool had some C libraries that it compiles to do regular expressions faster than in perl, which have to to be rebuilt, with sa-compile.

In the case of spamassassin I did this with:

test -x /usr/bin/sa-compile && (
    sudo sa-compile --list >/tmp/before
    sudo sa-compile
    sudo sa-compile --list >/tmp/after
    diff /tmp/before /tmp/after
    sudo service spamassassin restart
    sudo service spamass-milter restart
    sudo service postfix restart
)

Then once you have a final configuration, reboot the system one more time to make sure you are running your final package configuration:

sudo update-grub
sudo shutdown -r now

And watch your logs and monitoring very carefully for the next few hours to keep an eye out for other problems that need manual fixes.

From this point on, if you successfully got to a working system and fixed any data architecture dependencies, then it should act like any other Debian Linux amd64 install, including for any future upgrades.