Introduction
The (original) Linux i386 architecture is pretty close to being removed, especially from Debian Linux (but also from the Linux kernel). The last few years have brought several announcements of "this is the last release with i386", only for there to be one more "last release" with i386 as a supported architecture. And various third party software has dropped support for i386, or is in the process of doing so.
Most relevant to me, SaltStack announced a switch to "onedir" packaging about a year ago, from Salt 3005 and especially Salt 3006. The "onedir" packaging means that Salt becomes a binary distribution, bundling its own Python interpreter -- which is obviously architecture specific. This means that my work around of using the "amd64" Salt packages (for older Salt releases) with an i386 Python (and some helper back ported Python modules) will no longer work. So that prompted me to finally convert the last few i386 VMs that were managed by Salt across to amd64 VMs.
The i386 cross grade approach I followed was fairly similar to my
earlier test Debian i386 to amd64
crossgrade, but
required some tweaks to get it working for Debian Buster. Especially,
as the Debian CrossGrading
Wiki page points out, Debian Buster requires perl-base
cross
graded even earlier in the process, before the rest of the perl
packages. And there are some twisty perl / Debian packaging tool
dependencies to resolve early on.
This post documents the approach I used on three Debian Buster i386 systems to cross grade them to Debian Buster amd64. (At the time of this writing Debian Bookworm is the stable Debian Linux; but I decided to do the cross grade on these system before upgrading them to Debian Bookworm, as the Debian Buster cross grade challenges were better documented.)
If you are considering attempting this cross grade approach yourself:
know that you will end up with a broken system in the middle of the cross grade, and it will take care and experience to move forward into a working system again. Key packages you rely on might be auto-removed, or require data or configuration to be recreated to continue working.
make backups before you start. More than one. Be very sure you can restore from backups. If you are cross grading a VM, consider if you can minimise other writes to disk during the cross grade (eg, firewall the VM off from incoming network connections) and snapshot the VM disk image before you start, so you have an undo option of rolling back to the snapshot and pretending you did not try the risky crossgrade. But either way make sure you have a backup handy you can look at to see "how it was before" to guide you in putting the system back together.
especially if it is a VM, consider making a clone of the system in a virtual machine, and doing a few dry runs of the upgrade process in a VM you are happy to throw away. That should let you encounter most (but maybe not all) of the problems in a safer environment where you can roll back and try again.
at least twice during the cross grade you will have to tell the Debian packaging tools that you know what you are asking is a terrible idea, but you want to do it anyway (
Yes, do as I say!
literally entered). Neither Debian nor I take any responsibility for your broken system as a result of attempting to cross grade it from i386 to amd64 following this or any other guide. You need to assure yourself that you have the experience to carry out this complicated risky procedure through to a safe conclusion.be aware that some packages (especially databases) have architecture dependent file formats on disk, which will break horribly if you change architecture. Make sure you have a plan to recover a working system from that cross grade (eg, in the case of databases make sure you know how to dump and reload the database before/after the crossgrade).
My first few trial attempts (in a VM clone) of the cross grade with Debian Buster took 1-2 hours to figure out all the required steps. By the time I had reduced each VM clone upgrade to a runsheet of steps specific to that VM install, the production crossgrade took about 30 minutes. But it took at least half a dozen attempts to get a "do these things, in this order" run sheet, that I was happy to attempt in production. Per VM.
If you have some other way to reinstall the system (especially automatically) on a new VM that is 64-bit from the start, that would be a safer option. This risky crossgrade procedure is probably only relevant to long standing pets which are difficult to replace. (All of mine were originally installed 15+ years ago, and have been through many Debian hand upgrades of Debian Linux releases since the install; two of them had been through a physical to virtual conversion too.)
Crossgrade preparation
Make sure your system is capable of running 64-bit code (especially if
it is a VM, make sure the VM "hardware" is ready for 64-bit code). The
lm
flag ("Long Mode") needs to be visible in the CPU flags, eg:
ewen@debian:~$ grep ' lm ' /proc/cpuinfo
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
ewen@debian:~$
Or check with:
lscpu | grep 64-bit
if you have lscpu
installed.
Make sure you have access to the system console (eg, via a serial console
or a virtual machine console, or a remote KVM switch -- or maybe by
being local to the system). At one point in the upgrade you need to
select a specific kernel version, from the grub
boot menu, which requires
console access.
Finish any package installs, and upgrades on the Debian Linux i386 system,
and make sure any earlier upgrades have been tidied up. Eg, dpkg --audit
should be clean and dpkg -l | grep -v 'ii'
shouldn't show anything left.
Any obsolete packages that you cannot reinstall again should be removed.
Make sure the system boots cleanly on the Debian Linux i386 install.
Make a backup. That you are confident you can restore from and get a working system again.
Record the package status of the i386 system, for cross reference later on (eg, confirming the same packages got reinstalled, and the same package are marked as "automatic dependencies"):
apt-mark showauto >i386-auto-packages
dpkg --get-selections >i386-package-selections
Switch to dual architectures, and switch to a 64-bit Linux Kernel
Enable the amd64
architecture as a foreign architecture:
dpkg --print-architecture
sudo dpkg --add-architecture amd64
dpkg --print-foreign-architectures
sudo apt-get update
The base architecture at this point should be i386
, but you should
be able to install amd64
architecture packages by explicitly specifying
them at this point.
Then install the 64-bit amd64 kernel, and boot into that:
sudo apt-get install linux-image-amd64:amd64
sudo update-grub
sudo shutdown -r now
For this reboot (and only this reboot) you will have to interrupt the
grub
automatic boot, and pick a different kernel from the menu.
Specifically you need to go into the Advanced options for Debian GNU/Linux
menu option (second menu) and pick the amd64
kernel entry which
is named something like Debian GNU/Linux, with Linux 5.10.0-26-amd64
and will be the third entry in that submenu (because the existing i386
kernel sorts before the amd64 kernel in the generated list).
Make sure the system booted into a 64-bit kernel:
uname -m
should return x86_64
if you are running a 64-bit kernel. If not,
try rebooting again and choosing the correct kernel in the grub
boot menu.
Once you have the 64-bit kernel booting, remove the equivalent i386 (32-bit) kernel, and reboot again to make sure the 64-bit kernel is booted automatically from now on:
sudo apt-get purge linux-image-686-pae linux-image-5.10.0-26-686-pae
sudo update-grub
sudo shutdown -r now
If you wish you can pause at this point indefinitely -- the Linux i386
user land should run on the Linux amd64 kernel without any substantial
problems, as this was used for a long period in the early x86_64
kernel development.
Change the primary architecture to amd64
This is one of the more complicated steps. At this point we are changing
dpkg
/ apt
to the amd64
, which changes the primary architecture from
i386
to amd64
.
On Debian Buster, perl-base:i386
conflicts (indirectly, I think)
with perl-base:amd64
. So we have to force installing perl-base:amd64
at this point. The conflict is sufficiently bad that we cannot
even use apt
to download packages once dpkg
/ apt
have been
changed to amd64
. So we need to pre-download several amd64
packages first, then install them "all at once". And then let apt
fix up the conflicts any way it can, after we have installed a few
more perl
packages. (We cannot install those additional packages
at the same time as perl-base:amd64
as the conflicts are too complicated
to resolve, so apt
gives up.)
This combination of steps worked for me on my Debian Buster systems
(although apt-get -f install
removed some packages important to me
on each system, which had to be put back later on):
sudo -s
apt-get clean
apt-get --download-only install dpkg:amd64 tar:amd64 apt:amd64 apt-utils:amd64
(cd /var/cache/apt/archives/ && apt download perl-base:amd64)
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --print-architecture # Should show: amd64
dpkg --print-foreign-architectures # Should show: i386
Note the two runs of dpkg --install ...
for the same packages; this
is required because some packages will not install on the first run
due to missing amd64
dependencies; but they can install the second
time around. (There might be a topological sort of those dependencies
to install them in "the right order". But for these critical packages,
there are lots of circular dependencies, so simply running the install
of these key packages twice is the most pragmatic approach.)
Once perl-base:amd64
is installed, a bunch of other Perl dependencies
used for Debian package management will be broken, so some additional
perl packages need to be upgraded at this step, including
libtext-csv-xs-perl
and libencode-perl
:
(cd /var/cache/apt/archives/ && apt download libperl5.32:amd64 libgdbm-compat4:amd64 libgdbm6:amd64)
(cd /var/cache/apt/archives/ && apt download perl:amd64)
(cd /var/cache/apt/archives/ && apt download libencode-perl:amd64)
(cd /var/cache/apt/archives/ && apt download libtext-csv-xs-perl:amd64)
(cd /var/cache/apt/archives/ && apt download libdebconfclient0:amd64)
dpkg --install /var/cache/apt/archives/*_amd64.deb
At this point it should be possible to get apt
back to a point where
it is "happy" automatically, but do keep track of which packages it is
removing as some of them will be important to you and need to be
reinstalled later on. Those packages need to be removed at this point
to break dependency loops:
apt-get -f install
dpkg --configure -a
Switch the shell to amd64
Install the amd64
bash and dash, and run them instead of the
32-bit versions. This will pull in several other key dependencies,
and also force-remove the i386
versions, so you will have to agree
that you know this can break your system and you want to do it
anyway (Yes, do as I say!
):
apt-get install dash bash
exec /bin/bash
Install the remaining amd64
replacements for installed i386
packages
Download the packages which are needed to install amd64
replacements:
apt-get --download-only -y --no-install-recommends install \
`dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 | uniq |
grep -v 'linux-image.*686-pae' |
sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'`
Then install as many of the amd64
libraries as possible (since these
can mostly be installed in parallel with the i386
packages, and fix
dependency issues):
dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb
apt-get -f install
dpkg --configure -a
(we also install the outstanding perl packages at this point for certainty).
Then we can install the majority of the remaining amd64
packages:
dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386 | grep -vf /tmp/64-bit | cut -f 1 -d : |
xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' | uniq |
tee /tmp/packages-to-install
dpkg --install `cat /tmp/packages-to-install`
which might take a second pass to be able to install all the packages due to dependency issues:
dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386 | grep -vf /tmp/64-bit | cut -f 1 -d : |
grep -v 'linux-image.*686-pae' |
xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' | uniq |
tee /tmp/packages-to-install
test -s /tmp/packages-to-install && dpkg --install `cat /tmp/packages-to-install`
Tidy up the amd64
system
Check what else is left to be converted from i386
to amd64
. Some
packages will still have i386
versions installed, but will also have
an amd64
version -- packages like gcc-9-base
and gcc-10-base
are
effectively library packages and permit parallel installs of both i386
and amd64
versions.
dpkg --get-selections | grep :i386 | grep -v lib | grep -v deinstall | wc -l
dpkg --get-selections | grep :i386 | grep -v lib | grep -v deinstall
dpkg --get-selections | grep :i386 | grep -v lib | awk '$2 ~ /^install$/ { print $1; }' | grep -v 'linux-image.*686-pae' | sed 's/:i386/:amd64/;' | grep -vf /tmp/64-bit
There may still be conflicts at this point, in which case you will need
to figure out some way to resolve them for your combination of packages;
it might involve removing a package important to you, cross grading
other packages, and then reinstalling the packages you wanted on the system
once the dependencies have been resolved properly. Inevitably if apt
cannot figure out a solution, it will be a complicated problem, so assess
carefully what you can live without temporarily to be able to make forward
progress.
Once that is done, run a normal upgrade of the amd64
system to make
sure apt
is happy, and reboot the system to make sure all the running
programs are the amd64
versions:
apt-get update
apt-get upgrade
apt-get dist-upgrade
sudo update-grub
sudo shutdown -r now
Assuming the system came back up as a 64-bit system:
uname -m # Should show: x86_64
getconf LONG_BIT # Should show: 64
then clean up most of the i386
libraries next:
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get --purge autoremove
Then I found if gnuplot
was installed, I had to temporarily remove it
to make progress on the remaining cleanup (and install it again later):
HAVE_GNUPLOT=$(dpkg -l gnuplot >/dev/null 2>&1 && echo "true")
if [ -n "${HAVE_GNUPLOT}" ]; then sudo apt-get remove gnuplot; fi
before I could remove the essential i386
libraries, which hopefully
by this point your system does not need (just the amd64
versions of
those libraries). But apt
/ dpkg
does not know this so you will
have to confirm you really want to remove these essential libraries
(Yes, do as I say!
):
dpkg --get-selections | grep :i386 | grep -v deinstall | wc -l
sudo apt-get purge $(dpkg --get-selections | grep :i386 | grep -v deinstall | awk '{print $1;}')
Then you should be able to reinstall any packages that were removed to break dependency loops:
dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/'
sudo apt-get install --no-install-recommends $(dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/')
sudo apt-get -f install
sudo dpkg --configure -a
dpkg --get-selections | grep deinstall
sudo apt-get install $(dpkg --get-selections | awk '/deinstall/ { print $1; }')
Including reinstalling gnuplot
again if you had to remove it above:
if [ -n "${HAVE_GNUPLOT}" ]; then sudo apt-get install gnuplot; fi
Then confirm there is nothing else that needs to be reinstalled:
dpkg --get-selections | grep deinstall
dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/'
and no i386
packages still installed:
dpkg --get-selections | grep i386
dpkg -l | grep "^i.*:386"
Cleaning up from the crossgrade
Do a regular package update to make sure there are no complaints, and then reboot to make sure you have a working system:
sudo apt-get update
sudo apt-get dist-upgrade
sudo dpkg --configure -a
sudo update-grub
sudo shutdown -r now
Assuming the system booted okay, purge the i386
packages that were
removed, and remove the i386
architecture:
sudo apt-get --purge autoremove
sudo dpkg --remove-architecture i386
dpkg --print-architecture # Should show: amd64
dpkg --print-foreign-architectures # Should return nothing
Then make sure that dpkg
/ apt
are happy:
dpkg --audit
dpkg -l | grep -v "^ii"
and if the package state is tidy reboot again as a plain amd64
system:
sudo update-grub
sudo shutdown -r now
Ensure your production system is production ready again
Check for any other packages that might have been removed as dependencies
and need to be installed, by comparing the i386
package list to the
amd64
package list:
dpkg --get-selections >/tmp/amd64-package-selections
diff -bw <(sed 's/:i386/:amd64/;' i386-package-selections | sort) <(sort </tmp/amd64-package-selections) | grep "<" | grep -v "686-pae" | awk '{print $2;}' | grep -v "^lib" | grep -v pkgmonkey-client | tee /tmp/missing-packages
sudo apt-get install $(cat /tmp/missing-packages)
dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep "^>"
sudo apt-get install $(diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep -v "pkgmonkey-client" | awk '/^>/ {print $2;}')
dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep -v "pkgmonkey-client" | grep "^>"
Then check for any extra packages that got dragged in as dependencies during the upgrade which you did not have installed before and do not want, and consider if you want to remove them:
dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep "^<" | grep -v binutils | grep -v 'lib.san' | grep -v 'libunwind8'
Beware that some of these additional packages will be new amd64
dependencies. Eg, strace
on amd64
requires libunwind8
, and
gcc
on amd64
indirectly requires liblsan0
and libtsan0
. It
will take some systems administration experience, and some investigation,
to determine what is still needed for production and what was temporarily
installed to meet dependencies during the cross grade.
Once you are happy that the production packages are installed, and no unnecessary packages are installed, restore the "automatically installed" marks so that future upgrades will work better:
sed 's/:i386/:amd64/' i386-auto-packages | grep -v 686-pae | xargs sudo apt-mark auto
(which uses the list of i386
automatically installed packages, created at the beginning).
Then remove any packages which are now no longer required:
sudo apt-get -f install
sudo apt-get --purge autoremove
and confirm you do not have any 386
or 686
packages left:
dpkg -l | grep '[36]86'
Architecture upgrades for specific tools
Some packages have their own architecture dependent files on disk. Particularly databases, but also other programs. You will need to figure out, before committing to the production upgrade, how to handle these situations.
On my mail servers, I found that (a) postfix
got removed during the
crossgrade (replaced by exim
) and had to be reinstalled, and (b) the
spamassassin
tool had some C libraries that it compiles to do regular
expressions faster than in perl
, which have to to be rebuilt, with
sa-compile
.
In the case of spamassassin
I did this with:
test -x /usr/bin/sa-compile && (
sudo sa-compile --list >/tmp/before
sudo sa-compile
sudo sa-compile --list >/tmp/after
diff /tmp/before /tmp/after
sudo service spamassassin restart
sudo service spamass-milter restart
sudo service postfix restart
)
Then once you have a final configuration, reboot the system one more time to make sure you are running your final package configuration:
sudo update-grub
sudo shutdown -r now
And watch your logs and monitoring very carefully for the next few hours to keep an eye out for other problems that need manual fixes.
From this point on, if you successfully got to a working system and
fixed any data architecture dependencies, then it should act like
any other Debian Linux amd64
install, including for any future
upgrades.