Debian i386 to amd64 crossgrade

(If you are attempting this on a modern Debian Linux system, see my more up to date Debian Bullseye i386 to amd64 crossgrade.)

Introduction

I have a number of old Debian 32-bit (i386) installations, mostly on VMs, which were originally installed either when 32-bit (i386) was the only option or when 32-bit (i386) seemed like the best option for a small VM due to memmory usage. All of them have been upgraded, repeatedly, through Debian releases and are now running Debian Jessie (8), Stretch (9) or Sid ("unstable").

Because it is becoming obvious that x86 32-bit (i386) is a dead end in terms of software support -- particularly for hardware bug mitigations for issues like Meltdown and Spectre it is clear that these installs will need to be converted to x86_64 64-bit (amd64) installations. Either by reinstalling them (non-trivial for older "pet" installations which have been maintained for many years) or by CrossGrading them between 32-bit and 64-bit.

Today I decided to try CrossGrading a small Debian Unstable test VM that I had -- originally installed many years ago, and apt-get dist-upgraded all the way to the present day -- from 32-bit to 64-bit to find out for myself how complicated it was.

To put the conclusion first: if you have any easy way to reinstall the system, you will almost certainly want to reinstall the system. If the system is important, you probably also want to reinstall the system -- ideally with a "forklift" upgrade where you build the replacement in parallel first -- even if that is not easy to reinstall, as the broken system mid-cross grade will make you very sad.

If you try to cross-grade your system you will end up with a broken system. If you are lucky and experienced with Debian packaging and rescuing broken systems it will only be somewhat broken, and you will be able to fix it over the course of a few hours. If you are unlucky, or not very experienced, then your system may well end up permanently broken. I went into this with around 20 years of experience with Debian Linux as a Sysadmin, and mid-crossgrade the system was still about the most broken I have ever seen a Debian system. (It all worked out in the end, but it took 2-3 times as long as I had originally estimated to get working.)

Because "crossgrading" is an idea whose time is current, there are several other guides around, including:

Debian Stretch: Upgrade 32-bit to 64-bit
Rimu Hosting's 32 to 64-bit Distro Crossgrades, with an alpha tool that I have not tried myself
Upgrading Debian from 32-bit to 64-bit, a slighty older guide which also links to some other guides.

I would strongly recommend reading several of these guides before starting so you know what you are getting into. As I said above, it is probably easier to reinstall, or install a new system and swap it into place of the old one.

Preparation

Backup

Make a backup before you started. At minimum you will want something you can refer to the configuration files before you started. But you probably want a way to "undo" everything and get back to a working system. If you are using virtualisation that provides a snapshot function use it before you start.

Disk space

You will need a decent amount of free space on your / and /var/cache partitions during this upgrade, because at some points you will have the 64-bit packages for the entire system cached, as well as two sets of packages installed (32-bit and 64-bit packages, especially for libraries). Expect to need several GiB of free space.

In my case the VM I wanted to try upgrading (debian_unstable), had also been installed onto a tiny virtual disk (4GB), which was already tight without even allowing space for cross grading. So I chose to increase its disk size before starting. Because this was the root disk and it was booted from it, it was a little bit more complicated. But modern Linux will cope with online upgrades of a / ext4 file system.

Outside the VM, expand the underlying storage:

sudo lvextend -L 10G /dev/r1/debian_unstable

then bounce the VM so that it sees the new disk space;

sudo shutdown -h now

and start the VM up again from cold.

Next repartition the disk image to use all the new space:

sudo apt-get install parted
sudo parted /dev/vda
# Inside parted:
  print
  resizepart 1
    y    # Continue with mounted partition
    100% # Use entire remaining space
  print
  quit

This will look something like:

ewen@debian-unstable:~$ sudo parted /dev/vda
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  4297MB  4296MB  primary  ext3         boot

(parted)
(parted) resizepart 1
Warning: Partition /dev/vda1 is being used. Are you sure you want to continue?
Yes/No? y
End?  [4297MB]? 100%
(parted) print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  10.7GB  10.7GB  primary  ext3         boot

(parted) quit
Information: You may need to update /etc/fstab.

ewen@debian-unstable:~$
ewen@debian-unstable:~$ sudo fdisk -l /dev/vda
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000c1991

Device     Boot Start      End  Sectors Size Id Type
/dev/vda1  *     2048 20971519 20969472  10G 83 Linux
ewen@debian-unstable:~$

Reboot to force kernel to re-read the partition table:

sudo shutdown -r now

(or run partprobe to force the partition table to be re-read.)

Then resize the mounted partition online (requires relatively recent Linux kernel and tools, eg, the last few years):

sudo resize2fs -p /dev/vda1

which should give you a result like:

ewen@debian-unstable:~$ sudo resize2fs -p /dev/vda1
resize2fs 1.44.2 (14-May-2018)
Filesystem at /dev/vda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vda1 is now 2621184 (4k) blocks long.

ewen@debian-unstable:~$
ewen@debian-unstable:~$ df -Pm .
Filesystem     1048576-blocks  Used Available Capacity Mounted on
/dev/vda1                9951  2932      6569      31% /
ewen@debian-unstable:~$

(note the lack of progress (-p) due to the online resize; fortunately 4GiB to 10GiB is a fairly quick expansion).

One more reboot to ensure that it boots cleanly on the new partition:

sudo shutdown -r now

Grub serial boot menus

To get modern grub (last year or so) to actually show a menu during its timeout, on the serial console, you need in /etc/default/grub:

GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
[...]
GRUB_CMDLINE_LINUX="console=ttyS0,9600"
[...]
# Serial output
GRUB_TERMINAL="serial"
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

then run sudo update-grub, and reboot.

Unlike earlier grub versions, the GRUB_TIMEOUT_STYLE=menu no longer seems to be the default, and without GRUB_TIMEOUT_STYLE set, it will default to the "hidden" behaviour. This is a rather confusing change in default behaviour :-( You will need to be able to pick a custom kernel from the grub boot menu a little later during the cross-grade, so make sure you have a working way to get at the grub menu now.

64-bit hardware

If you are using a virtual machine, make sure that it is providing a 64-bit capable CPU. For my old VMs they were deliberately installed with only the 32-bit capable CPU flags visible, which does not work with a 64-bit kernel/programs (amd64) for obvious reasons.

For instance if you are using libvirt and KVM, use something like virsh edit to replace:

  arch='i686' machine='pc-0.12'

in the os hvm line, with equivalent values for 64-bit:

  arch='x86_64' machine='pc'

If you needed to make any changes, halt the VM and power it on again to make sure that the system still boots with the 32-bit install on the new hardware architecture before proceeding.

ETA: You can confirm the 64-bit capable CPU architecture is visible with:

lscpu | grep 64-bit

which should return something like:

CPU op-mode(s):                     32-bit, 64-bit

on a 64-bit capable CPU (and only show 32-bit on a 32-bit only CPU).

Install all pending updates

Make sure your 32-bit (i386) install is fully up to date, and you are running all the updated versions:

sudo apt-get update
sudo apt-get dist-upgrade
sudo shutdown -r now

Crossgrading from 32-bit to 64-bit

There are quite a few steps in the cross-grade process. This process is based extensively on the Debian CrossGrading guide, and Stefan's Upgrade 32-bit to 64-bit blog post, as well as the issues that I encountered on my test machine during the upgrade.

Set up dual-architecture, 32-bit and 64-bit

Enable the 64-bit (amd64) architecture as a foreign architecture:

dpkg --print-architecture
sudo dpkg --add-architecture amd64
dpkg --print-foreign-architectures

giving results something like:

ewen@debian-unstable:~$ dpkg --print-architecture
i386
ewen@debian-unstable:~$ sudo dpkg --add-architecture amd64
ewen@debian-unstable:~$ dpkg --print-foreign-architectures
amd64
ewen@debian-unstable:~$

and then update the package lists so you have both:

sudo apt-get update

You should see the amd64 package lists downloaded too.

Install and boot the 64-bit kernel

sudo apt-get install linux-image-amd64:amd64

which will also pull in a number of other 64-bit packages:

ewen@debian-unstable:~$ sudo apt-get install linux-image-amd64:amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libnuma1
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  apparmor:amd64 gcc-8-base:amd64 irqbalance:amd64 libblkid1:amd64 libc6:amd64
  libcap-ng0:amd64 libffi6:amd64 libgcc1:amd64 libgcrypt20:amd64
  libglib2.0-0:amd64 libgpg-error0:amd64 libgpm2:amd64 liblz4-1:amd64
  liblzma5:amd64 libmount1:amd64 libncursesw6:amd64 libnuma1:amd64
  libpcre3:amd64 libselinux1:amd64 libsystemd0:amd64 libtinfo6:amd64
  libuuid1:amd64 linux-image-4.16.0-2-amd64:amd64 zlib1g:amd64
Suggested packages:
  apparmor-profiles-extra:amd64 apparmor-utils:amd64 glibc-doc:amd64
  locales:amd64 rng-tools:amd64 gpm:amd64 linux-doc-4.16:amd64
  debian-kernel-handbook:amd64
The following packages will be REMOVED:
  apparmor irqbalance
The following NEW packages will be installed:
  apparmor:amd64 gcc-8-base:amd64 irqbalance:amd64 libblkid1:amd64 libc6:amd64
  libcap-ng0:amd64 libffi6:amd64 libgcc1:amd64 libgcrypt20:amd64
  libglib2.0-0:amd64 libgpg-error0:amd64 libgpm2:amd64 liblz4-1:amd64
  liblzma5:amd64 libmount1:amd64 libncursesw6:amd64 libnuma1:amd64
  libpcre3:amd64 libselinux1:amd64 libsystemd0:amd64 libtinfo6:amd64
  libuuid1:amd64 linux-image-4.16.0-2-amd64:amd64 linux-image-amd64:amd64
  zlib1g:amd64
0 upgraded, 25 newly installed, 2 to remove and 1 not upgraded.
Need to get 55.0 MB of archives.
After this operation, 282 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Chances are that if you just updated you will have the same kernel version in 686-pae and amd64 variants installed, and the default update-grub sorting will put the 32-bit (686-pae) one first. Your best option at this point is:

Reboot, and choose the 64-bit version from the grub boot menu, and make sure that it boots. (If it seems to just hang chances are that the CPU architecture has not been updated to be 64-bit (x86_64) compatible, in which case if you are using virtualisation you might need to update the CPU architecture. See the suggestions above.)

Check that you booted the 64-bit kernel:

ewen@debian-unstable:~$ uname -a
Linux debian-unstable 4.16.0-2-amd64 #1 SMP Debian 4.16.12-1 (2018-05-27) x86_64 GNU/Linux
ewen@debian-unstable:~$ uname -m
x86_64
ewen@debian-unstable:~$

Assuming all is well, uninstall the 32-bit kernel of the same version so that there are not two matching kernel versions, as well as the packages which are dragging it in:
```
sudo apt-get purge linux-image-2.6-686 linux-image-686-pae linux-image-4.16.0-2-686-pae
```
which should leave the newer amd64 kernel, and the older 686-pae kernel, and have the newer amd64 kernel as the default one booted.

Run update-grub and check that the amd64 kernel is now sorted first:

ewen@debian-unstable:~$ sudo update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.16.0-2-amd64
Found initrd image: /boot/initrd.img-4.16.0-2-amd64
Found linux image: /boot/vmlinuz-4.16.0-1-686-pae
Found initrd image: /boot/initrd.img-4.16.0-1-686-pae
done
ewen@debian-unstable:~$

Reboot again to make sure that it boots by default without intervention:
```
sudo shutdown -r now
```
checking that uname -m returns x86_64 as expected after rebooting.

Clean up package state and ensure everything is in sync

sudo apt-get clean
sudo apt-get update
sudo apt-get upgrade
sudo apt-get --purge autoremove

Convert `dpkg`, `tar`, and `apt` to 64-bit

This is the main cutover of the default architecture, and is the first tricky step. You need to download the packages first, which will also pull in their dependencies. Then install all the relevant packages together, using dpkg directly. Due to the dependencies, I found I needed to do this step twice before all the packages would install and find their 64-bit dependencies.

This step is probably best done as root for safety:

sudo -s
apt-get clean
apt-get --download-only install dpkg:amd64 tar:amd64 apt:amd64
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --install /var/cache/apt/archives/*_amd64.deb

Finally check that the default architecture and foreign architecture have swapped over:

dpkg --print-architecture
dpkg --print-foreign-architectures

which should show you something like:

debian-unstable:/home/ewen# dpkg --print-architecture
amd64
debian-unstable:/home/ewen# dpkg --print-foreign-architectures
i386
debian-unstable:/home/ewen#

Update package lists again, for new default architecture

apt-get update

Now draw the rest of the owl

Most of the instruction guides get a bit hand-wavy at this point, which feels a lot like How to draw an owl. Stefan's blog post contains some useful hints on where to start.

At this point there are a few dozen 64-bit (:amd64) packages installed, and hundreds of 32-bit (:i386) packages installed. But the system thinks it is a 64-bit (amd64) system, that happens to have a lot of foreign packages installed -- and has a kernel which is capable of running both. This means that new packages will default to installing the 64-bit (amd64) version, but the existing packages will stay 32-bit (i386) unless they are manually replaced.

Replacing the 32-bit (i386) packages with 64-bit (amd64) packages is complicated by the fact that apt does not directly understand this replacement -- only that it can install 32-bit and 64-bit packages along side each other if they are libraries, or other packages explicitly designed for multiarch installation (or they are independent packages). So we need to guide dpkg and apt through the conversion process.

Stefan's Blog Post on Upgrading 32-bit to 64-bit recommends downloading packages with apt-get --download-only and installing them directly with dpkg (ie, as above) in stages, to upgrade the system. Unfortuantely I did not find that blog post early enough to see the hint to install dash and bash before changing the default architecture, but given the trouble I had with dash (below) I would agree with the suggestio to get that upgrade out of the way early.

The key download commamd is:

apt-get --download-only -y --no-install-recommends install \
    `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 |
     sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'`

which tries to download 64-bit (amd64) versions of every installed 32-bit (i386) package.

The first complication, at least on a system that has been upgraded between Debian versions -- or Debian Unstable upgraded repeatedly for a long time -- is that there will be quite a few packages installed which are no longer current and thus cannot be installed again. The only reasonable option for those is to remove those old packages that no longer exist in Debian. Finding them mostly involves looking out for errors like:

Package gcc-4.7-base is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

and then unistalling the relevant packages, and the packages which depend on them, with dpkg:

dpkg --purge gcc-4.{6,7,8,9}{,-base} cpp-4.{6,7,8,9} libgcc-4.{7,8,9}-dev libasan{0,1}
dpkg --purge libssl0.9.8 consolekit sysvinit

The second complication was with grub, where grub-legacy was being auto-selected, instead of the newer grub-pc (grub2). Which would then create boot complications. To avoid ths problem explicitly remove the transitional package:

dpkg --purge grub

so that the cross-install happens based on the new package (grub-pc).

The third complication was that quite a few libraries could not be found, because they too had been outdated. These show up with errors like:

E: Unable to locate package libtasn1-0:amd64
E: Unable to locate package libtasn1-2:amd64
E: Unable to locate package libtasn1-3:amd64
E: Unable to locate package libtiff4:amd64
E: Unable to locate package libtxc-dxtn-s2tc0:amd64
E: Unable to locate package libvolume-id0:amd64
E: Unable to locate package libvolume-id1:amd64
E: Unable to locate package libzmq3:amd64

and so on. The fix for those, assuming that they are in fact old unneded libraries, is to remove the 32-bit (i386) version.

dpkg --purge libssl0.9.7
dpkg --purge libdb3 libdb4.2 libdb4.3 libdb4.4
dpkg --purge libgnutls11 libgnutls12 libgnutls13
dpkg --purge libtasn1-0 libtasn1-2 libtasn1-3
dpkg --purge libvolume-id0 libvolume-id1

and so on and so on. Where other things depend on those libraries, those other things will also have to be removed. But usually all of those packages are ancient packages removed from Debian, and either unnecessary or replaced by something else.

Also being hit by this will be the older 32-bit kernel that you may still have installed. You probably also want to remove that at this point as it will very shortly be unsuitable for booting the system (as key packages will be 64-bit, and not run on the 32-bit kernel):

dpkg --purge linux-image-4.16.0-1-686-pae

This will leave you with just one kernel, the latest 64-bit kernel. (As an alternative you could just grep -v that kernel package out of the equivalent packages to install. But beware that you will not be able to usefully boot that 32-bit kernel for much longer anyway.)

The fourth complication is that there might be some old development packages which depend on old libraries which no longer exist, so you might have unmet dependencies. Eg,

libgcc-5-dev : Depends: liblsan0 (>= 5.5.0-12) but it is not going to be installed
               Depends: libtsan0 (>= 5.5.0-12) but it is not going to be installed

and for those that are not relevant, the fix is the same, to remove the packages, and the things which depend on them, so that the dependencies are not an issue and you only have packages which can be installed:

dpkg --purge libgcc-{5,6}-dev gcc-{5,6}

This issue also happens for libgcc-7-dev, which g++, gcc, etc depend on:

libgcc-7-dev : Depends: liblsan0 (>= 7.3.0-23) but it is not going to be installed
               Depends: libtsan0 (>= 7.3.0-23) but it is not going to be installed

and:

dpkg: dependency problems prevent removal of gcc-7:i386:
 g++:i386 depends on gcc-7 (>= 7.3.0-12~).
 gcc:i386 depends on gcc-7 (>= 7.3.0-12~).
 g++-7:i386 depends on gcc-7 (= 7.3.0-23).

as well as binutils:

binutils : Depends: binutils-x86-64-linux-gnu (= 2.30-21)

So the best way forward for those is to explicitly specify the additional packages those packages need in the download list, to be installed.

Once everything seems tidy, download all the needed packages with something like:

apt-get clean
apt-get --download-only -y --no-install-recommends install \
    `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 |
     sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'` \
     binutils-x86-64-linux-gnu liblsan0 libtsan0A

And then all the packages that you need will be downloaded, but not installed, ending with something like:

Fetched 253 MB in 1min 36s (2,637 kB/s)
Download complete and in download only mode

if you are fairly close to your Debian mirror (and much longer if you are not).

Once the download succeeds, it is worth starting by installing all the 64-bit libraries along side the 32-bit libraries, as generally those can be installed in parallel. It is very useful to include perl packages in this as well, as several perl libraries contain -xs (ie, compiled C) functions, which depend on the perl packages:

dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb

Due to dpkg installing in the order listed, some packages may fail due to other packages not yet being installed (particularly those depending on perl), so it can help to run this same command a second time:

dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb

but there will still be a number of failures due to other dependencies.

However by getting the bulk of the libraries installed in parallel, it will provide most of the required dependencies for other packages.

At this point running:

dpkg --configure -a

will provide a hint as to why the remaining packages could not be configured. These can be solved individually by installing the packages mentioned, eg:

debian-unstable:/home/ewen# dpkg --configure -a
dpkg: dependency problems prevent configuration of libparted2:amd64:
 libparted2:amd64 depends on dmidecode.

can be fixed with:

dpkg --install /var/cache/apt/archives/dmidecode*deb

Some of them will just have failed due to earlier ones failing, so it is worth periodically trying:

dpkg --configure -a

to figure out what still needs manually solving.

I found that:

dpkg --install /var/cache/apt/archives/gcc*base*deb
dpkg --install /var/cache/apt/archives/binutils*deb
dpkg --install /var/cache/apt/archives/linux-libc-dev*deb

unlocked quite a few of the library packages.

And some packages just got stuck because Debian Unstable is, well, Unstable. And thus package versions may not always match up. Eg,

dpkg: error processing package libmagickwand-6.q16-5:i386 (--configure):
 package libmagickwand-6.q16-5:i386 8:6.9.9.34+dfsg-3+b1 cannot be configured because libmagickwand-6.q16-5:amd64 is at a different version (8:6.9.9.39+dfsg-1)

and for some of those I resolved them just by removing the uninstallable packages, in both 32-bit (i386) and 64-bit (amd64) variants, using dpkg --purge. Or just the uninstallable 64-bit (amd64) variant at present if the 32-bit program/library looked important.

You will want to continue manually resolving these issues until:

dpkg --configure -a

runs cleanly.

Then it is a matter of figuring out what programs are installed as 32-bit (i386) that need to be reinstalled as 64-bit (amd64). This can be done with:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit

I chose to install bash and dash next, as I had missed the hint to install those early on:

dpkg --install /var/cache/apt/archives/{d,b}ash*deb

bash seemed to install okay, but dash ran into install problems:

Preparing to unpack .../dash_0.5.8-2.10_amd64.deb ...
Ignoring request to remove shared diversion 'diversion of /bin/sh to /bin/sh.distrib by dash'.
dpkg-divert: error: 'diversion of /bin/sh to /bin/sh.distrib by bash' clashes with 'diversion of /bin/sh to /bin/sh.distrib by dash'
dash.preinst: dpkg-divert exited with status 2
dpkg: error processing archive /var/cache/apt/archives/dash_0.5.8-2.10_amd64.deb (--install):

however /bin/sh was still diverted to /bin/dash:

debian-unstable:/home/ewen# dpkg-divert --list /bin/sh
diversion of /bin/sh to /bin/sh.distrib by dash
debian-unstable:/home/ewen# readlink -f /bin/sh
/bin/dash
debian-unstable:/home/ewen#

Following advice in a blog post with a similar issue I tried manually removing the divert:

dpkg-divert --remove /bin/sh

and tried installing dash again, but ran into issues with sh.1.gz, so did both:

dpkg-divert --remove /bin/sh
dpkg-divert --remove /usr/share/man/man1/sh.1.gz

(because the first one got put back on the next install attempt), and then the installation was able to succeed with:

dpkg --install /var/cache/apt/archives/dash*deb

It appears with only one diversion present (the "background" one of bash), the scripts were then automatically able to cope. I have a feeling this might have resulted from a much earlier Debian Unstable change when the switch over to dash as /bin/sh first happened.

At this point I went back to look what was still installed as :i386 only:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit

Which I could then turn into a list of packages to install:

dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit | cut -f 1 -d : |
       xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' |
       tee /tmp/packages-to-install

And then install them:

dpkg --install `cat /tmp/packages-to-install`

The majority of them will install without problems, but not all of them. Some of the packages will have issues installing or configuring, which you will need to solve manually.

For the ones that fail, which in my case were:

Errors were encountered while processing:
 /var/cache/apt/archives/python2_2.7.15-3_amd64.deb
 /var/cache/apt/archives/python_2.7.15-3_amd64.deb
 /var/cache/apt/archives/python3_3.6.5-3_amd64.deb
 mercurial
 python-dumbnet
 python-lxml:amd64

trying them again manually in useful sets will probably help. In my case manually installing the python .deb packages again sorted those, and mercurial and python-dumbnet out in one go, just leaving python-lxml.

python-lxml failed because there were two instances of it installed, one 32-bit (i386) and one 64-bit (amd64) during the replacement, which confused its ability to self-compile:

dpkg: error processing archive /var/cache/apt/archives/python-lxml_4.2.1-1_amd64.deb (--install):
 new python-lxml:amd64 package pre-removal script subprocess returned error exit status 1
dpkg-query: error: --listfiles needs a valid package name but 'python-lxml' is not: ambiguous package name 'python-lxml' with more than one installed instance

And unfortunately dpkg --purge would not allow removing either instance, due to the .prerm scripts, which made it difficult to get down to one instance. Looking at the /var/lib/dpkg/info/python-lxml:i386.prerm and /var/lib/dpkg/info/python-lxml:amd64.prerm scripts, the problem was that they attempted to find the installed Python files for that package, to clean them up -- but assumed only one version was installed, due to:

pyclean -p python-lxml
[...]
dpkg -L python-lxml | grep '\.py$' | while read file

and I was able to fix that by manually editing the .prerm files to explicitly specify the version installed in each case (ie, adding ":i386" to the end of the package names in both cases. Then it was possible to do:

dpkg --purge python-lxml:i386

successfully, after which the 64-bit version could be installed again:

dpkg --install /var/cache/apt/archives/python-lxml_4.2.1-1_amd64.deb

And that seemed to be pretty much all packages converted over.

debian-unstable:/home/ewen# dpkg --get-selections | grep :i386 | grep -v lib | wc -l
9
debian-unstable:/home/ewen#

and it looked like most of the remaining non-lib packages were also parallel installed packages.

So next I ran:

apt-get update
apt-get upgrade

to figure out what apt-get considered broken. On my system that returned some 32-bit (i386) packages with unmet dependencies:

You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libespeak-ng1:i386 : Depends: libpcaudio0:i386 but it is not installed
 liblvm2cmd2.02:i386 : Depends: dmeventd:i386 but it is not installed
 libparted2:i386 : Depends: dmidecode:i386 but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

so I just removed the 32-bit (i386) library packages at this point.

dpkg --purge libespeak-ng1:i386 liblvm2cmd2.02:i386 libparted2:i386

and then ran:

apt-get upgrade

again. Which this time told me there were a lot of automatically installed 32-bit (i386) packages which were no longer required, but listed no other errors.

Finishing up

Before continuing I then rebooted the system to make sure it was running the 64-bit (amd64) versions of everything, including the shell that I was logged into, etc.

update-grub
shutdown -r now

Assuming it comes back up again properly, and you can log in, run:

getconf LONG_BIT

to check that the system libraries, not just the kernel (in uname -m) thinks that the system is 64-bit.

Assuming it looks good, go ahead and allow apt-get to remove the now unnecessary 32-bit (i386) libraries:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get --purge autoremove

That will remove some of the 32-bit libraries, but not all of them:

ewen@debian-unstable:~$ dpkg --get-selections | grep :i386 | wc -l
166
ewen@debian-unstable:~$

To remove the remainder, they need to be purged explicitly:

sudo apt-get purge `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386`

Before saying "yes", double check that apt-get does not want to purge or remove anything else, other than those i386 libraries due to dependencies. Assuming it looks okay, say "yes".

At this point there should be no 32-bit (i386) packages left:

ewen@debian-unstable:~$ dpkg --get-selections | grep :i386
ewen@debian-unstable:~$ dpkg -l | grep "^i.*:i386"
ewen@debian-unstable:~$

Do a final check there is nothing left hanging to do, and a final reboot to make sure all is well:

sudo apt-get update
sudo apt-get dist-upgrade
sudo dpkg --configure -a
sudo update-grub
sudo shutdown -r now

And then you can remove the 32-bit (i386) as a foreign architecture:

sudo dpkg --remove-architecture i386

and you should have a single-architecture, 64-bit (amd64) system:

ewen@debian-unstable:~$ dpkg --print-architecture
amd64
ewen@debian-unstable:~$ dpkg --print-foreign-architectures
ewen@debian-unstable:~$

Final cleanup

At this point it would be worth doing the usual Debian checks, eg:

dpkg --audit
dpkg -l | grep -v "^ii"

and so on to make sure that there are no stuck packages, or half installed/ removed packages.

If you needed to remove any packages in order to proceed with the upgrade you might want to put those back, eg:

sudo apt-get install apt-utils ndiff

(I removed ndiff because of the python-lxml issue above.)

Conclusion

For a relatively small install, this whole process took me about 4-5 hours, including keeping the notes to write this blog post. It might well still have been faster to reinstall. But I wanted to find out for myself how difficult the process was, as I have several other more important old Debian 32-bit installs which I will need to decide whether I am going to cross-grade them or reinstall them. (If you have an easy way to reinstall them -- for instance their configuration is already automated -- I think I would strongly recommend just reinstalling them. Crossgrading is possible, but it is a lot of work to get right.)