Migrate/Upgrade Ubuntu 18.04 LTS (Bionic Beaver) GNU/Linux from 32 bit to 64 bit HowTo

1. Introduction

Sorry, no more upgrades for this system
There will not be any further Ubuntu releases for this system's 'i386' architecture. 
Updates for Ubuntu 18.04 will continue until April 2023.
Restoring original system state
Aborting

This guide is for you if you've seen the above message and you want to upgrade your Ubuntu system to 20.04 LTS.

This brief HOWTO shows how to take a working 32 bit 18.04 installation and convert it into a 64 bit system. You will be able to keep all your programs, settings and data. You initially convert to a 32 bit system that can run 64 bit programs, then convert it to a 64 bit system that can run 32 bit programs. Once converted to 64 bit you can plan the upgrade to 20.04 LTS.

With the release of Debian GNU/Linux wheezy, a new feature called multiarch was introduced. The facility permits the installation of executable files (such as libraries and programs) from multiple architectures simultaneously on the same system. The primary use of this is to allow 32 bit programs to be run on a 64 bit system, but it has other uses.

With the release of Ubuntu LTS 14.04 (Trusty Tahr), the ability to use the multiarch facility to upgrade to 64 bit is easier than ever. Much simpler than upgrading 12.04 and an evolution of upgrading Debian 7 or 8. Now with this later release of Ubuntu, multi-arch is more error tolerant than ever, but care is needed since with this tolerance comes a higher chance of missing something subtle from the conversion.

2. Preparation

Before we change any packages, preserve the list of the packages we have installed. At the end we can run a comparison or reconciliation to see if there are any omissions or remaining work. All steps should be performed using a VT text console rather than the GUI because the GUI will become unavailable during parts of the process. In addition, the steps should be run as root directly, avoiding the use of sudo. Please refer to this comment for further details about this.

dpkg --get-selections "*:i386"|grep -v deinstall >dpkg-get-selections.cp.00.i386
dpkg --get-selections "*:amd64"|grep -v deinstall >dpkg-get-selections.cp.00.amd64
dpkg --get-selections "*:all"|grep -v deinstall >dpkg-get-selections.cp.00.all

3. Backups

Have I mentioned the importance of backups? Ensure you have a tested recoverable backup of your entire system, and also a means to access it should the network or kernel be unavailable. If you have encrypted files or home directories then verify that you can restore these successfully and still access them.

Some packages store their data in a binary format that differs between architectures. We must grab the data in an architecture-independent manner now (e.g. a text dump), in order to restore it in the 64 bit system later. Such packages I am aware of and have installed are OpenLDAP and PostgreSQL. There will be others.

3.1. OpenLDAP

Dump the OpenLDAP database and preserve its configuration. We'll have to purge the package as it will attempt and fail to upgrade its database upon migration.

umask 77
slapcat -n 1 >slapcat-n1
systemctl stop slapd
mkdir -p etc/ldap
rm -Rf etc/ldap/slapd.d
cp -ai /etc/ldap/slapd.d etc/ldap

3.2 PostgreSQL

Dump the PostgreSQL cluster data and preserve its configuration.

su - postgres -c "umask 77; pg_dumpall >/tmp/pg_dumpall_10.sql"
mv /tmp/pg_dumpall_10.sql .
xz -f -T4 pg_dumpall_10.sql
mkdir -p etc/postgresql/10/main
cp -pi /etc/postgresql/10/main/pg_hba.conf etc/postgresql/10/main

4. Boot into a 64 bit kernel

The existing 32 bit kernel does not allow the execution of 64 bit programs. However the 64 bit kernel allows the execution of both 32 bit and 64 bit programs, so this 64 bit kernel is the first thing needed. Run the following in turn to allow 64 bit programs to be installed as an additional architecture, then install the 64 bit kernel. We will also install debootstrap and we will use it later to download the base system.

The steps below will replace the 32 bit kernel with the 64 bit kernel. If you do not have any other 32 bit kernels installed then I recommend backing up the current kernel and initrd before running the commands, and also testing you can boot from the backups easily.

dpkg --add-architecture amd64
apt update
apt install debootstrap linux-generic:amd64 linux-image-generic:amd64 intel-microcode:amd64 amd64-microcode:amd64

Now reboot with the newly installed 64 bit amd64 (also known as x86_64) kernel before continuing.

5. Prepare base systems

Cross-grading the whole existing system is prone to error due to the mass of dependencies, tied with the fact that dpkg currently in general cannot always cross-grade (change architecture) and upgrade (change version) a package in the same operation.

Download the base system's essential packages in 32 bit and 64 bit varieties. The less we download here, the quicker the base system migration will be as we will need to cross-grade them all from 32 bit to 64 bit. If you do require networking then do include (using --include) openssh-client, openssh-server, network-manager and systemd in the --include list below. If you are not comfortable without the kernel, which can be very slow to migrate, do include linux-generic and grub-pc. If you do need LVM then do include lvm2. Whatever you decide, the includes for both architectures must match.

mkdir -p bootstrap-i386
rm -Rf bootstrap-i386/*
debootstrap --arch=i386 --download-only --include openssh-client,openssh-server,network-manager,systemd,linux-generic,grub-pc,lvm2 bionic ./bootstrap-i386

mkdir -p bootstrap-amd64
rm -Rf bootstrap-amd64/*
debootstrap --arch=amd64 --download-only --include openssh-client,openssh-server,network-manager,systemd,linux-generic,grub-pc,lvm2 bionic ./bootstrap-amd64

Now build a package list of the intended stripped-down base system.

ls -1 bootstrap-amd64/var/cache/apt/archives/|egrep "\.deb$"|cut -d "_" -f1|awk '{print $1 " install"}' >dpkg-get-selections.base

6. Purge architecture-independent packages

Purge any packages that cannot be migrated to 64 bit directly. I also strongly recommend removing non-standard package repositories and removing (or preferably purging) any packages installed from those now.

apt purge slapd
apt purge postgresql-10

7. Strip to 32 bit base system

Remove all non-essential packages from the system. Note that rebooting during this process is not recommended as the system will be unbootable until the process has completed and the 64 bit kernel reinstalled.

Update our package list from the base system prepared earlier.

dpkg --clear-selections
dpkg --set-selections <dpkg-get-selections.base

Now remove all the non-essential packages that are not part of the base system. This will take some time to complete. Say No when asked Abort kernel removal?

apt-get dselect-upgrade

Downgrade any packages to the versions downloaded by debootstrap. This will strip updates and security fixes. We must have the same versions of packages in the 32 bit and 64 bit base system to be able to cross-grade them. They will of course be upgraded again later. In my case this process took two passes.

RC=1; PASS=1; while [ $RC -ne 0 ]; do
    dpkg -i bootstrap-i386/var/cache/apt/archives/*.deb
    RC=$?
    echo "Completed pass=$PASS rc=$RC"
    PASS=`expr $PASS + 1`
done

Change the base system's dependency on using dash as the system shell to allow us to crossgrade both bash and dash together without a deadlock. Run the following and answer No when asked Use dash as the default system shell (/bin/sh).

dpkg-reconfigure dash

There is a bug/feature in snapd that caused package failures if not completely purged. If you use snapd, ensure you have an effective method of backing up and restoring before executing the below since this command will completely remove it and purge its package configuration.

dpkg --purge snapd:i386

Finally, install (migrate to) the 64 bit base system. To avoid complex dependency calculations, cross-grade the packages multiple times until all dependencies have been met. If we chose a small base system to download with debootstrap earlier then this step will be significantly faster. In my case this process took five passes.

dpkg --clear-selections
RC=1; PASS=1; while [ $RC -ne 0 ]; do
    dpkg --abort-after=1000 -i bootstrap-amd64/var/cache/apt/archives/*.deb
    RC=$?
    echo "Completed pass=$PASS rc=$RC"
    PASS=`expr $PASS + 1`
done

Now reinstall the kernel and anything else you consider vital, such as ssh, that was not included in the base system.

apt --fix-broken install
apt install linux-generic linux-image-generic intel-microcode amd64-microcode grub-pc lvm2 network-manager openssh-server rsync

Now a 64 bit kernel has been reinstalled, so if you need to reboot you could do so from this point onward.

dpkg -l "linux-image-*"

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version            Architecture       Description
+++-===========================-==================-==================-===========================================================
rc  linux-image-4.15.0-72-lowla 4.15.0-72.81       i386               Linux kernel image for version 4.15.0 on 32 bit x86 SMP
rc  linux-image-4.15.0-88-lowla 4.15.0-88.88       i386               Linux kernel image for version 4.15.0 on 32 bit x86 SMP
ii  linux-image-4.15.0-99-gener 4.15.0-99.100      amd64              Signed kernel image generic
ii  linux-image-generic         4.15.0.99.89       amd64              Generic Linux kernel image
un  linux-image-unsigned-4.15.0                           (no description available)
un  linux-image-unsigned-4.15.0                           (no description available)
un  linux-image-unsigned-4.15.0                           (no description available)

Checkpoint the list of installed packages and then upgrade the base system.

dpkg --get-selections "*:i386"|grep -v deinstall >dpkg-get-selections.cp.01.i386
dpkg --get-selections "*:amd64"|grep -v deinstall >dpkg-get-selections.cp.01.amd64
dpkg --get-selections "*:all"|grep -v deinstall >dpkg-get-selections.cp.01.all

apt autoremove
apt-get dselect-upgrade

Capture another package list checkpoint at the end of this process in case we need to refer back to it.

dpkg --get-selections "*:i386"|grep -v deinstall >dpkg-get-selections.cp.02.i386
dpkg --get-selections "*:amd64"|grep -v deinstall >dpkg-get-selections.cp.02.amd64
dpkg --get-selections "*:all"|grep -v deinstall >dpkg-get-selections.cp.02.all

Revert the base system's dependency to again use dash as the system shell. Answer Yes when asked Use dash as the default system shell (/bin/sh).

dpkg-reconfigure dash

8. Install 64 bit packages

8.1. Install virtual packages (optional)

To speed up the installation progress of the remaining 64 bit packages you may wish to manually install some of them now, especially the larger ones. Performing this step will also allow apt to take care of the dependencies for us.

Install the major pseudo-packages. I also had gnome-shell and texlive installed. Installing as much as possible now will ease the effort later in the process.

apt install ubuntu-desktop ubuntu-minimal ubuntu-standard gnome-shell texlive

Now that the desktop packages are installed the display manager should be running and a base GUI available.

8.2. Install remaining packages automatically (recommended)

Now we can calculate the list of packages outstanding for re-installation and install them. Firstly purge all library packages as they are no longer required and their 32 bit configurations may not be able to co-exist with their 64 bit replacements (as detailed earlier).

apt install aptitude
aptitude keep-all
aptitude forget-new
aptitude search '~c^lib' | wc -l
aptitude purge '~c^lib'

If you wish, take a peak at what is left to convert. No libraries should be remaining.

aptitude search '~c'

Build the final package list of all the packages we wish installed (i.e. the 64 bit versions of the original 32 bit packages and all of the previously-installed architecture-independent packages). Then we can finally remove the old 32 bit architecture.

cat dpkg-get-selections.cp.00.all|grep -v deinstall >dpkg-get-selections.final
cat dpkg-get-selections.cp.00.i386|grep -v ":i386"|grep -v deinstall|cut -f1|awk '{print $1 " install"}' >>dpkg-get-selections.final
cat dpkg-get-selections.cp.00.i386|grep ":i386"|grep -v deinstall|cut -d: -f1|awk '{print $1 " install"}' >>dpkg-get-selections.final
cat dpkg-get-selections.cp.00.amd64|grep ^linux-image|grep -v deinstall|cut -d: -f1|awk '{print $1 " install"}' >>dpkg-get-selections.final

dpkg --clear-selections
dpkg --remove-architecture i386
cat dpkg-get-selections.final|dpkg --set-selections

Reply No when asked Abort kernel removal.

apt-get dselect-upgrade

Finally install the kernel meta-package and cancel all pending actions in aptitude (should it still be installed).

apt install linux-generic
test -x /usr/bin/aptitude && aptitude keep-all

8.3. Install remaining packages manually

Use both of the following commands to find out what is remaining for conversion. When nothing is left we are done with this step and only then can we move onto the next step.

aptitude search '~c'
dpkg --get-selections "*" | grep ":i386" | grep -v deinstall

For each and every package, if you wish to keep it then install the 64 bit version now. For example if pidgin:i386 is shown and we want to keep pidgin and its configuration, then run the following to install the 64 bit version.

apt install pidgin

If you no longer need a package then purge it completely. For example if you are not interested in keeping xscreensaver-gl and its configuration then run the following to purge it.

apt purge xscreensaver-gl:i386

When you are absolutely happy you can purge all remaining 32 bit packages in a single shot. Please be really certain before running this.

dpkg --purge `dpkg --get-selections "*" | grep ":i386" | grep -v deinstall | awk '{print $1}'`

Repeat the above steps as appropriate until no packages remain to be converted. If you have not done so already then you can now remove the 32 bit architecture which is now redundant.

dpkg --remove-architecture i386

9. Restore backups

9.1. PostgreSQL

Restore the PostgreSQL configuration and database cluster data.

cp -p etc/postgresql/10/main/pg_hba.conf /etc/postgresql/10/main
systemctl reload postgresql
xzcat pg_dumpall_10.sql.xz >/tmp/pg_dumpall_10.sql
su - postgres -c "cd /tmp; psql <pg_dumpall_10.sql >psql.out 2>psql.err"
mv /tmp/psql.* .
cat psql.err
rm /tmp/pg_dumpall_10.sql

9.2. OpenLDAP

Restore the OpenLDAP configuration and database.

apt install slapd
systemctl stop slapd
rm /var/lib/ldap/*
rm -R /etc/ldap/slapd.d
cp -a etc/ldap/slapd.d /etc/ldap
slapadd -n1 <slapcat-n1
chown -R openldap:openldap /var/lib/ldap/*
systemctl start slapd

10. Final checks

Re-enable any third party package repositories and install any packages required from these.

Review the /var/lib/dpkg/status file and search for anything installed and still somehow referencing the 32 bit architecture. Optionally remove such packages now. For example below, googleearth-package recommends some 32 bit packages, so remove this package:

awk '{if ($1=="Package:") p=$2} /:i386/ {print p}' /var/lib/dpkg/status
apt purge googleearth-package

Grab a final checkpoint of the packages we have installed. Compare the latest list to the original package list. There should not be anything missing now except perhaps packages that are not available for 64 bit that were available for 32 bit.

dpkg --get-selections "*:i386"|grep -v deinstall >dpkg-get-selections.cp.03.i386
dpkg --get-selections "*:amd64"|grep -v deinstall >dpkg-get-selections.cp.03.amd64
dpkg --get-selections "*:all"|grep -v deinstall >dpkg-get-selections.cp.03.all

sdiff -s dpkg-get-selections.cp.00.i386 dpkg-get-selections.cp.03.amd64|grep "<"

11. Install 32 bit packages (optional)

If you still want to install those old 32 bit packages, then enable the 32 bit architecture and install them as normal.

dpkg --add-architecture i386
apt update
apt install packagename:i386
...