Fundamental Interconnectedness

I have a Brother HL-2270DW laser printer, that I bought in 2014, which is still going fine almost 10 years later. As the article says "buy the Brother laser printer, it's fine".

The Brother HL-2270DW supports (wired) network printing and WiFi printing, but unfortunately its design pre-dated Apple's creation of AirPrint and the printer industry creation of IPP Everywhere (both AirPrint and IPP Everywhere were created around 2010, but took a while to become common in printer designs). Which means (a) it needs a printer driver (brlaser), and (b) while printing from Linux and macOS works fine, printing from a tablet/phone did not work.

Mostly I just ignored the lack of printing from a tablet/phone as I did not have any reason to do it very often, and for those cases I could always re-open the link/document on my desktop to print it.

However a question on the fediverse, about relaying AirPrint to an older printer, got me investigating AirPrint support for older printers again, and I discovered that someone else had got CUPS working to provide an AirPrint wrapper to my exact printer. So I wanted to give it a try.

The process of getting CUPS working as an AirPrint relay seems to have changed a bit since CUPS 2.4.0 so while the guides I found online helped a lot to point me in the right direction, I had to do some debugging after the "and now it should just work" statments in others guides. The Debian CUPSAirPrint page had lots of useful information on how it was supposed to work, but also described an slightly older setup (around Debian Buster/10; whereas I have Debian Bookworm/12).

With Debian Bookworm (aka Debian Linux 12), the CUPS version provided is 2.4.2. And it seems like there are four key steps to getting CUPS working as an AirPrint relay:

1. Install cups and the printer-driver-brlaser (CUPS driver for the Brother laser printers)

2. Make sure that the printer is set up in CUPS as an ipp printer (or maybe ipps or dnssd; ipp with a static IP on the printer worked for me, and Dustin)

3. Ensure that the printer is shared in its printer setup

4. Ensure that printer sharing is enabled in CUPS (ie, system wide configuration to enable the printer sharing features of the printers that should be shared)

With all of those steps, then the CUPS daemon seems to be responsible for advertising the printer into mDNS / DNS-SD (aka Avahi, Bonjour, etc). Which is unlike earlier versions where there was glue logic in the Debian avahi which did this readvertisement.

So, after all the dead ends were eliminated, my process was to install CUPS and the drivers for the Brother HL-2270DW laser printer (brlaser):

sudo apt-get install cups printer-driver-brlaser

then set up the printer for local printing from Linux. In my case I did this via the CUPS web admin interface, as this Debian Linux system was a home server. The web admin interface is available at:

http://localhost:631/

and by default will authenticate with a Unix user/password (possibly in some sudo/admin group) for admin tasks.

Once the printer-driver-brlaser package was installed, there was a Brother section to the printer vendor list, and the HL-2270DW was explicitly listed. Then I could choose default options for the printer, like defaulting to double sided printing. (The printer showed up via DNS-SD in the list of network printers to set up, which I did iniitally, but I ended up explicitly setting it up as an ipp://DNSNAME/ipp/port1 printer when I initially had trouble betting it to show up in AirPrint, and never went back and tried the other options again.)

While setting up the printer (or if you missed it, while "modifying" the printer setup), you can choose to "Share this printer", and you do wnat to choose that for the Brother HL-2270DW that you want to re-share back onto the network.

The final key step to get the AirPrint advertisements from CUPS working is to go to the central admin page of CUPS:

http://localhost:631/admin

and make sure the "Share printers connected to this system" option on the right hand side is ticked. It seems to default to not ticked, which means no printer sharing happens at all (even if you indicated that a specific printer should be shared at the printer level). When you select "Share printers connected to this system" and click on "Change settings" the main CUPS configuration file is rewritten with that change, and the CUPS daemon is restarted.

After those steps, especially the final system wide enabling of CUPS sharing printers, I could see the printer available for AirPrint on my iPad.

The Debian CUPSAirPrint page contains some very useful advice on debugging this setup. In particular:

sudo apt-get install avahi-utils

will install the command line tools for examining what is advertised into mDNS / DNS-SD, and is invaluable for seeing whether the right announcements are being made for AirPrint to work.

Specifically you want to run:

avahi-browse -rt _ipp._tcp

and make sure there is an advertisement that includes image/urf (the Apple Raster Format). If you are not seeing an advertisment with image/urf, then the CUPS printer sharing is not working. In particular you are probably only seeing the Brother HL-2270DW native advertisement of its capabilities, which does not include image/urf (and hence AirPrint does not work).

While debugging this I ended up running;

avahi-browse -rt _ipp._tcp  | grep image/urf

repeatedly, and as soon as I could see an advertisement with image/urf then AirPrint worked (and AirPrint did not work at all before then). (Usually there will be multiple advertisements: one set from both the printer itself and another set from the Debian Linux system with CUPS running. These cover the various service features of the printer.)

You can also install avahi-discover if you have a GUI Linux system, which will provide a graphical view of the same information as avahi-browse finds; but I found it more useful to be able to grep the output of the command line tool.

Unfortuately CUPS is planning to deprecate printer drivers entirely, although what to do about some older printers still seems to be an open question. Which means using CUPS as an AirPrint relay will probably break in a future upgrade :-/ But it might at least be possible to keep an older Linux/CUPS install running as the AirPrint bridge at that point. Or maybe there will be some other application packaging the brlaser driver into an "IPP Everywhere" printer front end, that CUPS will be willing to talk to. (From the DNS-SD advertisements, it also looks like this 2024 CUPS install might be able to act as an IPP Everywhere relay too. So that may be sufficient for the next few years.)

Introduction

The (original) Linux i386 architecture is pretty close to being removed, especially from Debian Linux (but also from the Linux kernel). The last few years have brought several announcements of "this is the last release with i386", only for there to be one more "last release" with i386 as a supported architecture. And various third party software has dropped support for i386, or is in the process of doing so.

Most relevant to me, SaltStack announced a switch to "onedir" packaging about a year ago, from Salt 3005 and especially Salt 3006. The "onedir" packaging means that Salt becomes a binary distribution, bundling its own Python interpreter -- which is obviously architecture specific. This means that my work around of using the "amd64" Salt packages (for older Salt releases) with an i386 Python (and some helper back ported Python modules) will no longer work. So that prompted me to finally convert the last few i386 VMs that were managed by Salt across to amd64 VMs.

The i386 cross grade approach I followed was fairly similar to my earlier test Debian i386 to amd64 crossgrade, but required some tweaks to get it working for Debian Bullseye. Especially, as the Debian CrossGrading Wiki page points out, Debian Bullseye requires perl-base cross graded even earlier in the process, before the rest of the perl packages. And there are some twisty perl / Debian packaging tool dependencies to resolve early on.

This post documents the approach I used on three Debian Bullseye i386 systems to cross grade them to Debian Bullseye amd64. (At the time of this writing Debian Bookworm is the stable Debian Linux; but I decided to do the cross grade on these system before upgrading them to Debian Bookworm, as the Debian Bullseye cross grade challenges were better documented.)

If you are considering attempting this cross grade approach yourself:

• know that you will end up with a broken system in the middle of the cross grade, and it will take care and experience to move forward into a working system again. Key packages you rely on might be auto-removed, or require data or configuration to be recreated to continue working.

• make backups before you start. More than one. Be very sure you can restore from backups. If you are cross grading a VM, consider if you can minimise other writes to disk during the cross grade (eg, firewall the VM off from incoming network connections) and snapshot the VM disk image before you start, so you have an undo option of rolling back to the snapshot and pretending you did not try the risky crossgrade. But either way make sure you have a backup handy you can look at to see "how it was before" to guide you in putting the system back together.

• especially if it is a VM, consider making a clone of the system in a virtual machine, and doing a few dry runs of the upgrade process in a VM you are happy to throw away. That should let you encounter most (but maybe not all) of the problems in a safer environment where you can roll back and try again.

• at least twice during the cross grade you will have to tell the Debian packaging tools that you know what you are asking is a terrible idea, but you want to do it anyway (Yes, do as I say! literally entered). Neither Debian nor I take any responsibility for your broken system as a result of attempting to cross grade it from i386 to amd64 following this or any other guide. You need to assure yourself that you have the experience to carry out this complicated risky procedure through to a safe conclusion.

• be aware that some packages (especially databases) have architecture dependent file formats on disk, which will break horribly if you change architecture. Make sure you have a plan to recover a working system from that cross grade (eg, in the case of databases make sure you know how to dump and reload the database before/after the crossgrade).

My first few trial attempts (in a VM clone) of the cross grade with Debian Bullseye took 1-2 hours to figure out all the required steps. By the time I had reduced each VM clone upgrade to a runsheet of steps specific to that VM install, the production crossgrade took about 30 minutes. But it took at least half a dozen attempts to get a "do these things, in this order" run sheet, that I was happy to attempt in production. Per VM.

If you have some other way to reinstall the system (especially automatically) on a new VM that is 64-bit from the start, that would be a safer option. This risky crossgrade procedure is probably only relevant to long standing pets which are difficult to replace. (All of mine were originally installed 15+ years ago, and have been through many Debian hand upgrades of Debian Linux releases since the install; two of them had been through a physical to virtual conversion too.)

Crossgrade preparation

Make sure your system is capable of running 64-bit code (especially if it is a VM, make sure the VM "hardware" is ready for 64-bit code). The lm flag ("Long Mode") needs to be visible in the CPU flags, eg:

ewen@debian:~$ grep ' lm ' /proc/cpuinfo
flags       : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
ewen@debian:~$

Or check with:

lscpu | grep 64-bit

if you have lscpu installed.

Make sure you have access to the system console (eg, via a serial console or a virtual machine console, or a remote KVM switch -- or maybe by being local to the system). At one point in the upgrade you need to select a specific kernel version, from the grub boot menu, which requires console access.

Finish any package installs, and upgrades on the Debian Linux i386 system, and make sure any earlier upgrades have been tidied up. Eg, dpkg --audit should be clean and dpkg -l | grep -v 'ii' shouldn't show anything left. Any obsolete packages that you cannot reinstall again should be removed. Make sure the system boots cleanly on the Debian Linux i386 install.

Make a backup. That you are confident you can restore from and get a working system again.

Record the package status of the i386 system, for cross reference later on (eg, confirming the same packages got reinstalled, and the same package are marked as "automatic dependencies"):

apt-mark showauto >i386-auto-packages
dpkg --get-selections >i386-package-selections

Switch to dual architectures, and switch to a 64-bit Linux Kernel

Enable the amd64 architecture as a foreign architecture:

dpkg --print-architecture
sudo dpkg --add-architecture amd64
dpkg --print-foreign-architectures
sudo apt-get update

The base architecture at this point should be i386, but you should be able to install amd64 architecture packages by explicitly specifying them at this point.

Then install the 64-bit amd64 kernel, and boot into that:

sudo apt-get install linux-image-amd64:amd64
sudo update-grub
sudo shutdown -r now

For this reboot (and only this reboot) you will have to interrupt the grub automatic boot, and pick a different kernel from the menu. Specifically you need to go into the Advanced options for Debian GNU/Linux menu option (second menu) and pick the amd64 kernel entry which is named something like Debian GNU/Linux, with Linux 5.10.0-26-amd64 and will be the third entry in that submenu (because the existing i386 kernel sorts before the amd64 kernel in the generated list).

Make sure the system booted into a 64-bit kernel:

uname -m

should return x86_64 if you are running a 64-bit kernel. If not, try rebooting again and choosing the correct kernel in the grub boot menu.

Once you have the 64-bit kernel booting, remove the equivalent i386 (32-bit) kernel, and reboot again to make sure the 64-bit kernel is booted automatically from now on:

sudo apt-get purge linux-image-686-pae linux-image-5.10.0-26-686-pae
sudo update-grub
sudo shutdown -r now

If you wish you can pause at this point indefinitely -- the Linux i386 user land should run on the Linux amd64 kernel without any substantial problems, as this was used for a long period in the early x86_64 kernel development.

Change the primary architecture to amd64

This is one of the more complicated steps. At this point we are changing dpkg / apt to the amd64, which changes the primary architecture from i386 to amd64.

On Debian Bullseye, perl-base:i386 conflicts (indirectly, I think) with perl-base:amd64. So we have to force installing perl-base:amd64 at this point. The conflict is sufficiently bad that we cannot even use apt to download packages once dpkg / apt have been changed to amd64. So we need to pre-download several amd64 packages first, then install them "all at once". And then let apt fix up the conflicts any way it can, after we have installed a few more perl packages. (We cannot install those additional packages at the same time as perl-base:amd64 as the conflicts are too complicated to resolve, so apt gives up.)

This combination of steps worked for me on my Debian Bullseye systems (although apt-get -f install removed some packages important to me on each system, which had to be put back later on):

sudo -s
apt-get clean
apt-get --download-only install dpkg:amd64 tar:amd64 apt:amd64 apt-utils:amd64
(cd /var/cache/apt/archives/ && apt download perl-base:amd64)
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --print-architecture              # Should show: amd64
dpkg --print-foreign-architectures     # Should show: i386

Note the two runs of dpkg --install ... for the same packages; this is required because some packages will not install on the first run due to missing amd64 dependencies; but they can install the second time around. (There might be a topological sort of those dependencies to install them in "the right order". But for these critical packages, there are lots of circular dependencies, so simply running the install of these key packages twice is the most pragmatic approach.)

Once perl-base:amd64 is installed, a bunch of other Perl dependencies used for Debian package management will be broken, so some additional perl packages need to be upgraded at this step, including libtext-csv-xs-perl and libencode-perl:

(cd /var/cache/apt/archives/ && apt download libperl5.32:amd64 libgdbm-compat4:amd64 libgdbm6:amd64)
(cd /var/cache/apt/archives/ && apt download perl:amd64)
(cd /var/cache/apt/archives/ && apt download libencode-perl:amd64)
(cd /var/cache/apt/archives/ && apt download libtext-csv-xs-perl:amd64)
(cd /var/cache/apt/archives/ && apt download libdebconfclient0:amd64)
dpkg --install /var/cache/apt/archives/*_amd64.deb

At this point it should be possible to get apt back to a point where it is "happy" automatically, but do keep track of which packages it is removing as some of them will be important to you and need to be reinstalled later on. Those packages need to be removed at this point to break dependency loops:

apt-get -f install
dpkg --configure -a

Switch the shell to amd64

Install the amd64 bash and dash, and run them instead of the 32-bit versions. This will pull in several other key dependencies, and also force-remove the i386 versions, so you will have to agree that you know this can break your system and you want to do it anyway (Yes, do as I say!):

apt-get install dash bash
exec /bin/bash

Install the remaining amd64 replacements for installed i386 packages

Download the packages which are needed to install amd64 replacements:

apt-get --download-only -y --no-install-recommends install \
   `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 | uniq |
    grep -v 'linux-image.*686-pae' |
    sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'`

Then install as many of the amd64 libraries as possible (since these can mostly be installed in parallel with the i386 packages, and fix dependency issues):

dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb
apt-get -f install
dpkg --configure -a

(we also install the outstanding perl packages at this point for certainty).

Then we can install the majority of the remaining amd64 packages:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit | cut -f 1 -d : |
    xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' | uniq |
    tee /tmp/packages-to-install

dpkg --install `cat /tmp/packages-to-install`

which might take a second pass to be able to install all the packages due to dependency issues:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit | cut -f 1 -d : |
    grep -v 'linux-image.*686-pae' |
    xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' | uniq |
    tee /tmp/packages-to-install

test -s /tmp/packages-to-install && dpkg --install `cat /tmp/packages-to-install`

Tidy up the amd64 system

Check what else is left to be converted from i386 to amd64. Some packages will still have i386 versions installed, but will also have an amd64 version -- packages like gcc-9-base and gcc-10-base are effectively library packages and permit parallel installs of both i386 and amd64 versions.

dpkg --get-selections | grep :i386 | grep -v lib | grep -v deinstall | wc -l
dpkg --get-selections | grep :i386 | grep -v lib | grep -v deinstall
dpkg --get-selections | grep :i386 | grep -v lib | awk '$2 ~ /^install$/ { print $1; }' | grep -v 'linux-image.*686-pae' | sed 's/:i386/:amd64/;' | grep -vf /tmp/64-bit

There may still be conflicts at this point, in which case you will need to figure out some way to resolve them for your combination of packages; it might involve removing a package important to you, cross grading other packages, and then reinstalling the packages you wanted on the system once the dependencies have been resolved properly. Inevitably if apt cannot figure out a solution, it will be a complicated problem, so assess carefully what you can live without temporarily to be able to make forward progress.

Once that is done, run a normal upgrade of the amd64 system to make sure apt is happy, and reboot the system to make sure all the running programs are the amd64 versions:

apt-get update
apt-get upgrade
apt-get dist-upgrade
sudo update-grub
sudo shutdown -r now

Assuming the system came back up as a 64-bit system:

uname -m               # Should show: x86_64
getconf LONG_BIT       # Should show: 64

then clean up most of the i386 libraries next:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get --purge autoremove

Then I found if gnuplot was installed, I had to temporarily remove it to make progress on the remaining cleanup (and install it again later):

HAVE_GNUPLOT=$(dpkg -l gnuplot >/dev/null 2>&1 && echo "true")
if [ -n "${HAVE_GNUPLOT}" ]; then sudo apt-get remove gnuplot; fi

before I could remove the essential i386 libraries, which hopefully by this point your system does not need (just the amd64 versions of those libraries). But apt / dpkg does not know this so you will have to confirm you really want to remove these essential libraries (Yes, do as I say!):

dpkg --get-selections | grep :i386 | grep -v deinstall | wc -l
sudo apt-get purge $(dpkg --get-selections | grep :i386 | grep -v deinstall | awk '{print $1;}')

Then you should be able to reinstall any packages that were removed to break dependency loops:

dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/'
sudo apt-get install --no-install-recommends $(dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/')
sudo apt-get -f install
sudo dpkg --configure -a

dpkg --get-selections | grep deinstall
sudo apt-get install $(dpkg --get-selections | awk '/deinstall/ { print $1; }')

Including reinstalling gnuplot again if you had to remove it above:

if [ -n "${HAVE_GNUPLOT}" ]; then sudo apt-get install gnuplot; fi

Then confirm there is nothing else that needs to be reinstalled:

dpkg --get-selections | grep deinstall
dpkg --get-selections | awk '/:i386.*deinstall/ { print $1; }' | sed 's/:i386/:amd64/'

and no i386 packages still installed:

dpkg --get-selections | grep i386
dpkg -l | grep "^i.*:386"

Cleaning up from the crossgrade

Do a regular package update to make sure there are no complaints, and then reboot to make sure you have a working system:

sudo apt-get update
sudo apt-get dist-upgrade
sudo dpkg --configure -a
sudo update-grub
sudo shutdown -r now

Assuming the system booted okay, purge the i386 packages that were removed, and remove the i386 architecture:

sudo apt-get --purge autoremove

sudo dpkg --remove-architecture i386
dpkg --print-architecture               # Should show: amd64
dpkg --print-foreign-architectures      # Should return nothing

Then make sure that dpkg / apt are happy:

dpkg --audit
dpkg -l | grep -v "^ii"

and if the package state is tidy reboot again as a plain amd64 system:

sudo update-grub
sudo shutdown -r now

Ensure your production system is production ready again

Check for any other packages that might have been removed as dependencies and need to be installed, by comparing the i386 package list to the amd64 package list:

dpkg --get-selections >/tmp/amd64-package-selections
diff -bw <(sed 's/:i386/:amd64/;' i386-package-selections | sort) <(sort </tmp/amd64-package-selections) | grep "<" | grep -v "686-pae" | awk '{print $2;}' | grep -v "^lib" | grep -v pkgmonkey-client | tee /tmp/missing-packages
sudo apt-get install $(cat /tmp/missing-packages)

dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep "^>"
sudo apt-get install $(diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep -v "pkgmonkey-client" | awk '/^>/ {print $2;}')

dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep -v "pkgmonkey-client" | grep "^>"

Then check for any extra packages that got dragged in as dependencies during the upgrade which you did not have installed before and do not want, and consider if you want to remove them:

dpkg --get-selections >/tmp/amd64-package-selections
diff -wb /tmp/amd64-package-selections <(sed 's/:i386/:amd64/;' i386-package-selections) | grep -v "linux-image" | grep "^<" | grep -v binutils | grep -v 'lib.san' | grep -v 'libunwind8'

Beware that some of these additional packages will be new amd64 dependencies. Eg, strace on amd64 requires libunwind8, and gcc on amd64 indirectly requires liblsan0 and libtsan0. It will take some systems administration experience, and some investigation, to determine what is still needed for production and what was temporarily installed to meet dependencies during the cross grade.

Once you are happy that the production packages are installed, and no unnecessary packages are installed, restore the "automatically installed" marks so that future upgrades will work better:

sed 's/:i386/:amd64/' i386-auto-packages | grep -v 686-pae | xargs sudo apt-mark auto

(which uses the list of i386 automatically installed packages, created at the beginning).

Then remove any packages which are now no longer required:

sudo apt-get -f install
sudo apt-get --purge autoremove

and confirm you do not have any 386 or 686 packages left:

dpkg -l | grep '[36]86'

Architecture upgrades for specific tools

Some packages have their own architecture dependent files on disk. Particularly databases, but also other programs. You will need to figure out, before committing to the production upgrade, how to handle these situations.

On my mail servers, I found that (a) postfix got removed during the crossgrade (replaced by exim) and had to be reinstalled, and (b) the spamassassin tool had some C libraries that it compiles to do regular expressions faster than in perl, which have to to be rebuilt, with sa-compile.

In the case of spamassassin I did this with:

test -x /usr/bin/sa-compile && (
    sudo sa-compile --list >/tmp/before
    sudo sa-compile
    sudo sa-compile --list >/tmp/after
    diff /tmp/before /tmp/after
    sudo service spamassassin restart
    sudo service spamass-milter restart
    sudo service postfix restart
)

Then once you have a final configuration, reboot the system one more time to make sure you are running your final package configuration:

sudo update-grub
sudo shutdown -r now

And watch your logs and monitoring very carefully for the next few hours to keep an eye out for other problems that need manual fixes.

From this point on, if you successfully got to a working system and fixed any data architecture dependencies, then it should act like any other Debian Linux amd64 install, including for any future upgrades.

Introduction

Twitter was a social media network that operated from March 2006 to July 2023 (later replaced by "X", an almost entirely different social media network). From the late 201x years through late 2022 Twitter was a very useful social media network, often with the most up to date information available especially for current events and IT related tasks. (Then Elon Musk offered to buy Twitter at an unrealistically high value, on a whim; was forced to go through with his unconditional offer, and proceeded to fire most of the Twitter staff, and randomly turn off a lot of the hardware. Twitter was never the same after that.)

I used Twitter read-only actively for several years via the web interface, and eventually created a Twitter account in January 2019, just in time for Linux.Conf.Au 2019. Then I actively used my Twitter account until late 2022, when it became clear that Twitter was no longer a good place.

Along with many many others, I downloaded my Twitter data archive repeatedly in late 2022, with the final download in December 2022. Fortunately I also knew to use Twitter Archive Parser at the time, to fill out details that Twitter's own archive truncated or omitted (eg, expand out t.co links to the full version, full resolution images, etc). So I had a moderately complete archive of my Twitter account. But sitting in a zip file was not particularly accessible.

This week I decided that if Twitter was going to make my account mostly inaccessible without logging in (and more recently rumoured that they would make it inaccessible without a paid "X" account), I should do something myself to make my own posts useful to me again. So I created https://twitter.ewenmcneill.nz/, to put my Twitter account archive online again.

Rehydrating a Twitter Archive Parser archive

To bring the archive back online, in month-by-month pages, with the ability to link to individual Tweets -- but still view some context -- I chose to use the Twitter Archive Parser generated per-month Markdown files, and convert those back into HTML with some usability tweaks.

Assuming you too have a Twitter Archive Parser filled out version of a Twitter archive (ideally with Twitter Archive Parser run in 2022; I am not sure if it will even still work), the steps to put it online were something like:

mkdir /tmp/twitter
cd /tmp/twitter
unzip twitter-data-ewenmcneill-2022-12-18-with-md-and-full-images.zip
rm -r data                    # Remove the JSON files from Twitter
rm -r "Your archive.html"     # Remove top level "start here" page

# Download a copy of the CSS used in the Twitter archive
wget https://unpkg.com/@picocss/pico@latest/css/pico.min.css

# Install markdown_py
brew install python-markdown  # MacOS homebrew

# Translate each Markdown file into a HTML file
for FILE in *.md; do
  YEAR_MONTH=$(echo $FILE | cut -f 1-2 -d -);
  ./twittermd2html "${FILE}" >"${YEAR_MONTH}.html"
done

# Generate an index.html file
for FILE in 2*.html; do
   echo "        <li><a href=\"$FILE\">$FILE</a></li>"; 
done | sed 's/.html</</;' >index.html

vi index.html             # Add the rest of the HTML

and then put that online on a webserver somewhere, generate a TLS certificate for it, etc.

The twittermd2html script is a hacky shell script that I wrote which:

• parses the filename to figure out the month and year

• special cases hashtags starting a line in the Markdown, by injecting and removing a &zwsp; (otherwise Markdown will turn them into headings)

• Runs markdown_py over each file

• Injects a HTML header and footer into each HTML file, which references the local pico.min.css file and includes some local style overrides (mostly reducing excess white space)

• Transforms the HTML to move the "Tweet / datetime" above the Tweet text rather than below it (it seems more readable that way)

• Transforms the HTML to add an acchor tag with the Tweet ID number, and a link (on (#)) to that anchor tag

• Rewrite paragraphs which enclose images to have a CSS class on them, so they can be targetted to have reduced surrounding white space

The result is not perfect by any means, but is fairly readable, and gives me easy access to an anchor link for every rehydrated Tweet.

Among other issues that I have noted, it appears that retweeted Tweets (a) show with the storage RT @USER in them, and (b) seem to be truncated (possibly to the original very short Tweet length), which makes longer retweets less useful than they otherwise might be. (Some shorter retweeted Tweets came through perfectly though. And the date on each rehydrated Tweet links to the original Tweet, in case that still works and/or can be found on, eg, the Internet Archive WayBack Machine.)

There are also a bunch of things (like Twitter usernames and embedded links) that ideally would be turned back into links, and some formatting updates I would probably do for presentation purposes. But none of them are sufficiently urgent to spend more time on them now.

But the result -- at https://twitter.ewenmcneill.nz/ -- is much more useful to me than a zip file sitting on a hard drive. Especially since I used to mainly use Twitter as an alternative for a bookmarking service :-)