Fundamental Interconnectedness

This is the occasional blog of Ewen McNeill. It is also available on LiveJournal as ewen_mcneill, and Dreamwidth as ewen_mcneill_feed.

Introduction

Somewhere in the last 30 years, building a CPU went from being something only an elite few could do to something anyone with a bit of free time could do. These days we are pretty much measuring in "CPUs per square millimetre", people compete for how many CPU cores they can fit in a FPGA (hint: thousands, with the CoreScore benchmark), and there are games taking you from relays to programming languages (inspired by courses and books taking you from NAND to Tetris).

The Zilog Z80 CPU in my first computer seemed magical technology at the time, but last year I implemented (a subset) of a Z80 CPU in Python. As but one part of an extended Rube Goldberg Machine.

The Challenge

One of the great things to come out of the 2020 Covid-19 lockdowns were entirely online conferences like ComfyCon, and PyConlineAU. These looked for ways of having "audience participation" while being entirely remote: ComfyCon had active sharing of pet pictures, home labs, etc; and PyConlineAU 2020 had an entire day -- Sunday -- dedicated to social events.

The PyConlineAU 2020 "Rube Codeberg" competition came out on the Sunday "audience participation" day, when Lilly Ryan challenged attendees to "unnecessarily elaborately" engineer a piece of Python code which:

  • was written in Python 3, up to 200 lines

  • ran successful to print out "Hello World!" somehow

  • used the Beautiful Soup Python library

  • was "understandable/easy to parse by humans", and yet ridiculously over engineered a la Rube Goldberg machines.

I did not end up writing something on the conference Sunday, as there were many other things to do that day, but since I had a great idea for what I "should have written", the following weekend I sat down for a few hours and actually implemented my solution, in Python 3.

In addition to the conference rules, I also:

My solution

My brain decided the best way to combine all of this was to implement the program to print "Hello World!" in another language, and then implement a Python 3 program that could execute that other language implementation. Obviously 200 lines of "easy to parse by humans" Python code is not very much to implement an entire language in, so it was clear the best way to proceed was to implement something which was drive by byte codes. I briefly considered transpiling the language into Python byte codes, but I did not know the Python byte code implementation... which is when I realised the byte code implementation I knew best was the Z80 CPU. (I spent several years writing Z80 assembler, in the late 1980s and early 1990s, including a terminal emulation program.)

So that is how I ended up implementing (part of) a Z80 CPU, in Python 3, as a side quest of another project.

Obviously if you run the program it prints out "Hello World!" (assuming you have the required Beautiful Soup module installed from PyPI):

$ python3 hello
Hello World!
$

but the Python program does not contain that text anywhere within it, and the output takes a non-trivial amount of time to arrive:

$ time python3 hello
Hello World!

real   0m0.242s
user   0m0.199s
sys    0m0.032s
$

A quarter of a second might not seem long by historical standards of the Z80, but it is eons of time for a modern CPU.

One gets a hint at how much additional hidden depth is there by asking for --help:

$ python3 hello --help
usage: hello [-h] [--asm] [--bin BIN] [--html]

optional arguments:
  -h, --help  show this help message and exit
  --asm       Output assembly
  --bin BIN   Output binary to file
  --html      Output HTML
$

The assembly option gives you Z80 source code that can be assembled and run on a Z80 system:

$ python3 hello --asm
.ORG    0x100
        LD HL,text
        LD DE,key
decode: LD A,(HL)
        OR A
        JR Z,print
        LD B,A
        LD A,(DE)
        XOR B
        LD (HL),A
        INC DE
        INC HL
        JR decode
print:  LD HL,text
pc:     LD A,(HL)
        OR A
        JR Z, done
        LD E,A
        LD C,2
        PUSH HL
        CALL 5
        POP HL
        INC HL
        JR pc
done:   RST 0

text:   DB 0x18, 0x1c, 0x2f, 0x03, 0x01, 0x4c, 0x3e

        DB 0x01, 0x17, 0x4c, 0x56, 0x11, 0x3f, 0x3a

        DB 00
key:    DB 0x50, 0x79, 0x43, 0x6f, 0x6e, 0x6c, 0x69

        DB 0x6e, 0x65, 0x20, 0x32, 0x30, 0x32, 0x30

        DB 00
$

in particular a Z80 CP/M system. But you do not actually have to do that because the program can also output the Z80 binary for you, ready to run with the --bin OUTPUT.com option:

$ python3 hello --bin hello.com
$ ls -l hello.com
-rw-rw-r-- 1 ewen ewen 143 Sep 12  2020 hello.com
$ ../tools/cpm/cpm hello


Hello World!
$

(with the extra two newlines due to the CP/M emulator used for testing).

And finally the --html option prints out HTML containing the motivating text, and the assembler output listing. (One might hope there was a webserver providing that HTML, but alas I massively ran out of lines for Python, so the HTML is a little BYO browser.)

The Implementation

The Z80 CP/M "Hello World" program

The first part of the implementation was that I needed a Z80 program that would print "Hello World!", itself in a way that was non-obvious: in particular I wanted it not to contain the literal text as either a raw text string or in hexadecimal bytes. So there is both the "text" string and a "key" string, both zero terminated, that between the encode the bytes to print out. The "decode" section of the assembly simply walks the text and key bytes, and XORs them together, forming a simple one time pad. Then the print section of the assembly walks of the decoded text, and prints it out. In this case using the CP/M BDOS conventions, where the C register contains the function, the E (or DE) register contains the parameter, and you CALL 5 to access the BDOS functions. (Using the CP/M BDOS conventions was an obvious choice both because I was familiar with them, and they are fairly compact, but also because it enabled testing the Z80 program independently, as shown above.)

The text and key were themselves generated with a small Python script:

#! /usr/bin/env python3
# XOR string A with string B, and print result as hex
#
# Written by Ewen McNeill , 2020-09-12
#---------------------------------------------------------------------------

import sys

def xor(text, key):
    return [(ord(t) ^ ord(k)) for t, k in zip(text, key)]

def main():
    if (len(sys.argv) <= 2):
        print("Usage: {0} TEXT KEY".format(sys.argv[0]), file=sys.stderr)
        sys.exit(1)

    text = sys.argv[1]
    hexord = lambda x: hex(ord(x))
    key  = sys.argv[2]

    # Bourne style shells remove newlines in command line parameters during
    # command line substitution, so we may have to restore it to make our
    # plain text have the same  length as the key
    #
    # https://unix.stackexchange.com/questions/383217/shell-keep-trailing-newlines-n-in-command-substitution
    #
    if (len(text) == (len(key) - 1)):
        text += "\n"

    if (len(text) != len(key)):
        print("Plaintext and key must be the same length! ({0} != {1})".
                 format(len(text), len(key)), file=sys.stderr)
        sys.exit(1)

    print(", ".join(map(hex, xor(text, key))))  

if __name__ == "__main__":
    main()

and for this challenge I obviously chose the required "Hello World!" text, and "PyConline 2020" as an equal length string to be the one time pad (since it is xor, the pad itself is visible in the assembly output, but obscured by being specified as hex digits).

$ cat gentext
#! /bin/sh
./xor "$(printf 'Hello World!\r\n')" "PyConline 2020"
$ sh gentext
0x18, 0x1c, 0x2f, 0x3, 0x1, 0x4c, 0x3e, 0x1, 0x17, 0x4c, 0x56, 0x11, 0x3f, 0x3a
$

and then I translated those into Z80 Assembler formatting, and put it with the rest of the program I had written.

I assembled the initial program with the zasm -- Z80 Online Assembler for expedience, to avoid yak shaving a Z80 assembler locally as well. Which gave me the assembly listing included in my solution.

As shown above, that program prints out "Hello World!" when run under a Z80 emulator that understands CP/M BDOS printing conventions.

So I needed a Z80 emulator that understood CP/M BDOS printing conventions, written in Python.

The (partial) Z80 CPU implementation in Python

The Z80 emulator is (slightly hidden) in the hello() function of my implementation. Due to both time, and especially space, constraints it implements just enough of the Z80 CPU to be able to run the example program. The entire implementation is just a "forever" loop, that executes one instruction per clock tick (not 100% realistic, as the Z80 instructions were variable length from 4 clock ticks upwards, but time and space constraints limited the accuracy :-) ).

Each instruction is handled by a Switch/Case implementation in Python, inspired by Christopher Neugebauer's talk and blog post, for which the Switch/Case implementation can be seen near the top of the program, and the individual implementations are compactly implemented in the hello() function. Since the cases all needed distinct names (a limitation of the Switch/Case implementation), they are named after the hexcode of the Z80 opcode they are implementing, as that is both short and unique :-)

The majority of the implementations are trivial implementation of a "Z80 State Machine" (as much as was required to execute the example program), part from hcd() (for CALL, addr) which is special cased just to handle the one CP/M BDOS printing routine required, and hc7() (for RST 0) which I used as a "Z80 flavoured" end of program sentinel, and simply leaves exits the Python program. There are also a couple of helper functions to reduce the length of some of the implementations. The PUSH / POP implementations are skipped for space reasons, as they turn out not to be relevant since our CALL 5 implementation does not corrupt the HL variable, so we do not need special steps to preserve it (but they are present in the original Z80 assembly, as the CP/M BDOS does corrupt HL).

It turned out to be important that the newline after the Python function name is optional, in order to save some of the value 200 lines allowed, so all the short implementations are inline. But it also turned out to be important I could support multi-line case statements, an unstated requirement of the original implementation. Particularly LD C, immediate and JR dest / JR condition dest needed multiple line implementations.

Putting it all together

At this point I had a Z80 (CP/M) program that would print the required text, and a (partial) Z80 emulator written in Python that would execute that Z80 program, and produce the output required. The remaining step was to bring the two together, and ensure that I used the required Beautiful Soup module.

Since I wanted to somewhat hide the Z80 program inside the Python program, to make it stand alone, and not immediately obviously connected, I came up with the idea of hiding the Z80 assembly listing output in a Python docstring at the start of the program. From which the obvious thing to do was to bury it in a (trivial, space constrained) HTML page which made fun of how "difficult" it was to do anything in Z80 assembler, and provided a nice counterpoint to the "understandable/easy to parse by humans" Python program that followed.

Using HTML, gave me a direct reason to use the Python Beautiful Soup module as required, which extracts the HTML, and assembly listing, without which the whole program would not work. From that assembly listing, we can get both the assembly input, and the Z80 opcodes for the program, give certain assumptions about the assembly listing output. (Fortunately it turns out the many 1970s/1980s derived output formats were very column aligned, thanks to a punch card / line printer influence!)

Which just leaves the top level program to parse the arguments and dispatch to the right function which is not in main() method, because I ran out of lines (199 out of 200!). It too uses a Switch/Case implementation largely because it was there, and a fairly compact solution.

In the default run:

  • The hello() function (Z80 CPU partial implementation) is invoked on the output of get_bin() (which retrieves the Z80 opcodes from the assembly listing extracted from the Python docstring with Beautiful Soup)

  • The hello() function then steps through the Z80 implementation to decode the string to be printed out in the text section of RAM, and then

  • The hello() function steps through the Z80 implementation to print out the decoded Hello World! string, character by character

And hey presto, "Hello World!" is printed as required, simple as that. Taking only a quarter of a second, about 25x longer than a trivial implementation in Python:

$ time python -c 'print("Hello World!")'
Hello World!

real   0m0.012s
user   0m0.004s
sys    0m0.008s
$

(To be honest I am surprised it is only 25x slower, considering it has both an inefficient Switch/Case emulation, and an inefficient Z80 emulation to contend with!)

Most of the other run options also use Beautiful Soup, to get the assembly listing, with the exception of --html which simply prints out the whole docstring "as is".

Despite various steps to reclaim lines of text as I was running up against the line limit, the program still feels surprisingly readable over a year later.

The original competition did not actually have a line width limit, and Python does allow stacking multiple operations on a single line separated by ; -- semicolons -- but I tried to avoid that except where required to reach the line limit. And to stick to 80 column lines for readability. Spending 50 lines on the "alternative" implementation in Z80 Assembler, and another 20 lines on the Switch/Case implementation, did not exactly help my line count constraints, but despite that I still managed to retain another 33 entirely blank lines.

You can read my whole implementation on GitHub, and that also has links to all the other Rube Codeberg submissions I could find.

Conclusion

It was a fun program to write, and a nice distraction from the world outside in 2020. I was particularly surprised just how simple it was to implement a usable fraction of a Z80 CPU state machine in just a few hours, in a language entirely unsuited to state machine implementations. And it was a lovely loop back to one of the first open source projects I was involved with, "Libero", a Finite State Machine generator from iMatix.

It really is state machines all the way down :-)

Posted Thu Dec 30 16:20:08 2021 Tags:

Covid-19 Vaccine Passes

Late last week (2021-12-03) New Zealand moved from its 2020 "Alert Levels" system (which were constantly tweaked), to the to the Covid-19 Protection Framework, which rapidly got nicknamed the "Traffic Lights" system (archived copy), due to the poor choice of initial colours for regions (Red / Orange / Green; but "Red" means "stop" less than Alert Level 3/4 did, and "Orange" does not mean "prepare to stop", and "Green" does exactly not mean "proceed" either; perhaps they should have chosen different colours....).

The main feature of the Covid-19 Protection Framework is that basically all businesses can operate at all levels (with some capacity limits in "Red") if they choose to check "My Vaccine Pass" (archived copy) and ensure that both staff and visitors are fully vaccinated for Covid-19.

This naturally resulted in a lot of New Zealand businesses choosing to require Vaccine Passes for entry, including many hospitality and entertainment businesses (the core aim of the Vaccine Pass system) as well as many other businesses (eg, I have had "Vaccine Pass required" notifications from Data Centres, Research organisations, etc). The only businesses explicitly not allowed to require Vaccine Passes are "essential retail" (supermarkets and pharmacies, basically); other retail does not have the same number limits as hospitality/entertainment so does not need to check Vaccine Passes, but several of them are choosing to do so anyway.

Which means most vaccinated New Zealanders wanting to have a "normal life" will need to have their Vaccine Pass close at hand to show regularly.

For those with current vaccination status, Internet access, and an electronically verifiable New Zealand Government issued ID (basically NZ Passport or NZ Driver's License), the process to get a "My Vaccine Pass" online at "My Covid Record is fairly simple: create an account with an email address, verify the email address, connect the account to your "real world" identity by entering details from your NZ Government issued ID, and then request your Vaccine Pass to emailed to you. (Those without current NZ Government issued ID have a much harder time and might need assistance via the phone helpline and/or an in person visit somewhere like a Covid-19 vaccinating pharmacy; that part of the roll out seems to have been less well handled.)

What arrives in the Vaccine Pass email is:

  • an A4 sized PDF to print out, and cut out a "wallet sized" Vaccine Pass square from the middle of it

  • a link to "add my Vaccine Pass to Apple Wallet"

  • a link to "add my Vaccine Pass to Google Pay" (ie, on Android)

all of which contain a QR code that is the actual Vaccine Pass, and some printed text around it to help recognise whose Vaccine Pass it is supposed to be (the authoritative version of those details is included in the QR code, which is signed; see the NZ Covid Pass specification for details on how the signing works). (There's also an official app to verify the Vaccine Pass QR codes, although weirdly use of it is optional, which is disappointing as the entire Vaccine Pass is the QR code and that is not human verifiable; the surrounding text is just metadata to help identify whose pass you are looking for if you are holding several, eg for your whole family.)

Unfortunately the critical QR code is presented very small (about 24mm x 24mm on paper, and 19mm x 19mm in the Apple Wallet version on my phone), and contains a lot of detail (81 x 81 QR code in about 20mm square, so around 4 dots per millimeter!), which is likely to make it more difficult to scan in non-ideal conditions.

Vaccine Pass on an Apple iPhone

Since it was likely that I'd quickly need access to my Vaccine Pass for an extended period of time (it seems very likely to be needed all of 2022 for instance), I wanted to make it easier to access. The paper copy is in my wallet, but I'd prefer not to be getting that out all the time, so the obvious thing was to make the Vaccine Pass more accessible on my phone too (since I already need my phone out to scan the Covid-19 Tracing QR codes (archived copy)).

The obvious thing to do was to add my Vaccine Pass into Apple Wallet on my iPhone, by clicking on the link in the Vaccine Pass email, which I did, but (a) it is then buried below Apple's solicitation for me to give me my credit card details (for Apple Pay), and (b) the resulting QR code is quite small and might not scan well.

My next step was to:

  • screenshot the Vaccine Pass from Apple Wallet, and

  • screenshot the Vaccine Pass QR code from the PDF expanded as much as possible to fill the screen (fortunately it is a vector graphic so scales up well), to give the largest possible QR code to scan for easiest scanning

and then create a "Vaccine Pass" album in the "Photos" app to put those two in, to make them easier to find.

That helped, but finding a specific "Photos" album quickly is also non trivial, so I went looking for a more streamlined solution.

Quick access to My Vaccine Pass on an Apple iPhone

iOS 14 and 15 come with the Apple Shortcuts app, which allows automating multiple steps on the phone, and optionally saving those to the Home Screen behind a "shortcut" icon or to a "Shortcuts Widget".

In particular one Shortcut task is to Find Photos, and another allows you to preview a file, which provides a convenient way to see the results without having to dive into an application.

With some hints on the Internet I came up with a "Vaccine Pass" shortcut, with two action steps:

  1. Apps: Photos: Find Photos: Find all Photos where Album is "Vaccine Pass", sort by Date Taken, Latest First (which I limited to 2 items, as those are the two I care about, and it allows me to add replacement versions into that Photos album as they arrive, without changing anything else)

  2. Documents: Show Photos in Quick Look

and then when that is run, it displays a "quick look" (ie, pop over window) with my Vaccine Pass in both versions, as the screenshots I took earlier.

To make it easier to find you can choose to "Add to Home Screen" and give it a name/icon photo that you will recognise; I called mine "Vaccine Pass", and used the Vaccine Pass screenshot itself as the icon photo as that was fairly recognisable. Then I shuffled that to the front screen of my Home Page to make it easier to find (displacing something I was not using).

Somewhat annoyingly, the Home Screen version pops up an annoying "look at me, I did a thing for you" banner on every access (reminding you it used Photos for you). There is apparently no way to turn this off (thanks Apple, what were you thinking?!), although some people have managed to get a work around involving Screen Time blocking Shortcuts Banner notifications to work (it involves getting Screen Time to recognise Shortcuts is notifying you, and then blocking those interruptions; I never managed to get Screen Time to recognise the notifications were happening, so I could not block them).

While researching blocking the annoying Shortcuts banner notification I discovered that if you run the Shortcut from a "Widget", then you get an animation/tick in the Widget instead of a banner notification taking up the top half of the screen. That seemed like an improvement... but sadly the minimum size of a "Widget" is 2 x 2 icons on the Home Screen, which was a lot of space to give up just to avoid annoying notifications.

Fortunately I realised that I could combine another annoying iOS feature and get something that cancelled out both annoyances: the View Widgets screen, available by swiping right even from the iPhone Lock Screen, contains only widgets (no smaller icons), and I do not use it for anything else, so it was a good location to stash a "Shortcuts Widget" to access my Vaccine Pass.

To set up the Shortcuts Widget on the "View Widgets" screen, on an unlocked phone swipe right to get into the "View Widgets" screen (with the large Widget blocks, including calendar, weather, etc by default), and then all the way to the bottom of that to find the "Edit" button. Clock on the Edit button then the "+" at the top right (to add a new Widget), scroll down the list of options to find "Shortcuts", choose the "1 Shortcut" view (unless you have more you want available), and "Add Widget". Then painfully shuffle the new widget (added at the bottom), up to the top of the "View Widgets" screen so it is actually accessible.

The displayed Shortcut in the Shortcuts Widget uses the name/icon taken from the Shortcut app itself (not the one for the Home Screen), so you may wish to edit the Shortcut to change the colour / icon / name used to make it easier to identify.

(If you can justify the space for a 2 x Shortcuts widget, one possibility would be to create a shortcut that just opens the NZ Covid Tracer app, so that too is accessible from the Lock Screen. For now I have not done that, as the NZ Covid Tracer app has lived on my default Home Screen for months in a very "thumb friendly" location, and by default opens "ready to use".)

Conclusion

The result of all of this is that from the lock screen of my phone (or the left most Home Screen), I can swipe right to get to the "View Widgets" screen, and then there is a "Vaccine Pass" widget right there, which I can run by clicking on it, which will display my Vaccine Pass in two forms ("Apple Wallet" view, and "maximised QR code) that I can swipe between. And using it does not pop up a Banner notification on every use, just a "running" animation and tick.

If accessed from a locked phone, then there is an authentication prompt after you click on it, but it displays immediately the phone authentication succeeds. Which seems like the best "quick access" possible.

Hopefully I now never need to deal with the tiny QR code on the printed paper Vaccine Pass PDF, and it just lives in my wallet as an "emergency backup".

ETA, 2021-12-06: Since it seems to be a FAQ: yes, Apple Pay has a feature you can enable to bring up the Apple Pay Wallet for payments from the lock screen (Settings -> Wallet & Apple Pay -> Allow Access When Locked: Double-Click Home Button / Double-Click Side Button, depends on phone model). And as a side effect of that, if you have Apple Pay configured and that feature enabled, you can use that to find other things in your Apple Wallet, like your Vaccine Pass, from the Lock Screen. If you have Apple Pay configured, that might be as convenient (although you may have to hunt for the Vaccine Pass in the wallet each time).

Unfortunately because that is an Apple Pay feature, it only works if you have given a credit card to Apple to use with Apple Pay; it does not work with Apple Wallet alone (and in fact these days Apple dedicates the whole first page of Apple Wallet to an encouragement to give them a credit card for Apple Pay, making Apple Wallet less useful than it was in previous iOS versions -- ie, you can no longer have a "pass you need to use soon" on the top/front screen when Apple Wallet opens). I have tried quite hard not to give Apple (or anyone else, as much as possible) a credit card they can charge whenever they decide to; so I am very reluctant to add a credit card to Apple Pay just to unlock the "double tap" to open Apple Wallet functionality that Apple choose to hold hostage to providing a credit card.

@jme_nz suggested Accessibilty Zoom as a way to zoom in on the small QR code in the NZ Vaccine Pass Apple Wallet entry, which is a great idea. (You need to enable Accessibility Zoom first: Settings -> Accessibility -> Zoom; then double tap the screen with three fingers to get into the zoom feature. See also more detailed walk through of Accessibility Zoom feature.) I suspect my "Vaccine Pass" pre-zoomed in screenshot is a bit easier to use (just swipe left to move to that entry in te album), but it does require more setup in advance than just turning on the Accessibility Zoom feature.

Posted Mon Dec 6 18:57:45 2021 Tags:

Introduction

New Zealand has been in a Covid-19 "Level 4" lockdown for most of the past week, after a Covid-19 case was found in the community, announced on Tuesday 2021-08-17 (archive), which was later confirmed to be the "Delta" variant of Covid-19, linked to the NSW, Australia cluster by genome sequencing.

So it seems an appropriate time to finally tell the tale of last year's Level 4 lockdown (archive), when I needed to get an expanding laptop battery replaced during the Level 4 lockdown. Just before Easter. Less than two weeks into the Lockdown, so the rules were frequently changing and everyone was trying to figure out what it meant. It was "Interesting Times".

Telling this tale is somewhat inspired by a friend's story of a marathon drive to the US/Canada border (archive) early in the USA and Canadian Covid-19 lockdowns, to visit one of their mother's who was terminally ill, before she passed. Interesting Times (tm).

Background on the laptop

I have a Late 2013 model Apple MacBook Pro 15", bought in early 2014. Those models are sufficiently old to not be covered by the Apple MacBook Pro 15" Battery Recall, for models sold a year or two later (2015-2017), but seemingly new enough to possibly be prone to battery issues, as I have subsequently heard of a couple of others with similar models and expanding battery issues.

While on site at a client in mid March 2020, the week before the New Zealand Covid-19 lockdown was announced, I happened to notice that the laptop touchpad buttons were getting difficult to press making me wonder if the touchpad was failing (I mostly use an external keyboard/mouse in my office, so had not noticed earlier). At that stage I simply hoped it would not get worse, as Apple delivering several terrible laptop keyboards had discouraged me from replacing the laptop at a more usual interval (eg, 5 years after purchase).

Then a week later, New Zealand went into "Level 4 Lockdown" (archive), and my week passed in a huge rush assisting family members to be able to work at home, and clients to cope with all their staff suddenly working at home even before they had had time to test that scenario. Oh, and one client had a PostgreSQL performance emergency as well, so I spent several days debugging their slow performing queries and building indexes to optimise the pessimal queries their business intelligence tool was producing. The week went by in a rush.

Unfortunately by early April 2020 (archive) I noticed the laptop, back on my office desk, was no longer sitting completely flat on the table. There was a very slight wobble to it as either one or the other of the front feet could be on the table, but not both. The gap on the foot not on the table was maybe the size of a single piece of paper. But clearly something was changing.

When I tweeted about it I got a number of replies from people I knew... strongly encouraging me to get the battery out of the laptop. Which is non-trivial in unibody Apple MacBooks, as the battery is glued inside the case. And I did not have the special Apple specific security bit to open the laptop. Which also were not available for sale, due to the Level 4 Covid-19 lockdown (Level 4 lockdown rules only allowed "essential goods" to be purchased -- that apparently included laptop battery spares, but not the tools to install them :-/ ).

That was Friday 2020-04-03, a week before Easter Good Friday (a four day weekend in New Zealand).

Battery replacement required

Over the weekend, after a lot of thought about this laptop problem -- including trying to figure out how long Level 4 lockdown restrictions might last, and how long the battery might stay stable if I avoided charging or discharging it -- I ended up deciding on Monday 2020-04-06 that I was going to try to get the laptop battery replaced.

My first attempt was to try to call the two official Apple Authorised repair places in my city. After a considerable amount of time on hold, both of them eventually told me that they were only doing repairs for "essential businesses" computer equipment, and even then on a limited basis. While I could possibly have qualified as an essential business (the rules allowed for suppliers to essential businesses to be declared essential businesses, and some of my clients were definitely on the essential businesses list) that did not seem a very productive approach. Particularly since the official Apple replacement is a model specific battery/touchpad/keyboard/top case replacement, which may or may not have been in stock for my specific model.

Then just after noon I emailed The Apple Guy, to ask if they could assist, because (a) I had seen their stores around my city for some years (so at least knew they were more than just a website), and (b) they appeared to be the only Apple computer servicing location actually doing repairs of non "essential businesses" equipment at the time.

I got an (email) quote from The Apple Guy an hour later, for what seemed like a reasonable figure, for "courier both ways" service -- the only service option they were able to offer during the Level 4 lockdown. I told them I would let them know the next day (Tuesday 2020-04-07) whether I wanted to go ahead. Computer servicing just before Easter in an ordinary year in New Zealand was something likely to drag on for well over a week due to the public holidays. And 2020 was no ordinary year.

After some though, later that afternoon I made two full backups of my laptop, in addition to the ones I already had. And then discharged the laptop battery. At which point I was committed. Because there was no way I was going to take an expanding laptop battery through a full discharge/charge cycle, so I could neither use it from battery nor run it off the charger (as that would charge the battery).

I emailed The Apple Guy late Monday 2020-04-06 to confirm I did want to go ahead.

Courier pickup

I got a reply from The Apple Guy first thing the next morning, with instructions for coordinating the repair:

  • Package it well for transport

  • Address it to the home address of the person doing the repair (all stores were closed, due to the Level 4 Lockdown)

  • Include return details

  • Provide the password for the device, so they could test it

plus an invoice to be paid "before the device was returned".

I paid the invoice immediately, by online banking: if you want a supplier to think of your work favourably paying promptly always helps, and the repair total was considerably less than the cost of replacing the laptop.

The password was more of a struggle. After a little thought I concluded what they really needed was a password which could boot up the laptop, and test it, not actually my password. So I created a temporary service account, and sent them a photograph including that password (which seemed marginally better than including it as plain text; I could maybe have called them with the password but (a) they were already very busy, and (b) chances are they'd have misplaced the note with the password by the time they needed it). I even put a sticker inside the laptop with the temporary password, just in case.

So then I just needed to finish discharging the battery (down to a few percent, so it did not take long), and pack it for courier delivery. Fortunately my policy of keeping all potentially useful boxes and shipping material paid off, and I was able to find one sturdy box a little larger than the laptop, and lots of bubble wrap. So after wrapping the laptop in three layers of bubble wrap, I sealed it into the box along with a note taped to it with the repair needed and the return address details.

Their first email of the day said the "the courier will come to collect your device either today or the next working day.". So I read that as either Tuesday 2020-04-07 or Wednesday 2021-04-08.

No courier came on Tuesday 2020-04-07. In the middle of the night, unable to sleep from stress, I changed my mind about having a password for the laptop on a sticker on the laptop, and got up, unpacked the laptop entirely, removed the sticker, and packed it all up again, sans sticker. Relying on the emailed photo of the password to be sufficient (which it turned out to be).

Wednesday 2021-04-08, I needed to go shopping before the 4 day public holiday (when even the few shops open during the Level 4 Covid-19 lockdown would also be closed), and tried to figure out whether to wait for the courier pickup or shop first. I ended up calling the courier company directly, as I had a pickup reference from The Apple Guy, to try to find out when it might be picked up. The courier company could not tell me, but did offer to amend the pick up instructions -- so I decided I would just leave it on the back doorstep, and go out, and had the courier company update their instructions to pick up from the back doorstep.

Then I contacted the two nearest neighbours and asked them to keep an eye out for lost couriers and direct them to the package, and start getting ready to go out shopping -- an hour plus round trip, given the Level 4 queuing to get into the shops. None of it seemed particularly safe, but unusual circumstances call for unusual tactics. And I had already pretty much resigned myself to the laptop possibly not ever coming back (or at least not coming back working).

Just as I was walking out the back door to get in my car to go shopping, I was met by someone in a courier uniform, coming to pick up a package. It turned out that my phone call had prompted their pick up after all, not on their regular pick up route but because they happened to already be delivering near by.

What followed was one of the more surreal conversations of the whole experience. I was giving them a package with no courier ticket on it, and no other courier information on it (just destination and return addresses, as it was a supplier organised pickup -- and I was given shipping information hours before the supplier had any courier pickup reference). The courier asked if I was paying for the courier, and I said no, it was being paid for by the supplier. Of course I did not know the supplier's courier account number either.

After about a minute's discussion, we agreed that I would find the courier pickup reference number, write that on the package by hand, and the courier would take the package and put a courier tracking ticket on it. I never did find out what that tracking ticket was, so I had no way to track the parcel. The courier left. I went shopping. It took about an hour.

At this point I was even less certain the laptop would ever return, let alone return working. And I was certain it would not return before Easter (spoiler: it did not return before Easter).

Battery repair

Early afternoon of Thursday 2020-04-09 I got an email from The Apple Guy saying only "I received your MacBook just now". So the courier process worked. Overnight, across town, delivery, in a Level 4 pandemic lockdown, with just a promise that the supplier who organised the pick up would pay for the delivery.

Early afternoon Saturday 2020-04-11 I got an email from The Apple Guy saying they were replacing the battery, and had noticed one of the speakers was blown, quoting another $100 to replace the speaker. I do not know if the speaker was blown (I had not noticed it before), but (a) it was clearly going to be cheaper to replace while the laptop was open already, (b) even if it was damaged during shipping/repair that would be impossible to prove either way, and (c) even if it was a "value add" service the amount was sufficiently small that it was not worth arguing about. So I paid that $100 immediately, and said "yes please" (they had them in stock). I tend to believe the speaker was probably damaged earlier -- it was a 6 year old laptop by that point -- but there is really no way to be certain.

A little later that afternoon, my VPN server noticed my laptop connect to the Internet/VPN. So at least I knew it was alive again. (From examination of the files on the laptop after I got it back, it seems the battery calibration procedure is: charge battery to 100%, boot laptop, turn screen brightness up to maximum, run program that plays a video streamed over WiFi in a loop, disconnect charger and let the laptop play video until it turns off due to a flat battery. So it was connected to the VPN for quite a while, during the test.)

Monday 2020-04-13, two days later, I got an email from The Apple Guy saying the laptop was ready and would be couriered back the next day (Easter Monday is a public holiday in New Zealand). Wednesday morning, 2020-04-15, I got a tracking link for the return delivery.

Then at 10:30 on Wednesday 2020-04-15 the laptop was back in my hands. Almost exactly a week after it left my house. And about nine days after I stopped using it.

Aftermath

One of the first things I noticed when I got the laptop back was that "About this Mac" was reporting "Serial number: Unavailable". Which was causing some serial number locked software not to run, including Alfred which I use all the time.

From a bunch of research I found out the older Apple MacBooks stored their serial number in the firmware ROM, embedded near the end of the firmware image (it is replaced into the ROM image as it is programmed). The "hwc" (model information) and "ssn" (serial number) values are initially set by the Apple internal "Blank Board Serialiser", and then copied from firmware image to firmware image each time the firmware is updated. And it turned out that my MacBook Pro had indeed forgotten its exact model information too (it now shows "MacBook Pro" instead of a precise model).

I did ask The Apple Guy about this, but they had not encountered the serial number disappearing during the repair before. Some Macs definitely had no serial number information, but it was either there or not there. Despite speculation on Twitter that the repair place had changed the mainboard I do not believe that was what happened: everything else about the hardware was identical to what I sent, it was just the "hwc"/"ssn" information that had gone missing.

As best I can tell the most likely explanation is that an earlier, possibly much earlier, firmware upgrade had been interrupted, and that had caused the firmware image to be programmed without the "hwc"/"ssn" values being updated in the firmware image. But since the MacBook had never been fully powered down (ie, no mains, no battery) until the battery repair, the "hwc"/"ssn" values had been cached in the hardware controller RAM for years. The firmware update which left out the "hwc"/"ssn" values from the flash programming could have been months/years earlier.

Ultimately Alfred support told me I could just re-register with the same license information and it would work (and they put a note on my license file that "no serial number" was expected; they had seen other cases of no serial number being available, but not of it disappearing during repair. And I managed to get Adobe Creative Cloud (Photoshop/Lightroom) to work again by deleting a bunch of the installation and forcing a reinstallation.

Since then the laptop has worked normally, for over a year. I am writing this blog post on the laptop. The replacement battery seems to work okay, the laptop has sat flat on my desk for well over a year, and the touchpad worked properly with just the battery replaced, so there was physical room left to push the buttons down.

The now over 7 year old laptop is starting to suffer from thermal throttling issues. But that is unrelated to the repair. There seems to be a common issue with the thermal paste under the CPU heatsink eventually drying out, and being a poor thermal conductor, resulting in poor cooling. So the fan runs more loudly at times than it used to. But I am resisting trying to repair that thermal issue until I have fully moved to the laptop's replacement (a Mac Mini).

Oh, and I now have two screw driver sets with the Apple MacBook Pro bits included. Both purchased once New Zealand moved to Level 3 Covid-19 lockdown restrictions at the end of April 2020.

Conclusion

So that is the story of how I had the expanding laptop battery in my 6 year old laptop replaced, during a Level 4 Covid-19 lockdown. By someone I had never met, or spoken to (all communication by email). Who was working from home. On Easter Saturday. And somehow it all worked out. I am very impressed with what The Apple Guy managed, working from home, at the start of the Covid-19 lockdown, when every official repair place was turning away repairs.

(I am also glad that I ordered the Mac Mini to replace it before the 2021 electronics shortage really got started though; even if I still have not fully moved over to the replacement.)

Posted Mon Aug 23 16:25:32 2021 Tags: