Last Updated: 2016-10-29

Background

As I found last month while setting up an OS X Server, OS X 10.9 (Mavericks) is no longer supported by Apple -- since approximately the time that macOS Sierra (10.12) was released (it appears Apple only actively supports three releases). This also resulted in a lot of third party vendors rolling out new softare releases without support for OS X 10.9 (Mavericks): Adobe, VMWare, etc.

So it was clear that I needed to upgrade my last OS X 10.9 system fairly soon, so that it could have current security updates, and also so that current software could be installed (I already had 2-3 updates that I could not install due to an unsupported OS). Unfortunately my last OS X 10.9 system was also a system that I used all day, every day, so being without it for an extended period would be disruptive....

This weekend is the last "long weekend" in New Zealand before the end of the year, and originally seemed like the best opportunity to do the upgrade without interruptions. It did not quite turn out like that due to CVE-2016-5195 and being in the middle of dealing with multiple customer issues (including network latency). But knowing that I did not have another option in the near future, and following the success setting up OS X 10.11 El Capitan on a client and server it still seemed worth going ahead. Particularly since it happened to be a weekend just after my Internet bandwidth usage cap just reset, so I had less need to be concerned about the volume of downloads (around 40GB downloaded in two days, mostly for this upgrade -- the rest being a few hours of streaming TV).

This hardware is actually capable of running macOS Sierra, but I chose to go with OS X 10.11 (El Capitan) because I was familiar with setting it up, and because macOS Sierra is still quite new and applications software is still catching up.

Preparation

I started preparing for the update a few weeks ago, by checking all the commerical software I had for what was supported on OS X 10.9 (Mavericks) through OS X 10.11 (El Capitan). Most of the software that I had was supported on both, but not always with the same application version. In some cases I either had already purchased an upgrade to a version including support for OS X 10.11 (El Capitan) or there was a free upgrade available; in other cases the only way to go from the version I was running to a version that would work on OS X 10.11 (El Capitan) was to purchase an upgrade.

Where possible I upgraded applications to versions that were supported on OS X 10.9 and OS X 10.11 in advance of upgrading to OS X 10.11 (El Capitan). This was particularly useful with key tools like Alfred, where I had a free upgrade to Alfred 3. But also tools like SuperDuper! have been very good about updating for each new OS version.

I also made a list of the applications for which paid upgrades would be required, and their cost. The main two affecting me are:

  • VMWare Fusion -- I needed to go from 6 Pro to 8.5 Pro (6 does not support 10.11; 8.5 Pro does not support 10.9)

  • OmniGraffle Pro -- I needed to go from 5 Pro to 6 (which supported both 10.9 and 10.11), but unfortunately did not purchase the upgrade in time so can only upgrade to 7 (which does not support OS X 10.9)

In addition I made lots of backups -- not just the two Time Machine backups that happen automatically, but also two clones (with SuperDuper!) of the whole hard drive (one before the SuperDuper! upgrade, and one after) and also additional backups of the key things on my hard drive. Just in case. Two of the backups were to external drives that I usually leave detached; and I also detatched the external drive with my Time Machine backups during the upgrade.

I also identified about 30GB of files that could be "temporarily" removed (ie, relying only on the backup copies -- I made extra copies of those ones) to ensure that there was about 50GB of free space during the whole upgrade process.

OS X Upgrade

Before starting the upgrade, I installed the last few OS X 10.9 updates and then rebooted OS X 10.9 -- to ensure that the installer was running on a freshly booted system. Unfortunately that boot... failed to complete (grey screen, with a spinning cursor). I left it 15 minutes, before giving up -- powering it off (holding the power button down), then powering it on again. On the next boot (after entering the key to unlock the disk) I got told the machine had failed -- and choose not to restore the open applications. To be sure, I rebooted the machine again -- which worked fine. It seemed safe to proceed. I removed the 30GB of sacrifical files, and checked that there was 50GB free.

I chose to do this upgrade from within the OS (rather than booting off the USB key I made earlier), because the drive was encrypted and I wanted to retain the drive contents (thus a USB boot was not necessary). To start the upgrade:

  • Run /Applications/Install OS X El Capitan.app

  • Agree to the license terms

  • Agree to install to "Macintosh HD"

  • Click on the "Restart" button in the application

The tool reported that it would take about 26 minutes, and reboot multiple times -- which turned out to be basically true (it took around 30 minutes, and rebooted several times). The progress messages were updated somewhat sporadically, but were relatively useful in tracking how far the upgrade had got. There was one period where the fans spun up to full speed, but that settled down again after a couple of minute (I assume either a lot of compression/crypto work happening, or perhaps temperature control firmware changes).

After the base OS upgrade completed, it booted into an abbreviated form of the OS X 10.11 (El Capitan) setup pages -- asking about my Apple ID (which I chose not to provide at that step), and whether I wanted to share diagnostic/usage data with Apple (no thanks).

Then a fairly normal OS X 10.11 (El Capitan) desktop came up, mostly with my preferences for that laptop. The main surprises were:

  • Calendar showed (6) "sync" conflicts, but it was unclear where it was syncing to, in order to find those conflicts; most of the records looked basically the same, so I chose a winner somewaht arbitrarily (favouring my phone calendar first, then the OS X Calendar, over the iPad calendars -- as that is my experience with what I use to enter events, and also what most robustly handles recurring/day spanning events; the iPad does poorly at both, particularly "all day" events which basically forever have gradually creeped from one day to multiple days :-( ).

  • Apple's Mail.app also auto-launched, to my surprise -- but did show new mail, so clearly its preferences were okay. The main thing I noticed is that my Smart Folders were missing almost all the mail they should have had, but I assumed that was because the Spotlight database had been wiped and was being rebuilt (they did repopulate overnight, I assume once the Spotlight reindex completed).

At this point I forced a Time Machine backup (to my OS X Server, as I was not going to connect external drives yet), before carrying on. It had 33GB to back up. I let that finish before the next steps.

Installing OS X Updates

The next step was to install the Apple patches that had been released since the "Install OS X El Capitan.app" had last been released. I installed the OS X Security Update first (with a restart), and then Safari (another restart; unclear why they cannot be scheduled to be updated together).

Once those were done I installed the application updates that were present:

  • XCode (4GB)

  • iMovie (2GB)

  • iPhone (2GB) -- to the last released version, AFAICT, from 2015 (supposedly makes migration to the Photos app easier)

  • Aperture 3.6 (0.6GB), from 2014 (not sure if it will run under OS X 10.11 El Capitan, as the last supported version was supposed to be 10.10; but I basically gave up trying to use Aperture when Apple discontinued it so it does not matter either way

ETA, 2016-10-26: Then just a couple of days after I had the last set of security updates, another base OS X Security Update (10.11.6) came out -- in (partial?) mitigation of issues found with XNU task_t (it is not clear if they are sufficiently mitigated in 10.11.6; the blog post implies that only the macOS 10.12 Sierra refactoring significantly helped mitigate the root cause). So that was two more reboots -- for the 10.11.6 security update, and another Safari update.

MacPorts

My next step was to run sudo port selfupdate to see what the status of MacPorts was like. It turns out that MacPorts checks the OS version and refuses to run:

Error: Current platform "darwin 15" does not match expected platform "darwin 13"

which then directs you at the MacPorts Migration page.

The Migration process is:

  • Install the new version of XCode (done, above)

  • Ensure the XCode command line tools are installed, and recognised:

    xcode-select --install
sudo xcodebuild -license

    by choosing "Install" (rather than all of XCode), and agreeing to the license (typing "agree" at the end).

  • Test that the compiler now works, eg with a simple "Hello World" application

  • Download and install the latest MacPorts base package for your OS version (which fortunately I already had handy from my previous OS X 10.11 El Capitan installs); after which "sudo port ..." commands at least ran without errors

  • Remove all ports, keeping a list of what they were:

    port -qv installed | tee myports.txt
port echo requested | cut -d ' ' -f 1 | tee requested.txt
sudo port -f uninstall installed
sudo rm -rf /opt/local/var/macports/build/*

  • Reinstall all ports, using the restore_ports.tcl script:

    chmod +x restore_ports.tcl
sudo ./restore_ports.tcl myports.txt
sudo port unsetrequested installed
cat requested.txt | xargs sudo port setrequested

Because I had about 1700 ports installed (not all of them active; I usually keep the previous active version of each port around, just in case of problems, so that I can roll back)... that was always going to take quite a while.

It also failed in various ways, so I was glad that I had run it in a script session and could review the failures. The main ones seemed to be:

  • Failures due to packages that had been replaced (eg gnome-icon-theme with gnome-theme-standard) where the maintainer decided the correct solution was to throw an error, rather than install a transitional package which depended on the new package :-(

  • Failures due to packages that had been superceded, with the same "throw an error rather than install a transitional package" issue (Debian does this so much better). Perl 5.16 (p5.16) was an especially common case of this for me.

  • Perl 5.22 (p5.22) packages failing to build and/or download, claiming they were unable to download the dist files from anywhere (possibly a transient network errors, or possibly related to the logged reports of too many open files).

  • There is also a Perl 5.24 (p5.24) which seems more relevant now.

At the (failed) end of the first attempt about 240 ports (of 1700 installed before) had been reinstalled.

I get the impression that restore_ports.tcl had not really been tested with, and fixed for, large numbers of ports. And/or tested with lists of ports that are several years old. In particular it almost certainly should not be trying to install inactive ports, as either (a) there is an active port with the same name that is needed instead, or (b) the port is not being used. All my Perl 5.16 (p5.16) ports were inactive.

Ignoring the inactive ports, there were around 1000 active ports to reinstall; and ignoring Perl 5.22 (p5.22), which I am happy to consider replaced by Perl 5.24 (p5.24), there are about 800 ports to install.

In the end I restarted the port install process with:

 port install gnome-theme-standard

which pulled in quite a few dependencies. Then I built a variation of myports.txt without p5.16 or p5.22 ports in it, including only the active ports, and tried again with that:

 egrep -v 'p5.16|p5.22' myports.txt | grep active | tee myports-active.txt
 sudo ./restore_ports.tcl myports-active.txt

that did some cleanup for the already installed ports, and then started installing with the p5.24 ports, before carrying on to install other ports.

That got most of the way through, before failing out again with "too many open files" -- so I had to run:

 sudo ./restore_ports.tcl myports-active.txt

a second time to finish the missed ports. The second run with myports-active.txt ran to completion with few errors. I ran:

 sudo ./restore_ports.tcl myports-active.txt

again to check, which showed two packages that failed to install: py-dateutil and jpilot. For both of those sudo port install PACKAGE failed.

For py-dateutil:

sudo port clean py-dateutil
sudo port install py-dateutil

was sufficient to get it going.

For jpilot, which is important to me still, it turned out that the configure script was failing, due to a missing dependency for Perl XML::Parser, which turned out to be because the Perl being used was MacPorts Perl 5.22, but I had skipped installing the p5.22 packages, because they were all failing -- intending to switch to Perl 5.24. However I had omitted to switch to Perl 5.24, and it is not the default (yet). Since now seemed like as good a time as any to switch (and the port select framework does not work for Perl), I followed the magic incantation and reinstalled the perl5 package:

sudo port install perl5 +perl5_24

which then changed the /opt/local/bin/perl link to point at Perl 5.24 (as verified with perl -v).

After that, I was able to install jpilot again with:

sudo port clean jpilot
sudo port install jpilot

I ran:

sudo ./restore_ports.tcl myports-active.txt

one last time to be sure, and this time it completed without errors, getting to around 800 ports installed.

Finally I reset the "requested" flags appropriately with:

(egrep -v '^gnome-icon-theme|^p5-|^p5.16|^p5.22|git-core' requested.txt |
     sort | uniq | tee requested-kept.txt)
sudo port unsetrequested installed
cat requested.txt | xargs sudo port setrequested

where that list of things to exclude was found somewhat iteratively, as port setrequested will fail if it is given something that is no longer installed.

Overall the process took several hours, a lot of downloads, and some fincky manual attention to get back to the point of a mostly working set of ports.

I suspect the restore_ports.tcl script might benefit from either (a) explicitly closing files it does not need, (b) ensuring that the open file handles do not leak to children, or (c) ensuring that it reaps child processes promptly; I did not try to investigate which handle limit was being hit -- ie a process-wide one or a system-wide one. It would also clearly benefit from not trying to install ports which were not active before (the information needed is in the output it reads).

ETA, 2015-10-25: I found that the active perl version had reset itself back to 5.22, which seemed to be due to "perl5 @5.22.2_0+perl5_22" for some reason becoming the active port, possibly related to the "setrequested" above, or possibly related to updating something else. To fix this I did:

sudo port activate perl5 @5.22.2_0+perl5_24
sudo port uninstall perl5 @5.22.2_0+perl5_22

so that only the desired "flavour" of perl5 was installed.

I found that I had to forcably rebuild irssi (which uses perl as an extension module) to get it to pick up this change, as it appeared to have been built elsewhere with perl5 set to perl5.22:

sudo port upgrade -s -n --force irssi

(the "-s" forces a source rebuild; the "-n" limits the rebuild to just the named port; the "--force" ensures it gets rebuilt even if it is just the same version). After that my irssi helper scripts seemed to work again.

git-annex

Once MacPorts was sorted out (above), which provides git, rsync, etc, for my git-annex install, I think upgraded git-annex by:

  • Moving the old version aside:

    cd /Applications/OpenSource
mv git-annex.app git-annex-bkup-2016-10-23.app

  • Opening the git-annex.dmg that I had downloaded recently, and dragging that git-annex.app into /Applications/OpenSource

  • Verified that my git-annex links in /usr/local/bin still worked:

    git-annex

    (which they did as they pointed at the same path that I had just put the new version into)

I then did a bit of testing with git-annex to make sure that it was working as expected -- eg, git annex sync in some existing repositories, etc. It seemed okay -- and I did not expect any problems given I had the same version working on another system, albeit with a different setup.

Once that seemed to be working, I forced another Time Machine backup to my Time Machine server.

VMware Fusion 8.5 Pro

Even though VMware state that "A simple upgrade to support a new OS shouldn't cost you", that appears to only be true if you have already purchased all the other upgrades. So instead the best I get is VMWare Upgrade Pricing being allowed to jump from VMware Fusion Pro 6 to VMware Fusion Pro 8.5 -- and in theory VMware Fusion Pro 8.5 supporting both OS X 10.11 El Capitan and macOS Sierra to allow a bit more wiggle room for future upgrades. (It appears to be no cheaper to upgrade from an earlier Pro version to the latest Pro version than it is to upgrade from any supported earlier version or the current non-Pro version to the current Pro version... which is an odd penalty for repeatedly upgrading to the Pro version.)

I purchased the upgrade earlier this week in anticipation of upgrading; it cost around AUD$100 plus tax -- so a moderately priced upgrade, at least given jumping multiple versions in one go.

After purchasing and downloading VMWare Fusion 8.5 Pro (which appears to be one download file for both versions, without bundled drivers), the upgrade process seems to be:

  • Verify the checksum of VMware-Fusion-8.5.0-4352717.dmg:

    shasum -a 256 VMware-Fusion-8.5.0-4352717.dmg

    (which should be 2a19b1fd294e532b6781f1ebe88b173ec22f4b76d12a467b87648cc7ff8920f1 according to the information on the VMWare download page at present)

  • open VMware-Fusion-8.5.0-4352717.dmg

  • Double click on the icon in the dmg to run the installer

  • Agree to running the "application downloaded from the Internet" (so long as you are happy with how you downloaded it)

  • Enter your password to give the installer administrative privileges

  • Agree to the terms and conditions

  • Choose "I have a license key for VMWare Fusion 8", and cut'n'paste the license key you have into the license key box (you should be rewarded with a yellow tick). Choose Continue

  • Enter your password again to allow the installer to proceed

  • Verify the "thank you" screen mentions "VMware Fusion Pro" if you purchased the professional license

  • Untick "Yes, I would like to help improve VMware Fusion" to reduce the amount of information leaked back to VMware

  • Click "Done"

VMWare Fusion 8.5 should launch, and VMware Fusion->About should show that it is "Professional Version 8.5.0 (4352717) -- and if you had it installed before your existing VMs should be present in the Virtual Machine Library.

There is no further prompts for where to install, etc; as far as I can tell it automatically installs into /Applications/VMWare Fusion.app -- complete with embedded space -- and the previous version seems to be ignored if it was located somewhere else.

I prefer to have my purchased appplications in /Applications/Purchased if possible, and also to remove the older version (wich will not run on OS X 10.11 El Capitan anyway), so I:

  • Went to /Applications/Purchased and trashed the old VMware Fusion.app

  • Dragged the new VMware Fusion.app from /Applications/ into /Applications/Purchased

Launch some of the VMs to make sure that they run correctly. On upgrading packages in some of my Ubuntu Linux VMs, I noticed reports that:

/etc/vmware-tools/thinprint.sh: /usr/lib/vmware-tools/moduleScripts/thinprint-vmware-db.pl: not found

a few times; I'm not sure if they were a result of an automatic update happening, or an error with the older VMware Tools on the newer VMware Fusion.

So after upgrading the packages, and then rebooting, I also went to force the VMware Tools upgrade with Virtual Machine->Reinstall VMware Tools which told me to "Mount the virtual CD in the guest, uncompress the installer, and then execute vmware-install.pl to install VMware Tools". On Ubuntu Linux, assuming it is already set up for building the VMware Tools, that is something like:

sudo mount /dev/sr0 /media/cdrom
sudo mkdir -p /usr/local/src/vmware-tools-distrib
cd /usr/local/src/
sudo tar -xpf /media/cdrom/VMwareTools-10.0.10-4301679.tar.gz
cd vmware-tools-distrib
sudo ./vmware-install.pl

But it turns out that open-vm-tools is (a) available for (at least) Ubuntu 14.04 LTS, and (b) recommended by VMWare, as advertised in the installer.

So instead I uninstalled the existing VMWare tools by:

sudo /usr/local/bin/vmware-uninstall-tools.pl
sudo shutdown -r now

(exact location of the uninstaller depends on the version: /usr/bin/vmware-uninstall-tools.pl is mentioned in the VMWare documentation).

Once the system came back up again, I installed open-vm-tools:

sudo apt-get install open-vm-tools
sudo apt-get install open-vm-tools-desktop    # system with X11

and rebooted again to make sure they were running correctly on boot.

I also tested my wrapper script to launch VMs from Alfred, and they did successfully start the VMs -- but unfortunately they left the Virtual Machine Library window open. Previously I had a closewindow script to automatically close the Virtual Machine Library window after a short delay (via AppleScript), but even after fixing a capitalisation issue, that fails with:

31:96: execution error: VMware Fusion got an error: every winow whose
name = "Virtual Machine Library" doesn't under the "close" message. (-1708)

which is not very helpful -- and unfortunately there still does not seem to be an option to either (a) not show the Virtual Machine Library window or (b) automatically close it when a Virtual Machine is launched :-(

My "close window" script, which uses AppleScript, seems to have been broken by VMware removing AppleScript support in VMware Fusion 7 :-( It appears that vmware.sdef was removed in the release (GA) version of VMWare Fusion 7, and has never reappeared -- with VMWare Fusion 7, some people were able to make it work by restoring vmware.sdef from their VMware Fusion 6 install... but it is unclear if that also works for VMware Fusion 8.

After some hunting around I found that I could use vmrun to start the VMware Virtual Machine without the annoying Virtual Machine Library window appearing at all. Usage, at least in my case with the "custom" location of VMware Fusion.app is:

"/Applications/Purchased/VMware Fusion.app/Contents/Library/vmrun" start /VMDIR/VMNAME.vmx

where VMDIR is the directory containing the VMX file, and the rest is the name of the vmx file itself. It appears to work whether or not VMware Fusion is already running, and whether or not the VM is already running (without error; if it is already running then it just silently exits).

So I have changed my Automator Wrappers for Alfred to use that approach instead of the open VMDIR and closewindow approach I was using previously.

Adobe updates

Use the Adobe launcher to (a) upgrade itself, and then (b) install the outstanding application upgrades (which relied on being on OS X 10.10 or higher). Other than a few downloads being required, it was pretty easy to get back up to date.

OmniGraffle Pro

I have OmniGraffle Pro primarily for interaction with people who use Visio -- which means that I need OmniGraffle Pro. Version 5, which I already own, does not work on OS X 10.11 El Capitan, which means that I need to upgrade -- and the only upgrades available are paid upgrades to the current version, OmniGraffle 7 released earlier this month (OmniGraffle 7 Release Notes). Since OmniGraffle 7 did not work on OS X 10.9 Mavericks, I put off upgrading until after I had OS X 10.11 EL Capitan working.

Upgrading OmniGraffle to version 7 is a matter of:

  • Going to the OmniGroup Store

  • Clicking on OmniGraffle for Mac

  • Entering your existing license owner and key to "View Upgrades"

  • It should offer a US$99.99 upgrade to OmniGraffle 7 Pro; click "Buy"

  • Pay for the license

  • (Hopefully!) receive email with license key in it

  • Download OmniGraffle 7, noting that there are fairly recent updates at present (the OmniGraffle 7.0.2 I downloaded a week or so back in preparation is already out dated by an OmniGraffle 7.0.3...)

  • open OmniGraffle-7.0.3.dmg

  • Agree to the license agreement

  • Trash the old OmniGraffle Professional.app

  • Drag the new OmniGraffle.app into a suitable location, eg, /Applications/Purchased

  • Run the new OmniGraffle.app from Finder so as to be able to agree to running an application downloaded from the Internet

  • Close the "Welcome to OmniGraffle 7" window

  • Go to OmniGraffle->Licenses, and then choose "Add License..."

  • Cut'n'paste the Owner and License Key values from the email sent by the OmniGroup Store, and choose "Save"

  • Check that the "Thank You" window shows "You have licensed OmniGraffle 7 Pro", and that the "trial version expires" text has vanished from the top right.

After which hopefully OmniGraffle 7 Pro will work in an equivalent manner to OmniGraffle 5 Pro. (I suspect that the vendor stencils that I downloaded may have vanished, but I will leave figuring out how to get those going again for later.)

Backups, backups, test reboot

At this point I forced another Time Machine backup to each of the external Time Machine drive, and my Time Machine Server. Then did a test reboot.

Drivers

Printer

My printer driver seemed to work automatically, so I left that one alone.

Wacom Intuos Pro tablet

The Wacom Intuos Pro tablet drivers had been updated to 6.3.18-4 (released 2016-10-14) in the time since I installed them on the last system. And the system I was upgrading had a much older version (6.3.8 I think; it did not seem easy to find out which version was installed).

So I:

  • Ensured the tablet was not connected to the computer.

  • Downloaded the latest driver version

  • Removed the previous driver, which I had to look up on archive.org, as Wacom had managed to break their website/FAQ section somewhere since August :-( For reference: run the Wacom Utility, and click on "Remove" in the "Tablet Software" section, then enter your password to allow the removal to happen. (It should say "The Tablet Software has been removed", and the Menu Bar icon should go away if you had that enabled.)

  • open WacomTablet_6.3.18-4.dmg

  • Run the Install Wacom Tablet.pkg installer

  • Agree to the license, confirm the installation (158MB, down from 250MB last time I installed it!) and note that it is necessary to restart the computer after the install finishes. Enter your password to allow the installer to make changes.

  • After the installer has copied the files, click "Restart" to restart the computer (and thus load the drivers).

  • Once the system comes back up, plug in the tablet and make sure it is recognised, and can be used to move the pointer around. There is a "Wacom Tablet" tab added to the System Preferences which can be used to configure the tablet buttons, etc.

"FTDI" USB Serial adapter

As I found when installing my previous OS X 10.11 El Capitan system, the situation system for the "FTDI" USB Serial adapters (bought as a store branded item, about a decade ago -- the store is now out of business!) is... problematic. Due to having a different VID/PID (ProductId=0x0421 (1057), VendorId=0x0b39 (2873)) than the current modern devices, they do not work with Apples FTDI driver -- and OS X 10.11 El Capitan system protections prevent modifying the driver Info.plist to match the VID/PID while still leaving the normal system protections enabled.

FTDI do have a version 2.3.0 signed driver for OS X 10.9+, released 2015-04-15, which in theory will work with the ProductId/VendorId and OS X 10.11 El Capitan. But as I explained in my previous post, FTDI are very consumer hostile, so it is a lottery whether or not they will "brick" the devices -- permanently disable them.

I was expecting to have to undertake the "FTDI lottery" (install new driver, see if the device would ever work again) to find out whether or not my device would work with the later FTDI driver.

To my surprise I found that without doing anything, when the USB Serial device was plugged in to my upgraded system, it worked. It both appeared in /dev/ (eg, /dev/tty.usbserial-OCBAMDL5) as it did before the upgrade and seemed to pass serial data back and forth as before.

Doing:

ksextstat | grep -v com.apple

showed that "com.FTDI.driver.FTDIUSBSerialDriver (2.2.18)" (ie, the version I had installed before upgrading) was still being loaded, to my surprise.

The FTDI download page implies (by omission) that the 2.2.18 driver is not signed (ie, it is listed as only suitable through OS X 10.8; driver signing was introduced later), and digging around in:

/System/Library/Extensions/FTDIUSBSerialDriver.kext/Contents

seems to confirm that there is no code signature for it.

Some hunting on the Internet turned up online discussion suggesting there is an exception list of known older drivers that are allowed to load without their own signature (ie, Apple is vouching for them). This is in:

/System/Library/Extensions/AppleKextExcludeList.kext/Contents/Info.plist

and contains a list of kernel extensions and hashes, as a whitelist, in OSKextSigExceptionHashList. That list seems to include several versions of the com.FTDI.driver.FTDIUSBSerialDriver, including 2.2.18 (presumably with the hash for the version I already have installed on my upgraded system is one of them -- but the currently downloadable 2.2.18 may not be; some more information on kernel exceptions). Apparently the OSKextSigExceptionHashList contains most of the unsigned third party kexts known to Apple as of when the signing requirement was introduced with OS X 10.9 (Mavericks) -- which may be the reason why the FTDI 2.2.18 driver worked even in OS X 10.9 (Mavericks). The hashes are 160 bits long -- same length as a SHA1 sum, for instance, but not many other hash functions -- but it is not clear what they are a hash of, or exactly what algorithm is used (I cannot find a match for the hash of Info.plist or FTDIUsbSerialDriver with SHA1 or RIPEMD-160). Presumably it uses something similar to what codesign uses, but I have not managed to identify what algorithm that uses either, or precisely what the signature is calculated over (see also Apple TN2206 on code signing).

Since /System/Library/Extensions is not supposed to be written to by third party vendors (since OS X 10.10 Yosemite IIRC) I suspect it would not be possible to use the automatic installer of the FTDI 2.2.18 driver again -- but it may be possible to install it by hand onto another system in /Library/Extensions.

However this does mean that my upgraded system is working, without me having to risk installing a newer driver to brick the hardware. So I am declaring success for now.

ETA, 2016-10-29: Looks like I declared success a bit too soon. Today on trying to use the USB serial adapter that had been plugged in for days, I found that it seemed not to work although the driver was loaded and the /dev/tty.usb... device was present. Looking in dmesg I found:

211310.991669 USB to Serial Converter@41112000: AppleUSBDevice::ResetDevice: <software attempt to RESET>

repeated frequently. With some searching for that message I found a post about OS X El Capitan and its refusal to reset USB devices which suggests the problem is that OS X 10.11 El Capitan is does not wake the USB device from sleep with a "Reset" any longer -- and thus presumably the USB device stayed asleep. (Other people have had similar issues with older versions, and with Arduino Development on OS X 10.11 El Capitan.)

Armed with that information I found that I was able to wake the USB Serial converter up by stopping the application using it and then doing the usual "have you tried unplugging it and plugging it back in again". That is a (barely) tolerable work around for my main USB Serial usage (occassional Pilot backups). The only other periodic using is serial console where usually I'll start plug in a different adapter and start the client application almost immediately, which should not run into this issue.

My best guess is that OS X 10.11 El Capitan is now putting that USB port (on my keyboard) to sleep and not waking it up again unless something tries to access it -- whereas OS X 10.9 Mavericks may not have been putting it to sleep. Possibly the later driver version might actually call USBDeviceReEnumerate, rather than just ResetDevice, which seems like it might avoid this problem, but possibly not. (Apparently "they rewrote the USB stack in 10.11", and the driver has not been updated since OS X 10.9ish anyway.) Alternatively it might be possible to write something, perhaps with libusb that will wake the device up by forcing re-enumeration. There is a darwin_reset_device which will to force reenumeration in some situations, but it is unclear if it would be triggered in this situation. (If not it may be possible to follow the same logic and force re-enumeration, digging into the internal structures.)

(For amusement: found while debugging this, how to hook up a serial terminal to OS X.)

Subsequent issues found

ETA, 2016-10-28: On first trying to start a new backup with SuperDuper! by plugging in an external USB backup drive (which had a "scheduled" backup), I was greeted by the error:

sh: /usr/bin/lockfile: No such file or directory (127)

This turns out to be a known issue with SuperDuper! after upgrading to OS X 10.11 El Capitan, after beta 4 -- the lockfile command being used was part of procmail, which was removed in OS X 10.11 El Capitan. (And indeed "which lockfile" returns nothing now.)

Apparently the update of SuperDuper! that I had already installed (while still on OS X 10.9) includes a bundled lockfile command... but it appears that the auto-triggered backup scripts didn't get re-written automatically, even when running SuperDuper! after upgrading SuperDuper! and before upgrading to OS X 10.11 El Capitan

The (only, AFAICT) fix is to open SuperDuper! manually, remove every scheduled job, and recreate each one from scratch, at which point the scheduled jobs get written with the new lockfile path. This is hinted at by the "And that means, unfortunately, that users have to delete and recreate their schedules." line in the Shirt Pocket announcement, but the reason for needing to do so is never really explained in any detail.

As far as I can tell SuperDuper! handles these "Backup On Mount" commands using the AppleScript found in "SuperDuper!.app/Contents/Resources/Backup on Mount.scpt", which runs at login and on drive connect, from launchd. This is done via ~/Library/LaunchAgents/com.shirtpocket.backuponmount-login.plist and ~/Library/LaunchAgents/com.shirtpocket.backuponmount.plist which run a copy of the script, in "~/Library/Application Support/SuperDuper!/Scheduled Copies/Backup on Mount.scpt" for reasons that are not clear. It uses the StartOnMount feature of launchd to trigger that. Then if it finds a newly appeared drive which matches a drive for which it has a scheduled job, it will start that scheduled job.

After a lot of debugging I found that:

  • The copy of "Backup on Mount.scpt" appeared to have been updated to call the correct lockfile (although for some reason it was not byte for byte identical to the one in /Applications/Purchased/SuperDuper!.app; and I still do not understand why there is a copy made). At a guess this copy of "Backup on Mount.scpt" was updated when I ran SuperDuper! after upgrading it, on OS X 10.9 Mavericks, before the OS upgrade.

  • It appeared that the "Backup on Mount.scpt" ran correctly, but the /usr/bin/lockfile failure reports continued.

  • Digging deeper it turned out that every scheduled job had its own copy of "Copy Job.applescript", which itself had a hard coded call to /usr/bin/lockfile -- and presumably that call was the one that was failing :-( (These jobs are in "~/Library/Application Support/SuperDuper!/Scheduled Copies/Smart Update FOO from BAR.sdsp" for anyone looking; there appears to be a version compiled into an Application in "Copy Job.app`" too.)

  • It appears these scripts are built from a "template" with some sort of string substitution, and then saved into the Scheduled Job, and compiled into an application that can be run on drive attach. (AFAICT the template is /Applications/Purchased/SuperDuper!.app/Contents/Resources/Copy Job Script.template.)

  • Using directory names with spaces and especially with exclamation marks in them is SuperAnnoying! (at least when trying to look at anything in the command line); and using script files that are CR terminated only (Old School MacOS) is painful on modern macOS (which expects unix style LF termination).

The result of all of this is that there basically is no reasonable way to make SuperDuper! Scheduled Jobs work again on OS X 10.11 El Capitan other than deleting every Scheduled Jobs, and creating each one again from scratch. So I have deleted all the scheduled jobs. And now I have to remember as I get each external drive back from off site to re-add the scheduled job so that its backup runs automatically :-( As well as be SuperCareful! to recreate the jobs with the correct drives as source as destination, carefully navigating my way through the unfortunate SuperDuper! UI that states things backwards so that you have to read very slowly to ensure the backup source and destination are correct. ("Smart Update will copy and erase what's needed to make DESTINATION identical to your selections from SOURCE" is really not clear English, and that is what the final "are you sure" dialogue says :-( I have relied on having pre-triggered jobs to avoid having to navigate this confusion at the end of a long work week when I needed to make a backup and really did not want to overwrite the week's work with a several week old backup...)

The whole design feels... excessively fragile to problems like lockfile being removed between OS releases. It is rather unclear to me why the "Copy Job.applescipt" could not just be a tiny wrapper that passed a few strategic parameters to a centrally maintained script that would be updated any time SuperDuper! was updated -- entirely avoiding the need to manually delete and recreate things.

For future reference, the cache of "discovered drives" is kept in "~/Library/Caches/TemporaryItems/com.shirtpocket.lastVolumeList.plist" should you need to debug it further. It is created by the "Backup on Mount.scpt" on first run, eg when run at login with -runAtLogin as an argument. An equivalent file is created on each run in the same directory, to use to compare and figure out "what is new", to determine if a job should be run.

Also for future reference, launchd is ignoring the "ThrottleInterval" of 0, as logged to the system log:

Oct 28 19:07:52 ashram com.apple.xpc.launchd[1] (com.shirtpocket.backuponmount): ThrottleInterval set to zero. You're not that important. Ignoring.

(which is in ~/Library/LaunchAgents/com.shirtpocket.backuponmount.plist for unknown reasons).

Finally for the record, Time Machine also uses StartOnMount jobs to trigger its automatic backup via /System/Library/LaunchAgents/com.apple.TMHelperAgent.SetupOffer.plist. The other common way is apparently an AppleScript "Folder Action" on /Volumes (where external drives mount). (Also, more on launchd.)