There's an old tale of a person stopping to ask directions in a rural area, and being told with that rural drawl "well.... I wouldn't start from here". Upgrading firmware from particularly old versions feels just like that.

A client has an old pair of Juniper EX4200 switches, which were previously used as a Virtual Chassis pair, that they have finally removed from production (in favour of switches with higher port density) and wanted to repurpose the old switches, for a different use, as stand alone switches. One of the first things they wanted to do was upgrade the software on the switches to something comparable to what is on their current production switches.

The old switches had been in production for several years, from back when Virtual Chassis on the EX4200s was pretty new, and over time Juniper had made various software changes that required non-trivial upgrade steps. Since they were in a Virtual Chassis configuration (which means both had to be upgraded at once), and they were the core network switches for the first few years, the switches didn't get any software upgrades for several years. Which meant we knew the software upgrade on them would be non-trivial but we hadn't expected it to be as complicated as it was.

When we started the upgrade, the switches were running 9.2R3.5, from mid-2009 (ie, almost 3.5 years ago). We knew that there'd be a JLoader upgrade (sorry, Juniper login required) to be done to get past 10.3, but were surprised to find a couple more "helper upgrades" were required:

  1. 9.2R3.5 has a software version number check bug (sorry, Juniper login required) where the developers well into 9.xx assumed that the major revision number would always be a single digit; so to upgrade beyond 9.xx it's necessary to install a late release 9.xx version (9.2R4 or newer).

  2. The JLoader upgrade install package will only run on 10.x which means it's necessary to be running 10.0, 10.1, 10.2 or 10.3 before installing the new JLoader. Otherwise you get an error message about missing dynamic libraries: Shared object "libgcc.so.1" not found, required by "bootupgrade"

So the full upgrade process ended up being:

  1. Upgrade to 9.3R4.4 (last 9.x version available)

  2. Upgrade to 10.2R4.8 (last 10.x version available pre 10.4; release notes)

  3. Install JLoader upgrade

  4. Upgrade to 12.1R2.9 (to match other EX4200s in production)

All of which took several hours between research and downloading software to install it. Which clearly justifies the policy of not trying to do this type of upgrade while it was in production.

Also worth noting: since this switch is an EX4200 it kept reporting "Virtual Chassis Mode: Enabled" after each reboot, even though we tried various things to make it a "stand alone" switch (see, eg, Reverting to Factory Default configuration, "request system zeroize" (added in 11.1), etc). What seems to be only rarely mentioned in any searchable place is that the EX4200 in "stand alone" mode will still show Virtual Chassis functionality, it just won't have any record of any other switches listed/associated. So it will always look different from the output on, eg, an EX3200 (which has no virtual chassis functionality). It would help considerably if this was more obviously (searchably!) documented. (As far as I can tell most of the "remove various files" instructions for making an EX4200 switch separate from a previous Virtual Chassis are related to getting it into a state where it can be joined to another Virtual Chassis, rather than obtaining "Virtual Chassis Mode: Disabled". I now suspect the disabled only actually comes up if, eg, version matches mean that the virtual chassis configuration cannot join up correctly.)

ETA, 2013-01-08: If the alarm light on the chassis is showing amber there are various potential causes, amongst "chassis" causes and "system" causes, which are shown by two different commands:

show chassis alarms
show system alarms

The most likely one on a new switch/newly cleared switch is that it will say "Rescue configuration is not set". In which case the solution is to create the rescue configuration (ie, known good configuration) with:

request system configuration rescue save

and then hopefully the amber alarm light will clear, plus:

file list /config

should show a "rescue.conf.gz" file now present.

ETA, 2014-05-08: The J Series does not appear to have a "request system zeroize" command line option. The closest you can get appears to be a SRX/J series factory reset with "load factory-default" from edit mode, which will require setting a root password ("set system root-authentication plain-text-password") before the config can be saved -- and leave various old config recoverable ("file list /config", "file list /cf/var/log", etc). So some additional tidying up will be required. (And a later use may still require a password recovery to get back in again.)