Fundamental Interconnectedness

This is the occasional blog of Ewen McNeill. It is also available on LiveJournal as ewen_mcneill, and Dreamwidth as ewen_mcneill_feed.

Every year the NZIFF puts on an amazing International Film Festival in Wellington. The selection of films is great, and the venues are wonderful (although it is sad to lose Paramount late last year, and not have it part of the Festival; hopefully Reading is at least half as good a home for the NZIFF).

Unfortunately every year in the last 4 years that I have tried to buy tickets, the ticket purchasing website that NZIFF have chosen has either failed to cope with demand, or utterly failed to cope with demand. (The difference being whether it was possible to buy any tickets at all on the first morning.)

So far this year I have managed to pay for zero tickets, depsite trying for nearly two hours at this point (across two different web browsers, for a single film at a time in each attempt). Most attempts time out even before loading the first page from the ticketing site; some got part way through the process and then timed out.

For added excitement, a couple of ticket purchase attempts claimed my "order completed successfully" without taking any money -- and I am apparently not alone. When you try to "Print Tickets", it claims "your booking was successful but there was a problem creating your Print At Home Ticket" -- and tells you to print the confirmation and pick up your tickets at the counter. And no email is sent. So... maybe they are my seats? Maybe. That is an exciting new failure mode not seen in previous years. (I have already emailed the requested address with the relevant ticket confirmations, offering to pay for them, and got a reply back saying "I have printed that off for our friendly ticketing people to process as soon as they're off the current call", but it is still unclear if I should just try to buy more tickets once tickets are on sale again, or if the seats it claimed were mine will actually be mine if I pay for them.)

As with every previous year -- since NZIFF stopped using Ticketek -- an hour or so into the problem NZIFF's ticket vendor took ticket purchasing offline to add more capacity. Ticket purhasing seems to have been turned off for 45+ minutes so far. So perhaps they are also trying to clean out their database of failed attempts as well...

I am still left wondering why they did add (substantial!) additional capacity at 9:00 this morning (before ticket sales started), for the first day, rather than waiting until 11:00 this morning (after everyone had an hour of frustration). The ticket purchasing site has failed in pretty much the same way for multiple years -- my 2014, 2015, and 2017 experiences are basically all the same types of failures. At this point we are surely past the "no one could have predicted the demand" stage...

It is also puzzling that the ticketing website is still using exactly the same inefficient process as previous years -- none of the purchase flow optimisations I suggested last year seems to have been implemented, nor any other obvious technology improvements (eg, longer cache times for unchanging things, reduced requests, etc). Any of those potential changes that reduced the number of requests needing to be made to the central ticket server could have significantly increased the number of ticket sales it could have completed.

Clearly the NZIFF team is "really frustrated this has happened and greatly appreciate your patience", and honestly I do feel for them dealing with these problems every year. But there seems to be a pattern of being let down by the same vendor, year after year, in the same way -- apparently due to the vendor failing to plan in advance for the -- entirely expected -- rush of ticket purchases on day one. And that pattern of being let down also seems rather predictable at this point :-(

Hopefully I eventually get to buy the tickets to the 25+ sessions that I want to see, before all the good seats are gone. And the memory of the pain of ticket purchasing fades with the amazement at the films being shown.

ETA, 2018-07-05: Unsurprisingly the ticket sales worked much better after the ticket vendor had upgraded the ticket server to handle more capacity. Hopefully there will be a post mortem on this event when the question "why didn't you do that the day before?" is asked, and answered. I mostly seem to have even managed to get good to okay seats for all sessions, including the ones where I "ordered successfully" but got no tickets, and ended up just buying them again rather than waiting for the NZIFF staff to have time to process them when all the good seats were already gone.

When the server works properly the best process seems to be:

  • Save all the tickets you want into a logged in Wishlist

  • Have two browser sessions open, logged into that same Wishlist

  • Work down the Wishlist, one session at a time, booking the tickets, one session in each browser.

  • While one is stuck loading, swap over to the other one

  • Memmorise the good seats in the theatres you will be in, as you will have to pick a good seat every single time as the "best available seat" claim remains a complete lie. You also need to recognise when the page has reloaded after seat selection, as it displays the old seat for a long time after "done" on the seat selection, and anything entered there will be ignored.

  • Get very very good at typing in your name/email address/phone number, and postal address. As in minimal exact keystrokes good. (Firefox is better at assisting with this than Safari...)

  • Get very good at typing in your credit card details (again minimal keystrokes, from memory) as you will be doing that a lot too.

  • Proceed as quickly as you can, but do not be tempted to try reloading the pages -- they will either work, or timeout, and reloads will not help if the server is dying under load.

  • Keep a physical list of sesssions you want and tick them off by hand; the "ticketed" indicator on the Wishlist is misleading, as it just indicates "request sent to ticket server" and not "tickets available to print".

  • Double check you got everything, without duplicates, at the end before stopping.

Once again the ticket purchasing process took 3 hours, of which the first two hours were utter frustration and the last hour was fairly productive. Maybe next year the server upgrades can be done in advance and we can just have the single "fairly productive" hour.

Posted Thu Jul 5 11:47:09 2018 Tags:

Introduction

After opening the Amstrad CPC time capsule again, I wanted to transfer data off the Amstrad CPC 3" disks that I had, onto a modern PC for archiving. The modern approach to this seemed to be to use a Gotek USB Floppy Emulator, running FlashFloppy, connected to the Amstrad CPC as an external drive (drive B:) -- and then use disckit3.com to copy disk sides from drive A: (the 3" drive) to drive B: (the Gotek emulator). So that was what I set out to do.

Previously I proved the Gotek PC 1.44MB floppy emulator worked on an IBM PC clone so I knew the Gotek hardware worked. And earlier I repaired the two Amstrad CPC 3" drives that I have (by replacing the floppy drive belt), so all that remained to be able to rescue the contents of my 3" discs was to:

  • Install FlashFloppy onto my Gotek unit;

  • connect the Gotek unit to one of my Amstrad CPC6128s; and

  • copy the 3" disc sides onto the Gotek

Disclaimer

If you decide to follow these instructions, please note that you proceed at your own risk. I would advise reading some other install guides as well to be sure you understand the steps before starting. And if you choose the technique involving soldering make sure you are either familiar with soldering safely and know how to avoid creating short circuits, or get the soldering done by someone more experienced. There is also a risk that the FlashFloppy firmware programming might fail and leave you with a dead Gotek board. (I think most firmware programming issues can be fixed by trying the serial loader programming again, but this firmware programming will definitely void your warranty, and may leave you with a dead Gotek.)

If all of these instructions seem too complicated for you, note that there are several vendors online, including TechExpress, who will sell you Gotek units already pre-programmed with FlashFloppy (or HxC firmware for the Gotek) in a ready to use package. I chose to do it myself, as described below, because I wanted to understand the whole process, and was already at least somewhat familiar with all the steps involved.

Preparing to install FlashFloppy

There are two methods to install FlashFloppy onto a Gotek unit:

  1. Install via serial loader using a USB to TTL serial cable; or

  2. Install via Device Firmware Update (DFU) through the USB port on the Gotek, which requires a USB-A to USB-A male to male cable (relatively uncommon; people mostly seem to have made them by cutting the ends of other cables and soldering them together).

Since I already had a USB to TTL serial cable, which I had previously used for programming another ARM STM32 chip, and I knew from reviews that the Gotek contained an ARM STM32F105 chip, my plan was to program FlashFloppy onto the Gotek using the serial loader.

To be able to use the serial loader, it is necessary to be able to connect up to the serial pins in the circuit board and short two pins together to hold the STM32F105 in programming mode. These pins appear on a 8 pin header at the back of the board (actual photo in another programming article), near the power connector -- but on most boards do not have any pin headers on them. (My board seemed to have 9 pin headers: 5 + 4, but the relevant pins seemed to be in the same position.)

If you happen to have good probe connectors that can reliably connect into the through holes directly, you might be able do this without any soldering. Otherwise your best options are either to solder in some pin headers, or to solder in some wires to connect to your USB to TTL programmer and for the programming mode jumper. I did not have good probe connectors, and wanted an option that would make it easier to reprogram if required, so I chose to solder in a set of pin headers. The case can be opened with just two screws at the top centre, and then the circuit board will just lift out. The through holes are drilled to fit standard 0.1" (2.54mm) pitch pin header strips, which are easily obtained from hobby electronics stores. Soldering in the pin headers only takes a few minutes for anyone with even modest soldering skills -- I have only soldered a few things before, and was working with a very old and crusty soldering iron, and it still only took me a few minutes and worked first time. (I did use a multimeter to both continuity check for shorts before/after soldering in the pin header, and also check for continuity to the points I traced each header going to on the board to verify my soldering. I also tested the Gotek on my IBM PC clone floppy setup one more time before moving on to programming the firmware, and was glad to see it still worked just as before. If you have not soldered at all perhaps find a friend skilled in soldering and ask them a favour -- it will probably only take one with the right skills and equipment about 5 minutes to solder in a handful of 0.1" pin headers for you.)

Installing FlashFloppy

Once the Gotek circuit board is ready for programming, the next step is to disconnect it from everything else (including disconnect it from power), then connect it up to the USB to TTL serial interface and program in the FlashFloppy firmware to replace the stock firmware.

To do this:

  • Download the FlashFloppy binary installation, most likely the latest version which was 0.9.21a last week when I installed, and seems to be 0.9.22a as I write (0.9.22a apparently adds some more drive formats, and additional display/control functions); I have not tried 0.9.22a myself, and the examples below are with 0.9.21a that I used.

  • From the .zip file you download, extract out the .hex file that is needed for initial serial programming:

    unzip -j flashfloppy_v0.9.21a.zip flashfloppy_v0.9.21a/FF_Gotek-v0.9.21a.hex
    
  • Install the stm32flash tool if you do not already have it installed:

    sudo apt-get install stm32flash
    
  • Connect up the programming jumper and USB to TTL interface to the Gotek board, making sure nothing else is connected to the Gotek. Looking at the back of the Gotek board, with the power connector facing you on the left and the floppy data connector facing you on the right, connect up:

    • A jumper on the left most two pins in the row furthest from the power connector (to put it in "programming mode"). There are some suitable sized jumpers on the configuration jumper section and you can borrow one of those providing you remember where it came from and put it back later.

    • Connect the "RX" lead of your USB to TTL adapter to the pin next to that jumper (the "TX" pin of the ARM STM32F105)

    • Connect the "TX" lead of your USB to TTL adapter to the pin next to the RX one (the "RX" pin of your ARM STM32F105)

    This should give you a row:

    === RX TX o
        o  o  o
    

    with the other pins not used for this purpose (where RX and TX here are the leads from your USB to TTL adapter, and === is the jumper of the left most pins).

  • Connect Ground and +5V to the power connector of the Gotek (these also appear on the pin header pins if you soldered them all in, but the floppy connector is easier to describe; the 3.5" floppy power connector is a 4 pin polarised Berg connector, a LP4 -- note the signal order shown at that page is the order for the female connector on the power supply, and thus is mirror reversed from the male connector on the floppy drive so that they match up correctly).

    Looking from the back of the power connector (with it on the back left of the Gotek, as noted above), you want to:

    • Connect the ground lead to one of the middle pins of the connector (eg, second from the left)

    • Connect the +5V lead to the right most pin of the connector, furthest from the "programming mode" jumper you installed above (and nearest to the 34-pin floppy data connector).

    This should give you:

    o GND o +5V
    

    on the power connector looking at it from the back. Together you get:

    === RX TX o
        o  o  o
    
    o GND o +5V
    

    at the back left of the Gotek, looking at it from the back.

    I used the GND / +5V from my USB to TTL adapter (thus powering it from USB power), and that seemed to work okay (and I have done that for programming other ARM STM32 chips).

    Double and triple check your cabling against the photos on the FlashFloppy serial programming instructions or the Cortex firmware programming instructions to make sure you have your cabling right.

  • When you are happy with your cabling (and that your soldering did not create any shorts!), plus the USB to TTL interface into your computer. You should see, eg, /dev/ttyUSB0 appear after a few seconds.

  • Wait a few more seconds (eg, another 15 seconds) to give the ARM STM32 chip time to initialise into programming mode.

  • Check that the programmer can see the ARM STM32F105:

    ewen@parthenon:/src/gotek/FlashFloppy/release$ stm32flash /dev/ttyUSB0
    stm32flash 0.5
    
    http://stm32flash.sourceforge.net/
    
    Interface serial_posix: 57600 8E1
    Version      : 0x22
    Option 1     : 0x00
    Option 2     : 0x00
    Device ID    : 0x0418 (STM32F105xx/F107xx)
    - RAM        : 64KiB  (4096b reserved by bootloader)
    - Flash      : 256KiB (size first sector: 2x2048)
    - Option RAM : 16b
    - System RAM : 18KiB
    
    ewen@parthenon:/src/gotek/FlashFloppy/release$
    
  • As the Gotek is supplied, the ARM STM32F105 is protected, so you cannot read the firmware back (ie, reading will fail with a NACK) or write to it:

    ewen@parthenon:/src/gotek/FlashFloppy/release$ stm32flash -r gotek-default.hex /dev/ttyUSB0
    stm32flash 0.5
    
    http://stm32flash.sourceforge.net/
    
    Interface serial_posix: 57600 8E1
    Version      : 0x22
    Option 1     : 0x00
    Option 2     : 0x00
    Device ID    : 0x0418 (STM32F105xx/F107xx)
    - RAM        : 64KiB  (4096b reserved by bootloader)
    - Flash      : 256KiB (size first sector: 2x2048)
    - Option RAM : 16b
    - System RAM : 18KiB
    Memory read
    Got NACK from device on command 0x11
    Failed to read memory at address 0x08000000, target write-protected?
    
    ewen@parthenon:/src/gotek/FlashFloppy/release$
    
  • So the next step is to remove the protection, so that the ARM STM32 can be written. This also removes the factory firmware, and so this is the step where you commit yourself to going forward with FlashFloppy or HxC or another third party firmware. To remove the protection:

    ewen@parthenon:/src/gotek/FlashFloppy/release$ stm32flash -k /dev/ttyUSB0
    stm32flash 0.5
    
    http://stm32flash.sourceforge.net/
    
    Interface serial_posix: 57600 8E1
    Version      : 0x22
    Option 1     : 0x00
    Option 2     : 0x00
    Device ID    : 0x0418 (STM32F105xx/F107xx)
    - RAM        : 64KiB  (4096b reserved by bootloader)
    - Flash      : 256KiB (size first sector: 2x2048)
    - Option RAM : 16b
    - System RAM : 18KiB
    Read-UnProtecting flash
    Done.
    
    ewen@parthenon:/src/gotek/FlashFloppy/release$
    
  • At this point it seems like the ARM STM32F105 restarts again, and you will need to wait some seconds for it to be ready again. In my case it also stopped responding to serial programming commands, which apparently is a known issue. So after waiting a couple of minutes for it to work properly, I then unplugged the USB to TTL adapter from the computer, waited 10 seconds, then plugged it back in again. And checked with the identification command (stm32flash /dev/ttyUSB0 as above) that it was responding again before proceeding (after giving it some time to boot into programming mode again).

  • Now the STM32F105 is unprotected, and empty, we can flash the FlashFloppy firmware onto it:

    ewen@parthenon:/src/gotek/FlashFloppy/release$ stm32flash -vw FF_Gotek-v0.9.21a.hex /dev/ttyUSB0
    stm32flash 0.5
    
    http://stm32flash.sourceforge.net/
    
    Using Parser : Intel HEX
    Interface serial_posix: 57600 8E1
    Version      : 0x22
    Option 1     : 0x00
    Option 2     : 0x00
    Device ID    : 0x0418 (STM32F105xx/F107xx)
    - RAM        : 64KiB  (4096b reserved by bootloader)
    - Flash      : 256KiB (size first sector: 2x2048)
    - Option RAM : 16b
    - System RAM : 18KiB
    Write to memory
    Erasing memory
    Wrote and verified address 0x08016ba0 (100.00%) Done.
    
    ewen@parthenon:/src/gotek/FlashFloppy/release$
    

    The process took about 1-2 minutes.

  • I then waited a little while for that to settle, and double checked the STM32F105 was still responding:

    ewen@parthenon:/src/gotek/FlashFloppy/release$ stm32flash /dev/ttyUSB0
    stm32flash 0.5
    
    http://stm32flash.sourceforge.net/
    
    Interface serial_posix: 57600 8E1
    Version      : 0x22
    Option 1     : 0x00
    Option 2     : 0x00
    Device ID    : 0x0418 (STM32F105xx/F107xx)
    - RAM        : 64KiB  (4096b reserved by bootloader)
    - Flash      : 256KiB (size first sector: 2x2048)
    - Option RAM : 16b
    - System RAM : 18KiB
    
    ewen@parthenon:/src/gotek/FlashFloppy/release$
    
  • At this point the firmware programming is done. Unplug the USB to TTL adapter from the computer, and power down the Gotek (if you were powering it separately). Then unplug the USB to TTL adapter from the programming headers, remove the programming jumper (and if you borrowed it from somewhere else, put it back where you found it).

(Note that Firmware Updates of FlashFloppy are much simpler, as it supports programming an update straight from the USB stick; this difficult serial programming process is only needed to replace the factory stock firmware with FlashFloppy the very first time. Updates are basically: copy a .upd firmware update file onto the USB stick, removing all other .upd files, then power the Gotek on with the USB stick inserted and holding down both buttons. When "UPD" is displayed on the 7-segment LEDs, release both buttons, and wait for the Gotek to program itself and reboot into the new version. I suspect this only works to go to newer versions; rolling back to older versions probably requires serial programming or DFU.)

Testing the Gotek with FlashFloppy on an IBM PC clone

FlashFloppy can be connected to an IBM PC floppy interface, or to other micro computer floppy interfaces (most of which are closer to the original Shugart 34-pin interface). I wanted to test my Gotek with FlashFloppy on the same IBM PC clone I had used for testing the Gotek hardware, to make sure the hardware was still working before moving on to connecting it to the Amstrad CPC6128.

To use FlashFloppy on the Gotek with an IBM PC you need:

  • A jumper on the "JC" pins to select IBM PC floppy interface mode (eg, drive change rather than disk ready signals)

  • A jumper on the "S1" pins to select drive 1, as the IBM PC uses a twisted floppy cable and the position on the cable to select the drive letter rather than the drive jumper position -- and I had a twisted PC floppy cable in use.

  • The USB stick formatted as a FAT32 drive (super floppy mode). This is commonly how they arrive from the shop, but not how the default Gotek firmware expects them to be formatted.

  • Copy the FF.CFG example (FF.CFG documentation) onto the USB stick, in the root directory. The example FF.CFG works okay with the IBM PC for an initial test, so no changes are required at this point.

  • Copy a test floppy image (eg, the uncompressed version of the compressed FreeDOS 1.44MB boot floppy, that I used when testing the default Gotek firmware) onto the USB stick as DSKA0000.IMG, eg:

    wget http://www.fdos.org/bootdisks/autogen/FDSTD.144.imz
    unzip FDSTD.144.imz
    cp -p FDSTD.144 /media/ewen/GOTEK/DSKA0000.IMG
    
  • Eject the USB stick from the computer, and plug it into the Gotek.

  • Plug the Gotek unit into the IBM PC clone floppy interface/power, and power the IBM PC on (and thus the Gotek on).

When the Gotek first powers on you should briefly see "F F" on the LED display, followed by a virtual floppy image number (000). All going well, the IBM PC should be able to boot FreeDOS, from the virtual floppy image provided by the Gotek running the FlashFloppy firmware. If that works, the Gotek is successfully upgraded to the FlashFloppy firmware and still supports the IBM PC floppy mode of the factory firmware.

Connecting the Gotek with FlashFloppy to the Amstrad CPC6128

The Amstrad CPC6128 has an external floppy connector ("B:"), which is a 34-pin connector laid out almost like the Shugart 34 pin floppy connector de facto standard. The only significant difference is that the signals are 180 degrees reversed on the Amstrad CPC6128. This means that you need:

  • A straight through (no twists) 34-pin floppy cable (or to use the section of the cable which does not have twists and accept a little loss in signal quality)

  • Which has a 34-pin dual 0.1" IDC connectors (ie, like on the Gotek)

  • And a 34-pin connector to suit the Amstrad CPC, in my case a PCB edge connector.

  • Without any orientation guides, or blocked holes to force you to connect it a certain way, as we need to connect the whole cable 180 degrees out of straight-through alignment.

See this guide to Spectrum +3 related Floppy Connectors for an illustration of the connectors required (Amstrad were also making the Spectrum computers by the time the Spectrum +3 came out). Note that the signal 34 (ready) modifications there are not needed with the Gotek running FlashFloppy as the Gotek can be configured to provide the required Ready signal directly.

I happened to have a lot of old IBM PC floppy cables, which had both 34-pin dual 0.1" IDC connectors (for the motherboard and 3.5" drives) and 34-pin edge connectors (for 5.25" drives on them), and simply used the straight (untwisted) section of that cable. (There is a slight signal degradation in having excess cable hanging off the end beyond what you are using, due to it being after the transmission line terminator. But it seemed to work "okay" for me. Once you have it working a permanent solution would be to cut the final unused connectors off the floppy cable if you are not going to use them for something else; several Amstrad CPC users have invented ways to join signals via those other connectors.)

For me the most time consuming challenge was finding a way to power the Gotek when it was connected to the Amstrad CPC. My plan was to use an external (SCSI) drive case (of which I have several) and a 4-pin Molex (5.25" drive) to 4-pin Berg (3.5" drive) power adapter... of which I thought I had several as well. Unfortunately in reality it actually took me 90 minutes to even find one 4-pin Molex (5.25" drive) to 4-pin Berg (3.5" drive) power adapter around the house. Finding it involved opening a lot of boxes (and PC cases) that hadn't been opened for a decade :-( (Those adapters are still available new, at least from Ebay / AliBaba, so I might pick up a couple more just to have some where I know where they are! Another option would be an IBM PC power supply which comes with the right power connector, but note that most ATX power supplies need some persuasion to turn on without a PC motherboard connected -- and the first few IBM PC power supplies I tried were dead/not working any more. I did also find someone who kludged together a USB charger cable and a 3.5" power connector, to use a standard USB charger to power the Gotek; so with some electronics knowledge that could be an option.)

Having got all the right cables together, to connect the Gotek with Flash Floppy to the Amstrad CPC6128:

  • Power everything off (Amstrad CPC6128, Gotek, etc)

  • Unplug the "JC" jumper on the Gotek (if you connected it for testing on an IBM PC), but make sure the "S1" jumper is connected so it is drive 1 (ie, drive "B:"). There should not be any other jumpers connected.

  • Make sure the Amstrad CPC6128 edge connector is clean (seriously I wasted a lot of time, and had a lot of frustration, because I was trying to test on the old donated Amstrad CPC6128 which had accumulated a lot of dust over the years... and I had only partly cleaned it; the floppy interaction was very confusing with an unclean connector!).

  • Connect the floppy data cable to the Gotek, noting that pin 1 is on the right as you look at the Gotek from the front (ie, from the USB stick side). Connect the red wire of the floppy cable to pin 1.

  • Connect the floppy data cable to the Amstrad CPC6128, ensuring that the red wire of the floppy cable ends up connected to pin 34, which is on the left as you look at the Amstrad CPC6128 from a normal typing position. This puts a 180 degree twist in the whole cable, necessary to make the signals line up due to the Amstrad CPC6128 floppy connector numbering.

  • Plug in the power connector you are using for the Gotek, and turn it on.

  • Plug in the Amstrad CPC6128 power and turn it on.

  • All going well the Amstrad CPC6128 A: drive works okay, and the Gotek shows, eg, "F F" briefly on boot and then, eg "000" and is ready. (If the Amstrad CPC6128 A: drive does not work, chances are the external floppy connector is upside down, as this will ground all the signals, due to one side of the connector being ground, and many of those signals are shared with the internal drive, A:, too.)

Richard Oxley has two good blog posts on the Amstrad CPC6128 floppy interface technical details: HxC Floppy Drive Emulator, and Jumpers! In Adventure with Floppy Cables!. The latter explains how some of the "short this pair of wires" extra options work, including being able to make the external drive replace drive A: (by impairing the drive select line to the internal drive).

Preparing the USB stick for the Gotek on the Amstrad CPC6128

For my purpose of copying Amstrad CPC6128 3" discs onto the Gotek unit, the USB stick for the Gotek needs a bit of advanced preparation because:

  • We want to be very sure which floppy image we are writing to when aiming to archive a set of Amstrad CPC6128 3" disc (sides), because otherwise we might make a great copy of a disc... and then overwrite it minutes later by accident.

  • The only indicator (on stock Gotek hardware) of which floppy image is selected, is the 3 digit 7-segment LED.

  • So we want to have a predictable mapping from the filename on the USB stick to the 3 digit 7-segment number displayed, and vice versa.

  • FlashFloppy can only detect, and write to, disk images that already exist on the USB stick.

  • FlashFloppy can only write to the virtual floppy in the same format (including same sectors) that already exist; it does not support "formatting" a virtual floppy disk with new sectors. Since the Amstrad CPC uses different sector number for "Data" and "System" format discs, this requires some advance preparation to be able to copy both.

So we both need to put FlashFloppy into a predictable order mode, which we do by editing FF.CFG on the USB stick so that it says:

nav-mode = indexed

which means that it expects to find files:

DISKA0000.DSK
DISKA0001.DSK

and so on (including other extensions that it supports), and builds a list of the ones available when it powers up and lets you cycle amongst those -- displaying the last three digits of the filename as the value on the 3-digit 7-segment LED display. So we can now tell exactly which file is selected.

Then we need a bunch of blank disk images, in Amstrad CPC System format (a few) and Amstrad CPC Data format (many), since these both have to exist and have to be in the right floppy format to be able to copy the physical 3" disc directly onto them.

To create these disk images, I used dskform from libdsk to format some blank disk images. I chose that tool as it seemed to default to creating Amstrad CPC standard (ie not extended) .dsk files, which seemed likely to maximise compatibility with other tools later.

I first created a blank Data .dsk image, and a blank System .dsk image:

dskform -format cpcdata -type dsk cpc-data.dsk
dskform -format cpcsys -type dsk cpc-sys.dsk

and then used cpcxfs (written by Kevin Thacker, 2016 download site) to verify the disk formats were as I expected. I got:

$ cpcxfs cpc-data.dsk -s
Found format "DATA" matching disk image format

Image File     : cpc-data.dsk
Image Type     : STANDARD
Format         : DATA (Data Format)

CP/M           : No
Current user   : 0

Capacity       : 180 Blocks = 184320 Bytes
Directory      : 2 Blocks
Allocated      : 2 Blocks = 2048 Bytes =   1.1%
Free           : 177 Blocks = 181248 Bytes =  98.9%

Prompt="cpcfs> ";  Local directory="/tmp"
Verbosity=9;  Page length=25;  Mode=Binary;  Force=-off-

$

and:

$ cpcxfs cpc-sys.dsk -s
Found format "SYSTEM" matching disk image format

Image File     : cpc-sys.dsk
Image Type     : STANDARD
Format         : SYSTEM (System Format)

CP/M           : Yes
Current user   : 0

Capacity       : 171 Blocks = 175104 Bytes
Directory      : 2 Blocks
Allocated      : 2 Blocks = 2048 Bytes =   1.2%
Free           : 168 Blocks = 172032 Bytes =  98.8%

Prompt="cpcfs> ";  Local directory="/tmp"
Verbosity=9;  Page length=25;  Mode=Binary;  Force=-off-

$

so that looked good.

Then I decided to use the 1xx disk images to copy the Amstrad CPC6128 original system disks (two System disc sides -- 1 and 4, and two Data disc sides -- 2 and 3; disc sides 1 and 4 hold CPM/Plus and CPM/2.2 for the Amstrad CPC6128 respectively). And to use 2xx disk images to hold Data disk sides. When I need some more System disk sides (likely as I had some custom boot discs), I will made some more copies of cpc-sys.dsk in the 3xx range.

To set up the USB stick I did:

# 1xx -- Amstrad CPC official boot discs
cp -p cpc-sys.dsk DSKA0100.DSK
cp -p cpc-data.dsk DSKA0101.DSK
cp -p cpc-data.dsk DSKA0102.DSK
cp -p cpc-sys.dsk DSKA0103.DSK

# 2xx -- Amstrad CPC data discs
for SEQ in $(seq -f "%04.0f" 200 299); do
    cp -p cpc-data.dsk DSKA${SEQ}.DSK;
done

And then removed the OS X created files, and ejected the USB stick:

rm -r .Spotlight-V100 .Trashes .fseventsd/; cd; eject /Volumes/GOTEK

Once that was done, the USB stick was ready for the Gotek (with FlashFloppy firmware) to help with archiving the disks.

Archiving the 3" discs

To archive the 3" discs, I plugged the prepared USB stick into the Gotek, turned on the Gotek and the Amstrad CPC6128, and then booted into CP/M Plus using 3" boot disc:

|cpm

The CP/M Plus boot banner should show two drives connected.

Once CP/M Plus booted, I then ran:

disckit3

to start the supplied Amstrad CPC6128 disk copy program. (See also a history of the DiscKit program for the Amstrad CPC and PCW range.)

Then I used the buttons on the Gotek to select drive image 100, the first of the images for the official Amstrad CPC6128 boot disks -- a System disk image. The floppy auto-mounts after a couple of seconds.

The disk image can be copied by choosing:

  • If you are copying something other than the system disc itself, insert it now, and pick another Gotek floppy image number using the buttons on the Gotek. I would very strongly suggest write protecting the source 3" disc!

  • f7 to copy a disc

  • f8 to copy from A: (the 3" internal drive)

  • f6 to copy to B: (the external Gotek)

  • y to confirm the copy should proceed (this is the last chance to check you have the correct Gotek image selected, including one that is of the correct disc formatting type!)

The two drive copy proceeds in a similar manner to the single drive disc copy except that you do not have to swap discs at all. If anything it seems like the Gotek is slower than the physical 3" floppy disc at reading/writing -- but that is understandable given that it is a CPU emulating a floppy drive! (The Gotek feels about 33% slower than a real 3" floppy disc, to me, but it is still very convenient to use.)

The main challenge with this process (other than ensuring the Gotek points at a correctly pre-formatted image before you overwrite it!) is handling the situation at the end when disckit3 tells you to "remove both discs" and press any key. The Gotek does not have an "eject" button (at least by default). I found the easiest solution was to remove the 3" disc in drive A:, and then press the "next floppy" button on the Gotek and then immediately the space bar on the keyboard to tell disckit3 "check now" -- and the floppy was detected as ejected (because the Gotek does not auto mount it for 2 seconds -- leaving a small "no floppy present" window that we need).

Conveniently that process lines us up for copying the next floppy side, and we can just insert the next 3" disc side, and press y to copy it, and so on.

The other important point with this process is to keep detailed notes on which disc image number contains which floppy disc copy, so you can find them again later (and avoid overwriting them). I plan to keep notes on paper, and then once I have copied a batch of discs to turn off the Amstrad CPC6128/Gotek, and copy the images off the Gotek USB stick onto my computer for archiving with a more detailed filename, eg, DSKA0200-Contents_Of_Disk_Label.DSK, so I will be able to identify them later.

Now all I need to do is copy the other 120 or so sides of the 60-ish Amstrad CPC6128 3" discs that I have. With luck that only takes a few hours -- much less time than getting to the point where I was in a position to copy them!

Posted Sat Jun 23 19:25:24 2018 Tags:

Introduction

I have a number of old Debian 32-bit (i386) installations, mostly on VMs, which were originally installed either when 32-bit (i386) was the only option or when 32-bit (i386) seemed like the best option for a small VM due to memmory usage. All of them have been upgraded, repeatedly, through Debian releases and are now running Debian Jessie (8), Stretch (9) or Sid ("unstable").

Because it is becoming obvious that x86 32-bit (i386) is a dead end in terms of software support -- particularly for hardware bug mitigations for issues like Meltdown and Spectre it is clear that these installs will need to be converted to x86_64 64-bit (amd64) installations. Either by reinstalling them (non-trivial for older "pet" installations which have been maintained for many years) or by CrossGrading them between 32-bit and 64-bit.

Today I decided to try CrossGrading a small Debian Unstable test VM that I had -- originally installed many years ago, and apt-get dist-upgraded all the way to the present day -- from 32-bit to 64-bit to find out for myself how complicated it was.

To put the conclusion first: if you have any easy way to reinstall the system, you will almost certainly want to reinstall the system. If the system is important, you probably also want to reinstall the system -- ideally with a "forklift" upgrade where you build the replacement in parallel first -- even if that is not easy to reinstall, as the broken system mid-cross grade will make you very sad.

If you try to cross-grade your system you will end up with a broken system. If you are lucky and experienced with Debian packaging and rescuing broken systems it will only be somewhat broken, and you will be able to fix it over the course of a few hours. If you are unlucky, or not very experienced, then your system may well end up permanently broken. I went into this with around 20 years of experience with Debian Linux as a Sysadmin, and mid-crossgrade the system was still about the most broken I have ever seen a Debian system. (It all worked out in the end, but it took 2-3 times as long as I had originally estimated to get working.)

Because "crossgrading" is an idea whose time is current, there are several other guides around, including:

I would strongly recommend reading several of these guides before starting so you know what you are getting into. As I said above, it is probably easier to reinstall, or install a new system and swap it into place of the old one.

Preparation

Backup

Make a backup before you started. At minimum you will want something you can refer to the configuration files before you started. But you probably want a way to "undo" everything and get back to a working system. If you are using virtualisation that provides a snapshot function use it before you start.

Disk space

You will need a decent amount of free space on your / and /var/cache partitions during this upgrade, because at some points you will have the 64-bit packages for the entire system cached, as well as two sets of packages installed (32-bit and 64-bit packages, especially for libraries). Expect to need several GiB of free space.

In my case the VM I wanted to try upgrading (debian_unstable), had also been installed onto a tiny virtual disk (4GB), which was already tight without even allowing space for cross grading. So I chose to increase its disk size before starting. Because this was the root disk and it was booted from it, it was a little bit more complicated. But modern Linux will cope with online upgrades of a / ext4 file system.

Outside the VM, expand the underlying storage:

sudo lvextend -L 10G /dev/r1/debian_unstable

then bounce the VM so that it sees the new disk space;

sudo shutdown -h now

and start the VM up again from cold.

Next repartition the disk image to use all the new space:

sudo apt-get install parted
sudo parted /dev/vda
# Inside parted:
  print
  resizepart 1
    y    # Continue with mounted partition
    100% # Use entire remaining space
  print
  quit

This will look something like:

ewen@debian-unstable:~$ sudo parted /dev/vda
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  4297MB  4296MB  primary  ext3         boot

(parted)
(parted) resizepart 1
Warning: Partition /dev/vda1 is being used. Are you sure you want to continue?
Yes/No? y
End?  [4297MB]? 100%
(parted) print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  10.7GB  10.7GB  primary  ext3         boot

(parted) quit
Information: You may need to update /etc/fstab.

ewen@debian-unstable:~$
ewen@debian-unstable:~$ sudo fdisk -l /dev/vda
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000c1991

Device     Boot Start      End  Sectors Size Id Type
/dev/vda1  *     2048 20971519 20969472  10G 83 Linux
ewen@debian-unstable:~$

Reboot to force kernel to re-read the partition table:

sudo shutdown -r now

(or run partprobe to force the partition table to be re-read.)

Then resize the mounted partition online (requires relatively recent Linux kernel and tools, eg, the last few years):

sudo resize2fs -p /dev/vda1

which should give you a result like:

ewen@debian-unstable:~$ sudo resize2fs -p /dev/vda1
resize2fs 1.44.2 (14-May-2018)
Filesystem at /dev/vda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vda1 is now 2621184 (4k) blocks long.

ewen@debian-unstable:~$
ewen@debian-unstable:~$ df -Pm .
Filesystem     1048576-blocks  Used Available Capacity Mounted on
/dev/vda1                9951  2932      6569      31% /
ewen@debian-unstable:~$

(note the lack of progress (-p) due to the online resize; fortunately 4GiB to 10GiB is a fairly quick expansion).

One more reboot to ensure that it boots cleanly on the new partition:

sudo shutdown -r now

Grub serial boot menus

To get modern grub (last year or so) to actually show a menu during its timeout, on the serial console, you need in /etc/default/grub:

GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
[...]
GRUB_CMDLINE_LINUX="console=ttyS0,9600"
[...]
# Serial output
GRUB_TERMINAL="serial"
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

then run sudo update-grub, and reboot.

Unlike earlier grub versions, the GRUB_TIMEOUT_STYLE=menu no longer seems to be the default, and without GRUB_TIMEOUT_STYLE set, it will default to the "hidden" behaviour. This is a rather confusing change in default behaviour :-( You will need to be able to pick a custom kernel from the grub boot menu a little later during the cross-grade, so make sure you have a working way to get at the grub menu now.

64-bit hardware

If you are using a virtual machine, make sure that it is providing a 64-bit capable CPU. For my old VMs they were deliberately installed with only the 32-bit capable CPU flags visible, which does not work with a 64-bit kernel/programs (amd64) for obvious reasons.

For instance if you are using libvirt and KVM, use something like virsh edit to replace:

  arch='i686' machine='pc-0.12'

in the os hvm line, with equivalent values for 64-bit:

  arch='x86_64' machine='pc'

If you needed to make any changes, halt the VM and power it on again to make sure that the system still boots with the 32-bit install on the new hardware architecture before proceeding.

Install all pending updates

Make sure your 32-bit (i386) install is fully up to date, and you are running all the updated versions:

sudo apt-get update
sudo apt-get dist-upgrade
sudo shutdown -r now

Crossgrading from 32-bit to 64-bit

There are quite a few steps in the cross-grade process. This process is based extensively on the Debian CrossGrading guide, and Stefan's Upgrade 32-bit to 64-bit blog post, as well as the issues that I encountered on my test machine during the upgrade.

Set up dual-architecture, 32-bit and 64-bit

Enable the 64-bit (amd64) architecture as a foreign architecture:

dpkg --print-architecture
sudo dpkg --add-architecture amd64
dpkg --print-foreign-architectures

giving results something like:

ewen@debian-unstable:~$ dpkg --print-architecture
i386
ewen@debian-unstable:~$ sudo dpkg --add-architecture amd64
ewen@debian-unstable:~$ dpkg --print-foreign-architectures
amd64
ewen@debian-unstable:~$

and then update the package lists so you have both:

sudo apt-get update

You should see the amd64 package lists downloaded too.

Install and boot the 64-bit kernel

sudo apt-get install linux-image-amd64:amd64

which will also pull in a number of other 64-bit packages:

ewen@debian-unstable:~$ sudo apt-get install linux-image-amd64:amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libnuma1
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  apparmor:amd64 gcc-8-base:amd64 irqbalance:amd64 libblkid1:amd64 libc6:amd64
  libcap-ng0:amd64 libffi6:amd64 libgcc1:amd64 libgcrypt20:amd64
  libglib2.0-0:amd64 libgpg-error0:amd64 libgpm2:amd64 liblz4-1:amd64
  liblzma5:amd64 libmount1:amd64 libncursesw6:amd64 libnuma1:amd64
  libpcre3:amd64 libselinux1:amd64 libsystemd0:amd64 libtinfo6:amd64
  libuuid1:amd64 linux-image-4.16.0-2-amd64:amd64 zlib1g:amd64
Suggested packages:
  apparmor-profiles-extra:amd64 apparmor-utils:amd64 glibc-doc:amd64
  locales:amd64 rng-tools:amd64 gpm:amd64 linux-doc-4.16:amd64
  debian-kernel-handbook:amd64
The following packages will be REMOVED:
  apparmor irqbalance
The following NEW packages will be installed:
  apparmor:amd64 gcc-8-base:amd64 irqbalance:amd64 libblkid1:amd64 libc6:amd64
  libcap-ng0:amd64 libffi6:amd64 libgcc1:amd64 libgcrypt20:amd64
  libglib2.0-0:amd64 libgpg-error0:amd64 libgpm2:amd64 liblz4-1:amd64
  liblzma5:amd64 libmount1:amd64 libncursesw6:amd64 libnuma1:amd64
  libpcre3:amd64 libselinux1:amd64 libsystemd0:amd64 libtinfo6:amd64
  libuuid1:amd64 linux-image-4.16.0-2-amd64:amd64 linux-image-amd64:amd64
  zlib1g:amd64
0 upgraded, 25 newly installed, 2 to remove and 1 not upgraded.
Need to get 55.0 MB of archives.
After this operation, 282 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Chances are that if you just updated you will have the same kernel version in 686-pae and amd64 variants installed, and the default update-grub sorting will put the 32-bit (686-pae) one first. Your best option at this point is:

  1. Reboot, and choose the 64-bit version from the grub boot menu, and make sure that it boots. (If it seems to just hang chances are that the CPU architecture has not been updated to be 64-bit (x86_64) compatible, in which case if you are using virtualisation you might need to update the CPU architecture. See the suggestions above.)

  2. Check that you booted the 64-bit kernel:

    ewen@debian-unstable:~$ uname -a
    Linux debian-unstable 4.16.0-2-amd64 #1 SMP Debian 4.16.12-1 (2018-05-27) x86_64 GNU/Linux
    ewen@debian-unstable:~$ uname -m
    x86_64
    ewen@debian-unstable:~$
    
  3. Assuming all is well, uninstall the 32-bit kernel of the same version so that there are not two matching kernel versions, as well as the packages which are dragging it in:

    sudo apt-get purge linux-image-2.6-686 linux-image-686-pae linux-image-4.16.0-2-686-pae
    

    which should leave the newer amd64 kernel, and the older 686-pae kernel, and have the newer amd64 kernel as the default one booted.

  4. Run update-grub and check that the amd64 kernel is now sorted first:

    ewen@debian-unstable:~$ sudo update-grub
    Generating grub configuration file ...
    Found linux image: /boot/vmlinuz-4.16.0-2-amd64
    Found initrd image: /boot/initrd.img-4.16.0-2-amd64
    Found linux image: /boot/vmlinuz-4.16.0-1-686-pae
    Found initrd image: /boot/initrd.img-4.16.0-1-686-pae
    done
    ewen@debian-unstable:~$
    
  5. Reboot again to make sure that it boots by default without intervention:

    sudo shutdown -r now
    

    checking that uname -m returns x86_64 as expected after rebooting.

Clean up package state and ensure everything is in sync

sudo apt-get clean
sudo apt-get update
sudo apt-get upgrade
sudo apt-get --purge autoremove

Convert dpkg, tar, and apt to 64-bit

This is the main cutover of the default architecture, and is the first tricky step. You need to download the packages first, which will also pull in their dependencies. Then install all the relevant packages together, using dpkg directly. Due to the dependencies, I found I needed to do this step twice before all the packages would install and find their 64-bit dependencies.

This step is probably best done as root for safety:

sudo -s
apt-get clean
apt-get --download-only install dpkg:amd64 tar:amd64 apt:amd64
dpkg --install /var/cache/apt/archives/*_amd64.deb
dpkg --install /var/cache/apt/archives/*_amd64.deb

Finally check that the default architecture and foreign architecture have swapped over:

dpkg --print-architecture
dpkg --print-foreign-architectures

which should show you something like:

debian-unstable:/home/ewen# dpkg --print-architecture
amd64
debian-unstable:/home/ewen# dpkg --print-foreign-architectures
i386
debian-unstable:/home/ewen#

Update package lists again, for new default architecture

apt-get update

Now draw the rest of the owl

Most of the instruction guides get a bit hand-wavy at this point, which feels a lot like How to draw an owl. Stefan's blog post contains some useful hints on where to start.

At this point there are a few dozen 64-bit (:amd64) packages installed, and hundreds of 32-bit (:i386) packages installed. But the system thinks it is a 64-bit (amd64) system, that happens to have a lot of foreign packages installed -- and has a kernel which is capable of running both. This means that new packages will default to installing the 64-bit (amd64) version, but the existing packages will stay 32-bit (i386) unless they are manually replaced.

Replacing the 32-bit (i386) packages with 64-bit (amd64) packages is complicated by the fact that apt does not directly understand this replacement -- only that it can install 32-bit and 64-bit packages along side each other if they are libraries, or other packages explicitly designed for multiarch installation (or they are independent packages). So we need to guide dpkg and apt through the conversion process.

Stefan's Blog Post on Upgrading 32-bit to 64-bit recommends downloading packages with apt-get --download-only and installing them directly with dpkg (ie, as above) in stages, to upgrade the system. Unfortuantely I did not find that blog post early enough to see the hint to install dash and bash before changing the default architecture, but given the trouble I had with dash (below) I would agree with the suggestio to get that upgrade out of the way early.

The key download commamd is:

apt-get --download-only -y --no-install-recommends install \
    `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 |
     sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'`

which tries to download 64-bit (amd64) versions of every installed 32-bit (i386) package.

The first complication, at least on a system that has been upgraded between Debian versions -- or Debian Unstable upgraded repeatedly for a long time -- is that there will be quite a few packages installed which are no longer current and thus cannot be installed again. The only reasonable option for those is to remove those old packages that no longer exist in Debian. Finding them mostly involves looking out for errors like:

Package gcc-4.7-base is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

and then unistalling the relevant packages, and the packages which depend on them, with dpkg:

dpkg --purge gcc-4.{6,7,8,9}{,-base} cpp-4.{6,7,8,9} libgcc-4.{7,8,9}-dev libasan{0,1}
dpkg --purge libssl0.9.8 consolekit sysvinit

The second complication was with grub, where grub-legacy was being auto-selected, instead of the newer grub-pc (grub2). Which would then create boot complications. To avoid ths problem explicitly remove the transitional package:

dpkg --purge grub

so that the cross-install happens based on the new package (grub-pc).

The third complication was that quite a few libraries could not be found, because they too had been outdated. These show up with errors like:

E: Unable to locate package libtasn1-0:amd64
E: Unable to locate package libtasn1-2:amd64
E: Unable to locate package libtasn1-3:amd64
E: Unable to locate package libtiff4:amd64
E: Unable to locate package libtxc-dxtn-s2tc0:amd64
E: Unable to locate package libvolume-id0:amd64
E: Unable to locate package libvolume-id1:amd64
E: Unable to locate package libzmq3:amd64

and so on. The fix for those, assuming that they are in fact old unneded libraries, is to remove the 32-bit (i386) version.

dpkg --purge libssl0.9.7
dpkg --purge libdb3 libdb4.2 libdb4.3 libdb4.4
dpkg --purge libgnutls11 libgnutls12 libgnutls13
dpkg --purge libtasn1-0 libtasn1-2 libtasn1-3
dpkg --purge libvolume-id0 libvolume-id1

and so on and so on. Where other things depend on those libraries, those other things will also have to be removed. But usually all of those packages are ancient packages removed from Debian, and either unnecessary or replaced by something else.

Also being hit by this will be the older 32-bit kernel that you may still have installed. You probably also want to remove that at this point as it will very shortly be unsuitable for booting the system (as key packages will be 64-bit, and not run on the 32-bit kernel):

dpkg --purge linux-image-4.16.0-1-686-pae

This will leave you with just one kernel, the latest 64-bit kernel. (As an alternative you could just grep -v that kernel package out of the equivalent packages to install. But beware that you will not be able to usefully boot that 32-bit kernel for much longer anyway.)

The fourth complication is that there might be some old development packages which depend on old libraries which no longer exist, so you might have unmet dependencies. Eg,

libgcc-5-dev : Depends: liblsan0 (>= 5.5.0-12) but it is not going to be installed
               Depends: libtsan0 (>= 5.5.0-12) but it is not going to be installed

and for those that are not relevant, the fix is the same, to remove the packages, and the things which depend on them, so that the dependencies are not an issue and you only have packages which can be installed:

dpkg --purge libgcc-{5,6}-dev gcc-{5,6}

This issue also happens for libgcc-7-dev, which g++, gcc, etc depend on:

libgcc-7-dev : Depends: liblsan0 (>= 7.3.0-23) but it is not going to be installed
               Depends: libtsan0 (>= 7.3.0-23) but it is not going to be installed

and:

dpkg: dependency problems prevent removal of gcc-7:i386:
 g++:i386 depends on gcc-7 (>= 7.3.0-12~).
 gcc:i386 depends on gcc-7 (>= 7.3.0-12~).
 g++-7:i386 depends on gcc-7 (= 7.3.0-23).

as well as binutils:

binutils : Depends: binutils-x86-64-linux-gnu (= 2.30-21)

So the best way forward for those is to explicitly specify the additional packages those packages need in the download list, to be installed.

Once everything seems tidy, download all the needed packages with something like:

apt-get clean
apt-get --download-only -y --no-install-recommends install \
    `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386 |
     sed -e 's/\(.*\):i386/\1:i386- \1:amd64/'` \
     binutils-x86-64-linux-gnu liblsan0 libtsan0A

And then all the packages that you need will be downloaded, but not installed, ending with something like:

Fetched 253 MB in 1min 36s (2,637 kB/s)
Download complete and in download only mode

if you are fairly close to your Debian mirror (and much longer if you are not).

Once the download succeeds, it is worth starting by installing all the 64-bit libraries along side the 32-bit libraries, as generally those can be installed in parallel. It is very useful to include perl packages in this as well, as several perl libraries contain -xs (ie, compiled C) functions, which depend on the perl packages:

dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb

Due to dpkg installing in the order listed, some packages may fail due to other packages not yet being installed (particularly those depending on perl), so it can help to run this same command a second time:

dpkg --install /var/cache/apt/archives/lib*.deb /var/cache/apt/archives/perl*.deb

but there will still be a number of failures due to other dependencies.

However by getting the bulk of the libraries installed in parallel, it will provide most of the required dependencies for other packages.

At this point running:

dpkg --configure -a

will provide a hint as to why the remaining packages could not be configured. These can be solved individually by installing the packages mentioned, eg:

debian-unstable:/home/ewen# dpkg --configure -a
dpkg: dependency problems prevent configuration of libparted2:amd64:
 libparted2:amd64 depends on dmidecode.

can be fixed with:

dpkg --install /var/cache/apt/archives/dmidecode*deb

Some of them will just have failed due to earlier ones failing, so it is worth periodically trying:

dpkg --configure -a

to figure out what still needs manually solving.

I found that:

dpkg --install /var/cache/apt/archives/gcc*base*deb
dpkg --install /var/cache/apt/archives/binutils*deb
dpkg --install /var/cache/apt/archives/linux-libc-dev*deb

unlocked quite a few of the library packages.

And some packages just got stuck because Debian Unstable is, well, Unstable. And thus package versions may not always match up. Eg,

dpkg: error processing package libmagickwand-6.q16-5:i386 (--configure):
 package libmagickwand-6.q16-5:i386 8:6.9.9.34+dfsg-3+b1 cannot be configured because libmagickwand-6.q16-5:amd64 is at a different version (8:6.9.9.39+dfsg-1)

and for some of those I resolved them just by removing the uninstallable packages, in both 32-bit (i386) and 64-bit (amd64) variants, using dpkg --purge. Or just the uninstallable 64-bit (amd64) variant at present if the 32-bit program/library looked important.

You will want to continue manually resolving these issues until:

dpkg --configure -a

runs cleanly.

Then it is a matter of figuring out what programs are installed as 32-bit (i386) that need to be reinstalled as 64-bit (amd64). This can be done with:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit

I chose to install bash and dash next, as I had missed the hint to install those early on:

dpkg --install /var/cache/apt/archives/{d,b}ash*deb

bash seemed to install okay, but dash ran into install problems:

Preparing to unpack .../dash_0.5.8-2.10_amd64.deb ...
Ignoring request to remove shared diversion 'diversion of /bin/sh to /bin/sh.distrib by dash'.
dpkg-divert: error: 'diversion of /bin/sh to /bin/sh.distrib by bash' clashes with 'diversion of /bin/sh to /bin/sh.distrib by dash'
dash.preinst: dpkg-divert exited with status 2
dpkg: error processing archive /var/cache/apt/archives/dash_0.5.8-2.10_amd64.deb (--install):

however /bin/sh was still diverted to /bin/dash:

debian-unstable:/home/ewen# dpkg-divert --list /bin/sh
diversion of /bin/sh to /bin/sh.distrib by dash
debian-unstable:/home/ewen# readlink -f /bin/sh
/bin/dash
debian-unstable:/home/ewen#

Following advice in a blog post with a similar issue I tried manually removing the divert:

dpkg-divert --remove /bin/sh

and tried installing dash again, but ran into issues with sh.1.gz, so did both:

dpkg-divert --remove /bin/sh
dpkg-divert --remove /usr/share/man/man1/sh.1.gz

(because the first one got put back on the next install attempt), and then the installation was able to succeed with:

dpkg --install /var/cache/apt/archives/dash*deb

It appears with only one diversion present (the "background" one of bash), the scripts were then automatically able to cope. I have a feeling this might have resulted from a much earlier Debian Unstable change when the switch over to dash as /bin/sh first happened.

At this point I went back to look what was still installed as :i386 only:

dpkg --get-selections | grep :amd64 | cut -f 1 -d : | tee /tmp/64-bit
dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit

Which I could then turn into a list of packages to install:

dpkg --get-selections | grep :i386  | grep -vf /tmp/64-bit | cut -f 1 -d : |
       xargs -I {} sh -c 'ls /var/cache/apt/archives/{}*_amd64.deb' |
       tee /tmp/packages-to-install

And then install them:

dpkg --install `cat /tmp/packages-to-install`

The majority of them will install without problems, but not all of them. Some of the packages will have issues installing or configuring, which you will need to solve manually.

For the ones that fail, which in my case were:

Errors were encountered while processing:
 /var/cache/apt/archives/python2_2.7.15-3_amd64.deb
 /var/cache/apt/archives/python_2.7.15-3_amd64.deb
 /var/cache/apt/archives/python3_3.6.5-3_amd64.deb
 mercurial
 python-dumbnet
 python-lxml:amd64

trying them again manually in useful sets will probably help. In my case manually installing the python .deb packages again sorted those, and mercurial and python-dumbnet out in one go, just leaving python-lxml.

python-lxml failed because there were two instances of it installed, one 32-bit (i386) and one 64-bit (amd64) during the replacement, which confused its ability to self-compile:

dpkg: error processing archive /var/cache/apt/archives/python-lxml_4.2.1-1_amd64.deb (--install):
 new python-lxml:amd64 package pre-removal script subprocess returned error exit status 1
dpkg-query: error: --listfiles needs a valid package name but 'python-lxml' is not: ambiguous package name 'python-lxml' with more than one installed instance

And unfortunately dpkg --purge would not allow removing either instance, due to the .prerm scripts, which made it difficult to get down to one instance. Looking at the /var/lib/dpkg/info/python-lxml:i386.prerm and /var/lib/dpkg/info/python-lxml:amd64.prerm scripts, the problem was that they attempted to find the installed Python files for that package, to clean them up -- but assumed only one version was installed, due to:

pyclean -p python-lxml
[...]
dpkg -L python-lxml | grep '\.py$' | while read file

and I was able to fix that by manually editing the .prerm files to explicitly specify the version installed in each case (ie, adding ":i386" to the end of the package names in both cases. Then it was possible to do:

dpkg --purge python-lxml:i386

successfully, after which the 64-bit version could be installed again:

dpkg --install /var/cache/apt/archives/python-lxml_4.2.1-1_amd64.deb

And that seemed to be pretty much all packages converted over.

debian-unstable:/home/ewen# dpkg --get-selections | grep :i386 | grep -v lib | wc -l
9
debian-unstable:/home/ewen#

and it looked like most of the remaining non-lib packages were also parallel installed packages.

So next I ran:

apt-get update
apt-get upgrade

to figure out what apt-get considered broken. On my system that returned some 32-bit (i386) packages with unmet dependencies:

You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libespeak-ng1:i386 : Depends: libpcaudio0:i386 but it is not installed
 liblvm2cmd2.02:i386 : Depends: dmeventd:i386 but it is not installed
 libparted2:i386 : Depends: dmidecode:i386 but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

so I just removed the 32-bit (i386) library packages at this point.

dpkg --purge libespeak-ng1:i386 liblvm2cmd2.02:i386 libparted2:i386

and then ran:

apt-get upgrade

again. Which this time told me there were a lot of automatically installed 32-bit (i386) packages which were no longer required, but listed no other errors.

Finishing up

Before continuing I then rebooted the system to make sure it was running the 64-bit (amd64) versions of everything, including the shell that I was logged into, etc.

update-grub
shutdown -r now

Assuming it comes back up again properly, and you can log in, run:

getconf LONG_BIT

to check that the system libraries, not just the kernel (in uname -m) thinks that the system is 64-bit.

Assuming it looks good, go ahead and allow apt-get to remove the now unnecessary 32-bit (i386) libraries:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get --purge autoremove

That will remove some of the 32-bit libraries, but not all of them:

ewen@debian-unstable:~$ dpkg --get-selections | grep :i386 | wc -l
166
ewen@debian-unstable:~$

To remove the remainder, they need to be purged explicitly:

sudo apt-get purge `dpkg -l | grep '^.i' | awk '{print $2}' | grep :i386`

Before saying "yes", double check that apt-get does not want to purge or remove anything else, other than those i386 libraries due to dependencies. Assuming it looks okay, say "yes".

At this point there should be no 32-bit (i386) packages left:

ewen@debian-unstable:~$ dpkg --get-selections | grep :i386
ewen@debian-unstable:~$ dpkg -l | grep "^i.*:i386"
ewen@debian-unstable:~$

Do a final check there is nothing left hanging to do, and a final reboot to make sure all is well:

sudo apt-get update
sudo apt-get dist-upgrade
sudo dpkg --configure -a
sudo update-grub
sudo shutdown -r now

And then you can remove the 32-bit (i386) as a foreign architecture:

sudo dpkg --remove-architecture i386

and you should have a single-architecture, 64-bit (amd64) system:

ewen@debian-unstable:~$ dpkg --print-architecture
amd64
ewen@debian-unstable:~$ dpkg --print-foreign-architectures
ewen@debian-unstable:~$

Final cleanup

At this point it would be worth doing the usual Debian checks, eg:

dpkg --audit
dpkg -l | grep -v "^ii"

and so on to make sure that there are no stuck packages, or half installed/ removed packages.

If you needed to remove any packages in order to proceed with the upgrade you might want to put those back, eg:

sudo apt-get install apt-utils ndiff

(I removed ndiff because of the python-lxml issue above.)

Conclusion

For a relatively small install, this whole process took me about 4-5 hours, including keeping the notes to write this blog post. It might well still have been faster to reinstall. But I wanted to find out for myself how difficult the process was, as I have several other more important old Debian 32-bit installs which I will need to decide whether I am going to cross-grade them or reinstall them. (If you have an easy way to reinstall them -- for instance their configuration is already automated -- I think I would strongly recommend just reinstalling them. Crossgrading is possible, but it is a lot of work to get right.)

Posted Wed Jun 20 14:46:14 2018 Tags:

As part of opening the Amstrad CPC time capsule, I wanted to transfer data off the Amstrad CPC 3" disks that I had, onto a modern PC for archiving. While various methods have been used over the years, including serial cables and parallel cables, it seemed like the easiest modern solution would be to use a Gotek USB Floppy Emulator, running Flash Floppy, connected to the Amstrad CPC as an external drive (drive B:) -- and then use disckit3.com to copy disk sides from drive A: (the 3" drive) to drive B: (the Gotek emulator).

The Gotek USB Floppy emulators are available in a variety of models, from a bunch of online sources, with relatively little documentation. I bought mine from TechExpress who appear to be the main New Zealand importer. It took some weeks until they had stock again and I could order it, but otherwise they were very good to deal with. Mostly for aesthetic resons I bought the black cased model of the SFR1M44-FU. (As far as I can tell other than the external case -- black or beige -- the various models only differ in terms of the firmware loaded onto them by default, and in some cases the floppy connector on the back. It appears the internal components are almost identical, so in theory the model matters relatively little if you plan to replace the firmware before using it -- other than some of the more specialised models selling for higher prices.)

The SFR1M44-FU model appears to be an IBM PC style 1.44MB floppy disk emulator, intended to be connected directly to an IBM PC. I got no documentation or software with it, just the base unit, so I was left to Internet resoures to figure out how to test it with an IBM PC to make sure the hardware was working before I moved on to replacing the firmware (which also includes soldering some programing headers onto the PCB).

The most useful resources I found, before purchasing the unit, were two reviews by Lui Gough of the 100 floppy unit and the 1000 floppy unit; mine was the 1000 floppy unit, but to get it working I treated it as if it was a 100 floppy unit. (Someone also made some Windows CMD scripts which use the same technique described there.)

Those two reviews also linked to a specification PDF (cached copy) of one of the reviewed models which provided a reasonable idea how to it was intended to function. From looking around online it appears the drive was intended to come with a small CDROM with some Windows based tools -- UFDISKManager.exe and UFloppyManager.exe -- for setting up the USB drive, but these do not seem to be easily available from a reliable source, and besides I did not want to need to use a Windows system just to test the Gotek unit. (A couple of posts also suggested using a Windows program from ipcas.com, but that product/program appears to have been retired, and the link is only available in archive.org.)

After a bunch of experimentation and some dead ends I managed to get my Gotek unit to boot FreeDOS 1.2 on a PC clone by:

  • Reassembling a working PC clone motherboard with a floppy drive interface, and a power supply, video card, and keyboard. This step was somewhat non-trivial as many of the PC power supplies and PC motherboards I had left around had died of old age (eg, capacitor issues). I eventually got enough mostly working bits that the PC would reasonably reliably power on, and bring up the BIOS screen.

  • Connected up the Gotek unit to the floppy drive interface/power

  • Found a USB stick that could be completely overwritten. (A couple of GB is plenty; at most there's 1000 * 1.44MB, which is only 1.44GB. I used an 8GB stick but only because the smaller USB keys I had already had important things on them.)

  • Downloaded a bootable FreeDOS 1.2 floppy image, so I had something to test with. I downloaded FDSTD.144.imz.

  • Created a Gotek USB disk image for 100 floppies, following the instructions from Lui Gough:

      dd if=/dev/zero of=Desktop/gotek.img bs=1024 count=153600
      unzip FDSTD.144.imz
      dd if=FDSTD.144 of=Desktop/gotek.img bs=1024 count=1440 seek=0 conv=notrunc
    

    (which puts the floppy image at slot 0; see Lui Gough's blog post for offset values for other slots -- basically 1536 * slot_number).

    NOTE the conv=notrunc is very important, otherwise the Gotek image file will be truncated, eg, to the length of a single floppy.

  • Plug in the USB drive to the computer with Desktop/gotek.img on it, and then ensure that the USB drive is unmounted.

  • Copied that Gotek USB disk image file onto the raw USB drive, ie as if it was an entire disk image:

    sudo dd if=Desktop/gotek of=/dev/sdN bs=4M
    

    where /dev/sdN is the device for the whole USB drive. Triple check you have it right before copying over it, as this will overwrite the start of the drive, and thus all important partition information.

  • When the copy (dd) finishes, unplug the USB drive and plug it into the Gotek. Make sure the Gotek shows 000 as the selected floppy image, and reboot the PC. All going well the PC should then boot FreeDOS. Success!

Another blog post linked to a usbfd program for setting up the USB drive under Linux, and a blog post about some Python scripts to work with the USB drive -- but I did not find these until after I had proved my Gotek unit worked my satisfaction. If you are not used to using dd to work with raw disk images, you might want to investigate one of those options first.

From this investigation it appears:

  • The default Gotek SFR1M44-FU firmware treats the USB drive as a raw block device, without any underlying file system. I did see a suggestion that some models understood FAT file systems and IMG144 directories, with 000.IMG like files -- but the relevant "block mode` option did not appear to work on my model, so presumably that needs different firmware on the Gotek.

  • Individual floppy slots are simply offsets within that raw device. Which means that you need to put the floppy image at the right offset. Those offsets appear to be 1536*1024KB, which is larger than a 1.44MB disk -- but presumably chosen to ensure that they can store maximum track images, not just 80 tracks * 18 sectors/track * 2 sides disk images.

  • The two keys appear to work as follows:

    • right button cycles the lowest digit of the slot number, xx0 ... xx9

    • left button cycles the middle digit of the slot number, x0x ... x9x

    • both buttons together cycle the highest digit of the slot number, 1xx ... 9xx.

So in theory one could store multiple PC 1.44MB floppy images at appropriate offsets on the USB stick and cycle amongst them with the buttons, to boot different floppy based programs. But in practice for a PC it is probably more useful just to use the USB stick directly, as most semi-modern (eg, last 10 years) PCs support booting from a USB stick.

Anyway, getting FreeDOS booting on this old PC this way allowed me to confirm the USB interface, and floppy interface hardware on my Gotek unit was working, before I changed anything. I plan to test it again after soldering on the programming headers, but before flashing it with Flash Floppy as well, just to be sure that adding the programming headers themselves has not affected anything else.

Posted Sun Jun 10 18:31:14 2018 Tags:

Last year, I upgraded my Vodafone cable connection to Vodafone FibreX installation, and then installed a Mikrotik RB750Gr3 (hEX) router as the home gateway. That installation was on RouterOS 6.40.3, which was the latest at the time.

After a bit of configuration tweaking (showing in updates notes in the original blog post, the Mikrotik RB750Gr3 (hEX) worked pretty well. The main issue was that periodically the IPv4 DHCP session would get stuck for an extended period in a "Renewing" or "Rebinding" state, which seems to have been a later 6.xx release issue.

The Mikrotik RouterOS changelog for 6.42.2 (released last week, 2018-05-17) included the very exciting line:

*) dhcpv4-client - fixed DHCP client stuck in renewing state;

which sounded like it might fix this issue, so I wanted to upgrade.

However there was also another major change introduced with 6.41:

Important note!!! Backup before upgrade!
RouterOS (v6.40rc36-rc40 and) v6.41rc1+ contains new bridge
implementation that supports hardware offloading (hw-offload).

This update will convert all interface "master-port" configuration
into new bridge configuration, and eliminate "master-port" option
as such.

Bridge will handle all Layer2 forwarding and the use of switch-chip
(hw-offload) will be automatically turned on based on appropriate
conditions.

The rest of RouterOS Switch specific configuration remains untouched
in usual menus for now.

Please, note that downgrading to previous RouterOS versions will
not restore "master-port" configuration, so use backups to restore
configuration on downgrade.

which was the main reason I had delayed upgrading from 6.40.3 while other travel was going on. (I was aware of various attacks on Mikrotik default platforms, but I'd deliberately turned off most services, and implemented a strict firewall to minimise those risks.)

I implemented this upgrade over the weekend, and eventually got a working configuration again -- but it did not happen automatically, and required several configuration changes made via mac-telnet to make it work at all let alone equivalent to the older configuration. While investigating what was needed to make it work, I found limited documentation available, so I wanted to document what was required to make it work again.

At a basic level the configuration changes are:

  • The interface ethernet master-port=NNN option is now gone, which means there's no direct way to just group switch ports together on Mikrotik Routerboards -- you have to use a interface bridge instead.

  • The interface bridge port functionality has a hw=yes option available, while enables that port to try use the switch chip for hardware acceleration if the bridge features required are simple enough that the switch chip can do it.

So for a simple switch usage like my "group lan ports together in hardware" there is equivalent functionality, but it is not immediately obvious how to make it work -- and changes are required in many other places that reference the interfaces.

Doing the RouterOS upgrade is the same as usual, but for this particular upgrade there's a high likely hood of your "LAN" interfaces no longer working correctly until you make more configuration changes (at least if you have something different from the Mikrotik RouterOS out of the box defaults), so I'd suggest making sure that mac-telnet also works for you (I used MAC-Telnet from Mac OS X 10.11 which I compiled myself).

The basic upgrade is then:

  • Make sure mac-telnet logins work for you, as well as ssh logins, to the Mikrotik

  • Log into Mikrotik

  • Export configuration: /export name=mikrotik-rb750gr3-config.rsc

  • Download a copy of the mikrotik-rb750gr3-config.rsc to your desktop for reference (you will need to search it for LAN interface names later), eg, scp A.B.C.D:mikrotik-rb750gr3-config.rsc .

  • Download the appropriate version of RouterOS (eg, mmips for the RB750Gr3 / hEX)

  • Verify the checksums for the downloaded file against the checksums link in the download page

  • Upload the routeros-mmips-6.42.3.npk file to the router, eg, scp routeros-mmips-6.42.3.npk A.B.C.D:

  • Log into the Mikrotik, and do a soft reboot (the upgrade happens during the restart cycle only, so a power cycle will not trigger it): /system reboot

  • Wait for the "beep" start of boot and "beep beep" fully booted indications (which for a software upgrade take 1-2 minutes to arrive; much longer than for a normal reboot).

If you are lucky, ssh logins to the Mikrotik will work again, and you will be able to reach the Internet. That indicates most/all of the conversion was done automatically.

If not, like me, you may find that you cannot reach the Mikrotik via ssh or ping, nor can you reach the Internet, in which case manual config changes will be required. At this point you will be very glad you tested mac-telnet access in advance.

To fix things up to work with 6.41+ (eg, 6.42.3) in the same way as before:

  • Mac-telnet into the Mikrotik and log in. While there, verify that your Internet connection did actually come up, eg:

    /ip dhcp-client print detail
    /ipv6 dhcp-client print detail
    /ping IP-ON-INTERNET
    
  • Check if a bridge got automatically created, and if not, you will need to create one:

    /interface bridge print
    /interface bridge add name=lan
    
  • Make sure the bridge is created without spanning tree enabled (ie, no STP/RSTP/MSTP) as the switch chip cannot handle that, and so using those features will force slow software switching. The default mode is RSTP, which will turn off hardware switching on several chipsets, including the RB750Gr3/hEX one.

    /interface bridge set protocol-mode=none
    /interface bridge settings set allow-fast-path=yes
    

    (The switch chipset in some higher end models can support RSTP in the switch chip, in which case the default protocol-mode=rstp might be okay on those models. But it took me a long time to figure out that protocol-mode=none was essential on the RB750Gr3/hEX to get hardware acceleration back.)

  • Add each LAN ethernet port into the bridge, and update their comments to match:

    /interface bridge port add bridge=lan interface=ether2
    /interface bridge port add bridge=lan interface=ether3
    /interface bridge port add bridge=lan interface=ether4
    /interface set ether2 comment="LAN (bridged)"
    /interface set ether3 comment="LAN (bridged)"
    /interface set ether4 comment="LAN (bridged)"
    

    Note that adding ports into the bridge will change those ports around in a way that will break the existing connections, so you may have to abandon your first mac-telnet session and start a new one at one point in that process. That was a bit disconcerting, but expected from mac-telnet in other situations. Fortunately it reconnected easily.

  • At this point, doing:

    /interface bridge port print
    

    should show "yes" in the "HW" column (to indicate they are capable of hardware acceleration) and a leading "H" in each line to indicate hardware acceleration is actually being used. Eg,

    [ewen@naos-rb750gr3] > /interface bridge port print 
    Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
     #     INTERFACE     BRIDGE        HW  PVID PR  PATH-COST INTERNA...    HORIZON
     0   H ether2        lan           yes    1 0x         10         10       none
     1   H ether3        lan           yes    1 0x         10         10       none
     2   H ether4        lan           yes    1 0x         10         10       none
    [ewen@naos-rb750gr3] > 
    
  • Update all the firewall rules that refer to ether2 (the old LAN switch master-port) to refer to the new bridge interface. You can find the affected rules by doing:

    /ip firewall filter print
    /ipv6 firewall filter print
    

    and looking for red comment text like:

     34 I  ;;; in/out-interface matcher not possible when interface (ether2) is slave - use master instead (lan)
    

    where the "I" means the rule is "invalid", and thus being ignored (hence no access via ssh or to the Internet). Something like:

    /ip firewall filter print where invalid
    /ipv6 firewall filter print where invalid
    

    might help.

    Find the affected rule numbers, and note if they are in-interface=ether2 or out-interface=ether2 and then update the in-interface=... or out-interface=... parts of the rules with something like:

    /ip firewall filter set 14,24,25,26,34 in-interface=lan
    /ip firewall filter set 28,49,50 out-interface=lan
    /ipv6 firewall filter set 21,22,23,37,38,45 in-interface=lan
    /ipv6 firewall filter set 33,40,47,57,58,59 out-interface=lan
    
  • At this stage you should be able to ssh back into the Mikrotik again, and reach the Internet. Test that works.

  • It would be a good idea to make another config export at this point, and download that, as a work in progress:

    /export name=rb750gr3-6.42.3-partially-working.rsc
    

    as then you can search that for stray references to ether2.

  • Clean up the stray references to ether2 that you can find, by replacing them with the lan bridge interface. I found:

    /ipv6 address set 0 interface=lan
    /ipv6 nd set 1 interface=lan
    /ip address set 0 interface=lan
    /interface list member set 0 interface=lan
    /interface list member add list=mactel interface=lan
    

    plus some references in /system script entries I had created, which needed updating, and a need to remove ether2, ether3 and ether4 from /interface list member. Amongst other things this moves the actual IPs over to the lan bridge interface (which on IPv6 assigns another prefix from the pool; on IPv4 there will be a short service interruption as the LAN IP moves).

    You might want to search for ether3 and ether4 as well; in my case I only found those in /interface list member (tidied up above), and in the /interface bridge port configuration (which obviously needs to stay).

  • Test ssh, and mac-telnet, to the Mikrotik work. Test access to the Internet.

  • Assuming it looks good, I'd suggest rebooting the router again to make sure everything starts cleanly:

    /system reboot
    

    and re-testing access again.

  • If that is also good, explicitly upgrade the Mikrotik RouterBoard firmware:

    /system routerboard upgrade
    /system reboot
    

    and check access again after rebooting. At this point:

    /system resource print
    /system routerboard print
    

    should (as of 6.42.3) show version 6.42.3 as the active version (for both; older Mikrotik had a different numbering scheme for the RouterBoard firmware, but recently they moved to using the same numbering system for both).

Hopefully at this point you are both on the latest RouterOS/Firmware and have a working configuration and have hardware acceleration on the LAN (switch) bridge again. The upgrade process definitely could have been simpler/more automated, particularly for a "minor" version upgrade. But I am glad that mac-telnet worked, and that Mikrotik did provide a warning that this was a non-trivial upgrade (even if that warning is now increasingly buried down in the changelog...).

Posted Mon May 28 14:46:24 2018 Tags:

Repairing the Amstrad CPC6128 floppy drives

The Amstrad CPC6128 (and Amstrad CPC664 for about 6 months before the CPC6128), was supplied with a relatively rare 3" (not 3.5") single sided floppy drive. They were used in a few other computers, including the Amstrad PCW82xx range, and some Sinclair Spectrum computers (Amstrad bought Sinclair's computer assets in the mid-1980s).

Thanks to a generous donation by a LCA2018 attendee, I now have two Amstrad CPC6128 units -- my original one, and a second one that had spent 20 years sitting in someone's garage (it was very dusty and needed a lot of cleaning!).

Because the 3" drives are relatively rare, both the 3" floppy discs (CF2) are fairly rare and rather expensive, and the drives need to be repaired, rather than trying to find a replacement drive. The drives essentially only exist in 30+ year old machines, and are likely to need repair themselves. The 3" floppy discs were over NZ$10 each new in the 1980s, and used 3" floppy discs are still around NZ$10 each second hand in 2018; I bought five (5) "new old stock" -- ie, unopened shrink wrap -- 3" discs earlier this year for NZ$75, which seemed about the going rate.

Other than general cleaning of the floppy drive, of 30+ years of accumulated dust and grime, they also need lubricating the stepper motor worm drive, and inevitably the floppy drive belt needs replacing. The floppy drive belt runs between the capstan connected to the motor for the spindle (near the back of the drive) and the drive spindle (in the centre!) to spin the disc, and is made of thin (about 0.5mm rubber) and over time dries out, goes tacky, and breaks (like most rubber).

DataServe Retro, in the United Kingdom provides an excellent service supplying replacement parts/kits for servicing the drives, as well as detailed instructions on servicing the drives. A few other people have also posted their instructions on repairing the drive, and there is also a video demonstration of the most common problem with the Amstrad CPC6128 drive, and replacing the drive belt to repair it. (See also repairing an Amstrad FD1 external drive, and replacing the Amstrad CPC6128 drive write protect sense pin; also for completeness an Amstrad CPC464 overview from the same site.)

I had previously tried to replace the drive belt in my original Amstrad CPC6128 in 2004, with a drive belt purchased locally based on trying to measure the (broken) disc belt -- and it almost worked, but only sort of worked if the drive was already spun up to speed, so it required a lot of "R"(etry) attempts to do anything. That unreliability was the main reason why I did not try to copy the 3" discs back in 2004 when I imaged a lot of other floppy discs.

Given that I now had two Amstrad CPC6128s, neither with a working disc drive, I decided it was worth trying to replace both drive belts. Due to the distance and international postage costs, I bought the Amstrad CPC6128 drive servicing kit and a twin pack of drive belts (for a total of three drive belts) as well as a spare write protect pin just in case (which turned out not to be needed; I believe one of my drives does use the write protect sensing pin, but I was very careful not to turn the drives over during the belt replacement, having been forewarned of problems, so it appears the existing pin stayed in place).

I also bought some second hand CF2 3" discs, and another set of Amstrad CPC6128 original system discs (CP/M Plus, CP/M 2.2, utilities, etc) just in case my originals were unreadable due to age. (It appears they are now sold out of second hand CF2 3" discs.)

In both cases, replacing the drive belt was fiddly, but possible. I had two different 3" drive models (archived version) to work on in the two systems (assembled about 9-12 months apart), but the same drive belt worked for both of them, despite the fairly different internal geometry. Based on the pictures I believe the drive in my original Amstrad CPC6128 is an EME-150A (archived version; the original model, manufactured by Matsushita); the drive in the Amstrad CPC6128 I was given was labelled as an EME-156 (archived version; which does use a mechanical sensing pin for write protect; in hindsight, I did see the sense switch, but I did not see the pin itself -- fortunately I was very careful not to turn the drive upside down while it was open, and writing to the discs still works, so apparently the pin is still in place).

My original Amstrad CPC6128

The locally obtained (2004) replacement (wrong size) drive belt, in my original Amstrad CPC6128, was fortunately still intact, so I had very little cleaning to do on that drive and it was a fairly quick replacement. The drive appeared to work properly immediately after reassembling it with the correct drive belt replacement, which was extremely pleasing -- I was able to copy the spare system discs onto one of the spare floppies immediately without errors.

Comparing the locally obtained drive belt from 2004 with the exact replacement part from DataServe Retro, I found that my guessed replacement part was 10mm too long (circumference) and 0.5mm too wide. I assume my original drive belt had stretched in addition to breaking, before I measured it, causing me to overestimate the required length of the drive belt (I had ordered a 72mm x 0.5mm x 3mm flat belt in 2004; I believe the the 72mm is a notional diameter, assuming the belt is laid out as a circle. That dimension probably relates to the manufacturing process.)

As best I can tell from measuring it, the exact replacement belt is about 70mm x 0.5mm x 2.5mm -- about 2mm smaller in diameter, which works out to 7-10mm shorter in circumference (it may be a notional 69mm, but I think it is 70mm; I measured it as about 220mm circumference). That explains why my guessed replacement drive belt in 2004 sort of worked -- it was slipping a bit, but if the disc was up to speed it was almost close enough to keep it spinning. (Ironically if I had realised at the time I could have ordered the slightly shorter belt; I suspect the 2.5mm/3mm width difference might work as the capistan is pretty wide, but the 2.5mm width fits the spindle better. Since 2004 the local distributor of drive belts I used appears to have gone out of business, or at least taken their ordering website down.)

Second (donated) Amstrad CPC6128

The second Amstrad CPC6128 -- very dusty from 20 years in a garage -- was more difficult to get working properly. The drive belt had snapped, but fortunately just in one place, so it was fairly easy to clean the capistan and replace the drive belt. However, getting the head stepper motor/worm drive working properly was rather more difficult, and after a small amount of use the drive started squealing loudly at times when the disc was spinning. I ended up:

  • taking the drive apart a second time, and cleaning it again, including taking the drive belt off again and cleaning the capistan/spindle a second time;

  • lubricating the stepper motor worm drive a second time;

  • lubricating the top mounting point pressure plate of the drive spindle and the botton of the spindle mount point, as well as the capistan itself

  • exercising the drive a lot to encourage the stepper motor worm drive to pick up the lubrication added.

Initially after the first cleaning/drive belt replacement the head stepper motor was totally stuck. It took it took multiple very noisey attempts to use the drive before it managed to break free of its stiction and get back to track 0 so "cat" would work. After that it took several more attempts to get it to even step to the second track reliably, and many more attempts to get to the point where it could format/verify/copy an entire disc and sound like it was working properly.

After several hours of effort it appears the second drive is pretty much working reliably, but I do not yet trust it as much as my original Amstrad CPC6128 drive given how long it spent accumulating dust in a garage.

For cleaning I just used isopropyl alcohol (fairly easily obtained from electronics supply stores), and cotton buds. For lubrication I used Abloy Lock Oil, mostly because I already had it and it is designed not to pick up additional dust in use (DataServe Retro recommend a grease like vaseline, on the stepper motor worm drive, but I did not have any at hand). I guess time will tell whether I made the right choice on lubrication (the main thing I would stress is to use small amounts of lubricant -- the last thing you want is excess lubricant spun out over the floppy discs during use!).

Exercising the Amstrad CPC6128 floppy drive

From BASIC

From BASIC the main way to exercise the floppy drive is to try loading or saving BASIC programs from the floppy disc. To do this I used one of the secondhand CF2 3" discs that I had bought, so I could test with a disc that I knew did not contain anything important. (While it would have been possible to attempt to read from a disc with existing data on it, I was reluctant to trust a disc with existing data on it to the "not yet working properly" drive, so stuck with just trying simple cat/load/save commands on known reformatted disc.)

My initial attempts with the dusty Amstrad CPC6128 were:

  • get a disc catalogue -- with cat -- to force the drive to read from track 0 (and thus pulll the drive head back to track 0)

  • write a tiny BASIC program (eg, a few lines) and then save that to the disc under multiple names:

    save "test1
    save "test2
    save "test3
    

    and so on. It appears the fourth or fifth copy of a 1KiB program saved involves stepping to the second track, which proved problematic for a while (and at one point it appeared to write track 1 over track 0, which caused "cat" to no longer work on that disc -- eventually fixed again by reformatting that side of the disc again after I had the drive working better; see below).

  • try loading BASIC programs back off the disc (eg, reset and load the tiny program I had written).

Doing this repeatedly got the drive to the point where it could move the head enough to read in the CP/M boot disc (which it was not able to do so at first). Then I was able to exercise the drive more from CP/M.

From CP/M Plus

On the Amstrad CPC6128, the only official way to format or copy a whole disc is from CP/M Plus, which needs to be booted from disc -- the first system disc supplied with the Amstrad CPC6128 (or in my case also bought as a second hand copy). It took me some time to find/remember how to format discs or copy them, and it does not appear to be very well documented online, so I am recording the instructions from the Amstrad CPC6128 manual -- Chapter 1 Pages 39-42 -- here for future reference (by me, or someone else).

To boot up CP/M plus, insert side 1 of the system discs, and then type:

|cpm

where the "|" introduces a RSX command -- basically a extension command for AMSDOS, the disc system for the built in Amstrad CPC BASIC; there were about a dozen commands built in to the system ROMs, and lots of third party ROMs, discs, peripherals, etc, defined their own RSX commands.

Typing "|cpm" causes the Amstrad CPC6128 to read CP/M off the disc in the first disc drive -- which is expected to be a "System Disc" -- ie with boot sectors, and a copy of the CPM system image. Actually booting CP/M Plus involves reading about 30KiB off the disc, which requires reading multiple tracks -- so on the rather dirty Amstrad CPC6128 drive it took multple attempts and other drive exercise/cleaning before it would seek well enough to boot CP/M Plus.

Once CP/M Plus has booted to the prompt announcing it is drive A, it is possible to work with whole floppy discs by running:

disckit3

which is a menu-based program for copying, formatting, and verifying floppy discs. It can work either with one disc drive (in which case copying discs involves changing discs several times) or with two disc drives (if you have an external drive).

From the disckit3 menu, you have the choices of:

  • f7: Copy disc

  • f4: Format disc

  • f1: Verify disc

  • f0: Exit from program

These are selected using the function keys at the right hand side of the keyboard, even though the menu displayed on screen shows just 7 / 4 / 1 / 0 -- you just have to know to use the function keys :-( Invalid keys just result in a system bell beep as the only feedback, so it was initially very confusing until I found the relevant manual pages and reminded myself to use the function keys at the right.

For exercising the drive (ie, the dirty problematic Amstrad CPC6128 drive) the best options are:

  • Verify disc (which will try to read the whole disc); and

  • Format disc (which will reformat each track of the disc)

both of which involve seeking to every track on the disc. I did both about a dozen times on the problematic drive, and eventually -- after a second cleaning -- it seemed to start moving freely enough to work properly. (I believe "Verify disc" only checks that it can read each sector/track of the disc, without actually checking the contents beyond the sector checksums; but for forcing the drive head to step it is ideal if you have an already formatted disc to test with.)

When formatting a disc you have three choices of format:

  • f9: System format

  • f6: Data format

  • f3: Vendor format

  • .: Exit menu (back to the top menu)

The system format, data format, and vendor format use different sector numbers, but have the same native capacity -- 180KiB. The system format has a copy of CP/Ms boot information on it (and when formatting a system disc it will first ask for an existing system disc to read the required system tracks); the vendor format is the same as the system format, but without the system tracks initialised (just the space reserved). The data format uses different sector numbers, and makes most of the native space available for storing data (178KiB usable, with 2KiB reserved for directory information; by comparison the system and vendor formats have 169KiB usable).

Once I had exercised the problematic drive enough that it appeared to be working reliably, I then copied the original CP/M Plus system disc that I had purchased onto one of the second hand CF2 3" reformatted discs, so that I could use that instead of the originals (this is also recommended early on in the Amstrad CPC6128 manual).

To do this, boot CP/M Plus and start disckit3 (as above), then:

  • Insert the CP/M Plus system disc (if it is not already inserted)

  • Press f7 to Copy a disc

  • Press "Y" to confirm you want to start copying the disc

  • On a single drive system, disckit3 will now read the first 15 tracks into RAM (67 KiB, which interestingly is larger than the 61KiB CP/M TPA, so I assume it is also buffering some of the data in the second 64KiB RAM bank...)

  • When prompted to insert the disc for WRITE, remove the original and insert the blank disc to be overwritten, then press a key to continue. It will write out the first 15 tracks to the new disc.

  • After the first 15 tracks are written, it will prompt you to insert the disc for READ, and then when you press a key to continue, it will read the next 15 tracks into RAM, and prompt you to insert the disc for WRITE. When you swap discs and press a key to continue the next 15 tracks are written out.

  • On the final pass, you are requested to insert the disc to READ, and when you press a key the final 10 tracks are read into memory, and then written out to the copy after you insert the disc to WRITE and press a key.

If you are doing this on a single drive machine it is very good practice to write protect the READ disc, and write enable the WRITE disc, to minimise the chances of inserting the wrong disc at the wrong time in the six disc swaps. (The program does check that you removed/inserted a disc, but obviously cannot easily check which disc you inserted!)

With a second hand disc and second hand drive, using the "Verify disc" function at the main menu after copying the disc, to check that the copy is readable, is probably also a very good idea.

(From memory with two disc drives connected, it will simply copy all tracks from A: to B: without needing further user intervention beyond putting the discs in the right drives and confirming you want to overwrite the destination disc. No disc swaps required, which means the chances of inserting the wrong disc at the wrong time are pretty low; but I would still suggest write protecting the source disc just to be safe!)

Alternative disc formatting

While there is no user accessible routines to format a disc from AMSDOS, there are some hidden RSX functions, which can be called from machine code, to format a disc, using a combination of Seek Track and Format Track. (They cannot be called directly from AMSDOS, as their names are non-ASCII characters.)

While hunting for a way to reformat floppy discs on the Amstrad CPC6128 (partly to exercise the drive, partly because non-working seeking had caused track 1's information to overwrite track 0 on one of my test blank discs), I came across a blog post providing a BASIC listing that loaded a RSX to format a disc from AMSDOS (archived version. It is a fairly short assembly program, with a HEX loader in BASIC, small enough that it could be typed in by hand if necessary. I have not used it as I eventually got CP/M Plus to boot even on the partly working drive, so used disckit3 from the CP/M Plus system discs instead. But it could be useful for systems without any working system discs. (The blog post includes a simple, uncommented, disassembly of the source code, which does not confirm it is using the hidden RSX functions, but given the short length that seems likely. There is also a format.dsk linked from this CPC forum thread, apparently including source, which I asume is likely the same one.)

Of course given that the Amstrad CPC has no memory protection/IO controller, at all, with care (to avoid anything else accessing the drive) it would also be possible to send raw seek/track format commands to the NEC 765 floppy controller directly. I suspect that would also require some machine code assistance, to issue IN and OUT instructions (see also Z80 Assembly Guide and other Z80 Programming Information; Locomotive BASIC has PEEK, POKE, to access memory, but not anything to access the IO bus. I assume direct instructions to the NEC 765 floppy controller is how the various alternative disc format tools worked.

Posted Sun Mar 18 17:04:18 2018 Tags:

Today, "Do Not Reply" at the Inland Revenue Department (IRD) got in touch (again) to let me know that they were updating the look of their website for filing GST returns:

What’s changing?

Here’s a summary of the myIR changes:

* The ‘My GST’ section will change to ‘My business’.

* The design, layout and tab labels will change to make it
  clearer and easier for you to find your way around and use our
  online services, as well as give you assurance of where you are
  in the process. You’ll notice we’ve:

  * Created more whitespace

  * Increased the text size

  * Changed the tabs on the first page of the new ‘My business’
    section.  They will change to: Accounts, Submitted (where
    you can see any saved drafts, submitted or processed returns),
    Correspondence (messages or letters previously under ‘Activity
    centre’), Registration details and Logons

  * Changed ‘Quick links’ to be 'I want to...' 

* Continue to file, pay and amend your GST.

and advises "Information about these new changes is also available on our website." (archived version) for which the description of the changes is also online (archived version) and it does include a few non-GST related changes.

The email goes on to advise that there will be four and a half days of downtime for the IRD website to make these changes:

Some of our services will be unavailable while we make these changes

In order to make these important changes some of our key services
will be unavailable between the afternoon of Thursday 12 April
and the morning of Tuesday 17 April. During this time you will
not be able to access myIR Secure Online Services or contact
us through our contact centres. Any secure mail messages saved
as drafts and any draft returns within myIR will be deleted as
part of this process. Please therefore check your secure mail
messages and submit any draft returns before Thursday 12 April.

Those date are confirmed on the IRD website (archived version), and repeated on a second IRD web page (archived version) whose URL hints that they are treating this as a fork lift upgrade.

As someone who works in the IT infrastructure industry, and watches the technical blogs of major online technical sites, that appears to be an outage window at least four days longer than is common in modern IT practice, particularly for an upgrade that is described to end users as changing the names/ordering of some tabs on a website, and updating the layout. (Many high tech online sites do regular A/B testing of site layout changes all the time, in production, without any interruption.) Even major banking web sites, which do periodically announce outages, seem to manage to do their work with only 12-24 hour maintenance windows.

It is also conventional to schedule an extrememly long maintenance window on a public holiday long weekend, such as Easter occurring in a couple of weeks, rather than on a random weekend where a four and a half day outage inevitably ends up crossing at least two business days (thus resulting in the need for a lot of business process to manage being effectively "closed for business" on a regular business day).

To add to the fun, this apparently major upgrade -- four and a half days, presumably ending in a mainframe being wheeled out the door -- is being done immediately after the end of the financial year for most New Zealanders and New Zealand businesses. If it is, as it appears, a bigger upgrade than "changed the names of some tabs" and "updated the layout", then one would hope that IRD is very very certain that the new version works perfectly; major upgrades are usually scheduled for a quieter time of year where possible, for good reasons!

I guess I will not be filing my GST return in the middle of April....

Posted Thu Mar 15 11:11:55 2018 Tags:

Introduction

Almost 32 years ago (on 1986-04-30), after much begging, my parents bought an Amstrad CPC6128 with an Amstrad GT65 Green Screen Monitor from Porterfield Computers Ltd in Wellington. Together with a word processor (Tasword 6128), the Pitman Typing Tutor, and two 3" Floppy Discs (narrower than the much more common 3.5" floppy discs), this all cost NZ$1500.65 -- in 1986 dollars (over NZ$3500 in 2017 dollars, according to the Reserve Bank Inflation Calculator; also packing slip from Grandstand). An Amstrad DMP-2000 printer came a week later, for NZ$695.00 (again 1986 dollars; over NZ$1600 in 2017 dollars), and about a year later an Amstrad FD1 3" external second drive for NZ$299.00 (in 1987; around NZ$700 in 2017 dollars).

For a middle class family in New Zealand in the 1980s that was quite a bit of money. It changed my life forever. While it was originally a "family" computer, it fairly rapidly became my computer, because I was the one using it all the time.

I learnt to touch type on the Amstrad CPC, learnt to program on the Amstrad CPC, and even learnt about computer hardware from expanding the Amstrad CPC6128. For about 6-7 years the Amstrad CPC6128 was my "everyday" computer -- a long time for any computer, and an especially long time when the personal computer market was changing so rapidly. Over the years I added two Dk'tronics 256KiB Memory Expansion modules (for a total of 576KiB -- bank switched on a 8-bit microprocessor! -- as the first 64KiB of the expansion overlaid the second 64KiB on the Amstrad CPC6128), an Amstrad Serial Interface, an external 5.25" floppy drive and even a MFM hard drive via a SCSI controller board and a home built SCSI adapter.

The Amstrad Serial Interface (and the poor software with it, that could not even keep up with 2400bps let alone display IBM PC extended characters and formatting) led to me write EwenTerm, the largest Z80 Assembler program that I wrote (and one I used daily for several years to keep up with multiple BBSes) -- which also changed my life forever). EwenTerm's source was also the basis of Glenn Wilton's BBS Terminal, developed by another Wellington, New Zealand local. Glenn took the source I wrote and polished it into a more useful communication program, including file transfers (EwenTerm development basically stopped once I had reliable "ANSI terminal" functionality working).

(For several years I had "EwenTerm" online with a URL that suggested it was "ansiterm" -- a URL I had chosen because the program did not really have a name and was written to be an ANSI terminal. More recently this has led to some confusion, as there was an AnsiTerm by Paul Martin, which was well reviewed in Amstrad Computer User Issue 4 on pages 48 and 49, but seems to be a different program, independently developed at about the same time. Unfortunately I appear to have added to the confusion by responding to an email from Kevin Thacker saying "I would like to contact Ewen McNeill concerning his program for the Amstrad CPC called 'Ansiterm'." with the link to my software, resulting in an AnsiTerm page on the CPC that links to both my EwenTerm source and the review about Paul Martin's AnsiTerm. I have emailed Kevin Thacker again to try to get this corrected, and it appears to be corrected there too now.)

The hard drive leads us to our story today. Around 1989 I bought a second hand ST-506 interface MFM hard drive to add to my Amstrad CPC6128. It was 10MB, and even second hand cost a few hundred dollars. The Amstrad CPC6128 did not have a hard drive interface (although various third parties created some later), but I eventually found out about SCSI to MFM Controller cards and ordered one of those second hand (from the USA). Then to link the Amstrad CPC6128 to the SCSI 1 interface I built a simple adapter card that plugged into the Amstrad CPC6128 expansion interface. On a 4MHz Z80 CPU reaching even the 3.5MB/s to 5MB/s of SCSI 1 was a challenge, but due to the buffering in the SCSI to MFM controller, it worked although probably not as fast as the controller and drive were capable of working.

I used the hard drive on the Amstrad CPC6128, under CP/M Plus along with the two Dk'tronics RAM expansion modules, and CP/M BIOS extensions written by Bevan Arps (then of Christchurch, New Zealand) for a couple of years, from around 1991 to 1993. Eventually the call of IBM PC Compatible hardware became too strong, and I put together the first of many PC Compatible machines -- from second hand parts -- which gradually supplanted my Amstrad CPC6128 as my "daily compute".

Floppy disc backup

In the middle of 1993 I made a backup of my Amstrad CPC6128 hard drive, onto floppy disks, and then on 1994-03-31 (date I wrote on the box!) the Amstrad CPC was packed away in boxes. I got those boxes out about 14 years ago -- in 2004 -- and made an attempt at transferring some of the floppy disks onto a PC, but between issues with reading the floppy disks (due to age), and difficulty reading the contents of the floppy disks (due to CP/M file formats) that project gradually got overtaken by events and sat on the hard drive of one of my Linux machines for a decade.

Several recent events encouraged me to take another look, including randomly being given a second Amstrad CPC6128 at a conference earlier this year, as well as being contacted by Kevin Thacker about AnsiTerm, and Jason Scott's repeated encouragement to Close the Air Gap and get things online (someone "closing the air gap" gave us the Walnut Creek CP/M CDROM from 1994 back again -- I bought a copy of it in 1994 when it came out, but somewhere in the last decade I put it "somewhere safe" and cannot remember where that is... :-( ).

Fortunately the backup of my Amstrad CPC6128 hard drive was one of the sets of disks where I managed to copy all the backup disks that I could find into disk images back in 2004, so the "air gap" had already been closed -- they just had not been unpacked. (Unfortunately I could find only discs 1/11 to 11/15, suggesting that 4 discs are missing -- but I no longer remember if there ended up being 11 discs total or 15 discs total. It appears maybe the disks were re-used from an earlier backup, which maybe did require 15 disks.)

Unfortunately, the backups were on 5.25" double sided floppy discs in an extended disc format for Nigdos, developed in New Zealand, with CP/M Plus support released on WACCI PD Disc 7. There is also a NigDos 2.24 ROM image available, but that has no documentation. It seemed like extracting the data from the backup disks would be non-trivial.

It turned out that WACCI PD Disk 7 can be downloaded (Side A CRC A2CDE9B5; Side B CRC 12FEA3FC; two 180KiB disk images). The disk image checksums can be checked with crc32:

ewen@linux:~$ crc32 *
a2cde9b5    WACCI-PD-Disc-07-Side-A.dsk
12fea3fc    WACCI-PD-Disc-07-Side-B.dsk
ewen@linux:~$

which match the ones listed on the download page.)

So I thought it was worth reading those disks to see if I could find out any more information about the disk format used.

Side quest: Accessing Amstrad CPC .dsk files

Which means that we get to find out (again!) how to read files from Amstrad CPC .dsk images on Linux. There seem to be a few options:

As it turns out the .dsk images downloaded above were created with SamDisk, and appear to be unreadable with anything other than SamDisk. So in order to extract them we need to use SamDisk to convert them. (I did try converting them with dskconv from libdsk, but it also was unable to read them :-( I am unclear how SamDisk extended the extended disk format, but the extension appears to confuse other tools...).

However SamDisk works with full disk images, rather than the file system within the disk image, so it cannot extract the files from the disk image. Which means that we need to use SamDisk to convert the disk image to something that we can read with another tool. The output formats of SamDisk are relatively limited -- it will write .dsk files, but it appears only in its Extended format, which is what we already have.

After conversion to a .raw file, we can then use cpmtools to read files out of the disk image.

SamDisk

To build SamDisk:

git clone git clone https://github.com/simonowen/samdisk
cd samdisk
sudo apt-get install cmake
cmake -DCMAKE_BUILD_TARGET=Release .
make

which after a while, will give us a samdisk binary, that can be installed with:

sudo make install

(The build takes a couple of minutes as SamDisk is written in C++ which always compiles relatively slowly on Linux.)

After installaton, we can test that it recognises the disk images with:

samdisk dir WACCI-PD-Disc-07-Side-A.dsk

and then convert them to .raw disk images (ie, no sector headers) which cpmtools can read with:

samdisk copy WACCI-PD-Disc-07-Side-A.dsk WACCI-PD-Disc-07-Side-A.raw
samdisk copy WACCI-PD-Disc-07-Side-B.dsk WACCI-PD-Disc-07-Side-B.raw

There is a complaint on these files that "source missing 27 sectors from 43/1/9/512 regular format", but this appears to be due to the .dsk headers indicating there are 43 tracks, but the actual data contains only the standard 40 tracks, so I ignored it.

cpmtools

cpmtools has been packaged in Debian Linux for years, so once we have a .raw image file, we can just use the packaged version directly and specify the implicit disk format of the sectors. To do this:

sudo apt-get install cpmtools
cpmls -f cpcdata WACCI-PD-Disc-07-Side-A.raw
cpmls -f cpcdata WACCI-PD-Disc-07-Side-B.raw
(mkdir side-a && cd side-a && cpmcp -f cpcdata ../WACCI-PD-Disc-07-Side-A.raw 0:* .)
(mkdir side-b && cd side-b && cpmcp -f cpcdata ../WACCI-PD-Disc-07-Side-B.raw 0:* .)

Other tools

Ultimately SamDisk and cpmtools was the only combination that worked with the *.dsk files that I had downloaded (imaged by SamDisk). But I did try several of the other tools listed above before determining this, and record how to build them, so I am keeping those details below for future reference. They do appear to work with other .dsk images, presumably those extracted earlier or with different tools.

dskinfo

To build dskinfo:

wget http://cpctech.cpc-live.com/download/dskinfo.zip
unzip dskinfo.zip
cd dskinfo
mv makefile Makefile
rm -f dskinfo *.o
make

which builds a single dskinfo binary in the source directory (and the archive ships with a pre-built version, built in 2011).

dskinfo usage is simple:

dskinfo disk_image.dsk

dskinfo outputs verbose (!) information about the sectors in the .dsk file, but also confirms that it is an "Extended CPCEMU style" .dsk file image. However the tool just outputs metadata; it does not actually read the data out of the disk.

cpcxfs

Encouraged by that I built cpcxfs:

wget http://cpctech.cpc-live.com/download/cpcxfs.zip
unzip cpcxfs.zip
cd cpcxfs/src
mv makefile.lnx Makefile
make clean
make

(There are lots of warnings, mostly about const safeness and signedness safeness, but it does build.)

cpcxfs usage is either via the command line or interactive (if run without a command it goes into interactive mode). To use from the command line:

  • List files on disk: cpcxfs DISK_IMAGE.dsk -d

  • Put a file on the disk: cpcxfs DISK_IMAGE.dsk -g ...

  • Get a file from the disk: cpcxfs DISK_IMAGE.dsk -p ...

  • Put multiple files on the disk: cpcxfs DISK_IMAGE.dsk -mg ...

  • Get multiple files from the disk: cpcxfs DISK_IMAGE.dsk -mp ...

(-mg and -mp take CP/M style wildcards, eg, *.*; -g and -p take single filenames.)

Unfortunately (a) cpcxfs also does not support the "Extended CPCEMU style" .dsk format, at least as used in the WACCI PD Disk 7 .dsk files created by SamDisk that I downloaded, and (b) it will complain it cannot open the disk image, even for a directory listing, if the file is write protected which seems unfortunate.

The lack of support for (all? some?) Extended CPCEMU style disk images is particularly unfortunate as Kevin Thacker (who released cpcxfs) is one of the creators of the Amstrad CPCEMU Extended Disk Format.

That format information helped me confirm that the WACCI PD .dsk images I had downloaded were created with SamDisk under Microsoft Windows, because "SAMdisk130107" appears in the .dsk image header. Given that these disks were public domain disks the use of the "Extended" .dsk format appears to be accidental (the "Extended" .dsk format was intended for copy protected disks), but one needs to work with what one can find. And unfortunately those .dsk images appear to be the only one available online. (SamDisk supports a lot of formats.)

iDSK

To build iDSK:

wget http://sid.cpc.free.fr/iDSK.0.13-src.tgz
tar -xzf iDSK.0.13-src.tgz
cd iDSK.0.13/iDSK
aclocal
automake
autoconf
./configure --prefix=/usr/local
make clean    # the downloaded archive includes .o files
make

Then the resulting binary is in src/iDSK which you could copy somewhere on your PATH manually, or use sudo make install (which appears to contain a lot of shell magic to do the same thing).

iDSK is written in French, including French help, but the commands are fairly obvious from context:

  • List files on disk: iDSK disk_image.dsk -l

  • Import file: iDSK -i file.bin -t 1 -s disk_image.dsk

  • Export file: iDSK -g file.bas -s disk_image.dsk

where the "type" (-t) is 0 for ASCII and 1 for binary.

Unfortunately iDSK did not support the Amstrad/Spectrum Extended .DSK data files used by the WACCI PD Disk files I downloaded so it was not useful to me here, and since the output is in French (which I do not read very well) I am more likely to use other tools.

libdsk

To build libdsk:

wget http://www.seasip.info/Unix/LibDsk/libdsk-1.5.8.tar.gz
tar -xzf libdsk-1.5.8.tar.gz
cd libdsk-1.5.8
aclocal
automake --add-missing
autoconf
./configure --prefix=/usr/local
make

That then gives a library, and a series of tools:

  • dskconv: disk image conversion (new in 1.5.x)

  • dskdump: sector level copy from disk/image to another disk/image

  • dskform: format a floppy disk/image

  • dskid: identify a floppy disk/image

  • dskscan: scan a floppy disk for sectors

  • dsktrans: transfer from one disk/image to another disk/image

  • dskutil: a sector editor

which can be installed into /usr/local/ by running:

sudo make install

To actually run them on modern Linux it is necessary to ensure they can find the shared library:

echo /usr/local/lib | sudo tee /etc/ld.so.conf.d/usr-local-lib.conf
sudo ldconfig

After that, in theory we can use dskconv (from the libdsk 1.5.x series) to convert from the Extended .dsk image file to a standard .dsk image file. There is no man page for dskconv yet, so the help is just output by running the raw command:

ewen@linux:/var/tmp/amstrad$ dskconv
Syntax:
      dskconv {options} in-image out-image

Options are:
-itype <type>   type of input disc image
-otype <type>   type of output disc image
                'dskconv -types' lists valid types.
-format         Force a specified format name
                'dskconv -formats' lists valid formats.

Default in-image type is autodetect.
Default out-image type is LDBS.

eg: dskconv diag800b._01 diag800b.ldbs
    dskconv -otype raw diag800b._01 diag800b.ufi
ewen@linux:/var/tmp/amstrad$

The supported image file types are:

ewen@linux:/var/tmp/amstrad$ dskconv -types
Disk image types supported:

   remote     : Remote LibDsk instance
   rcpmfs     : Reverse CP/MFS driver
   floppy     : Linux floppy driver
   dsk        : CPCEMU .DSK driver
   edsk       : Extended .DSK driver
   apridisk   : APRIDISK file driver
   copyqm     : CopyQM file driver
   tele       : TeleDisk file driver
   ldbs       : LibDsk block store
   qrst       : Quick Release Sector Transfer
   imd        : IMD file driver
   ydsk       : YAZE YDSK driver
   raw        : Raw file driver (alternate sides)
   rawoo      : Raw file driver (out and out)
   rawob      : Raw file driver (out and back)
   myz80      : MYZ80 hard drive driver
   simh       : SIMH disc image driver
   nanowasp   : NanoWasp image file driver
   logical    : Raw file logical sector order
   jv3        : JV3 file driver
   cfi        : CFI file driver
ewen@linux:/var/tmp/amstrad$

And we could in theory convert a disk image with:

mkdir edsk dsk
# Put source files in edsk
dskconv -otype dsk edsk/WACCI-PD-Disc-07-Side-A.dsk dsk/WACCI-PD-Disc-07-Side-A.dsk

but unfortunately libdsk also reports the disk image is unreadable :-(

At this point I looked harder at SamDisk and discovered that there was source for a development version on GitHub, so built that and used SamDisk and cpmtools to extract the files (as described above). It appears that SamDisk (and maybe some emulators) support an Extended Extended version of the .dsk format, and relatively few other tools support that specific format.

DiskImager

More searching turned up DiskImager, which looks like it should be able to visualise the extended .dsk files, and is written using the Lazarus FreePascal system (a Delphi-compatible free IDE). It does not support file extraction directly, but it appears it can convert between Extended and Standard .dsk file formats. Because of the need to install a custom development environment I decided to skip past this one for now (but it does look very useful for preservation work). Because I have not tried it I do not know if it supports the same Extended .dsk format variations as SamDisk.

dsktools

Other searching turned up dsktools originally written by Andreas Micklei), and originally released on SourceForge and Berlios (now gone). The most recent version of dsktools is on GitHub

To build dsktools:

git clone https://github.com/cpcitor/dsktools
cd dsktools/dsktools
make clean     # Ignore errors of files not existing
make

(Due to the way that the source was rescued from Berlios, both the git repository and the directory are called dsktools, so there is effectively a second level in the git repository.)

This gives two tools dskread and dskwrite, which can be installed in /usr/local/bin with:

sudo make install

Usage is dskread and dskwrite, but they will only read from /dev/fd0 and write to /dev/fd0, so they are not useful for working with already imaged .dsk files :-(

Back to the main quest: reading the NigDos backups

Nigdos disk format

As mentioned above, I previously imaged the NigDos backup floppies on Linux with a simple C program (readnigdos -- readnigdos.c, and Makefile) that read raw sectors using the Linux floppy driver and output the raw sector data to a file, in the order I believed was in use.

"format.doc" and "extdisc.doc" from the WACCI-PD-Disc-07-Side-B extracted above (with SamDisk and cpmtools) give information on the NigDos formats:

The Extra Formats you can create are as follows:

        Name:                          Usable Space:

     V  Vendor Format                      169K
     D  Data Format                        178K
     P  PCW A Drive Format                 173K
     B  Nigdos Big Format                  208K
     T  Nigdos Two Side Format             416K
     C  CPM Big Format                     416K
     L  CPM Large Format                   716K
     H  CPM Huge Format                    816K

and that reminded me that the most likely format was the "Nigdos Big Format". But it provided no information on the precise layout of the disk format.

From my earlier guesses when I wrote readnigdos (14 years ago!), I believe the Nigdos Big Format is:

  • 512 byte sectors (size_ind=2 in CP/M terminology)

  • 10 sectors per track, numbered 0x91 to 0x9a (following the Amstrad CPC convention of using the sector number as a hint of the format)

  • 42 tracks per side (numbered 0 through 41)

  • 2 sides (0 and 1)

  • Tracks ordered "out then out", ie all of side 0 is used and then all of side 1 is used, ie:

    Track 0, Side 0
    Track 1, Side 0
    Track 2, Side 0
    ...
    Track 41, Side 0
    Track 0, Side 1
    Track 1, Side 1
    Track 2, Side 1
    ...
    Track 41, Side 1
    

I actually tried asking Bevan Arps if he remembered the track layout, but after 25 years he was guessing as much as I was. Bevan's thought was that it was probably alternating sides -- ie Track 0, Side 0 then Track 0, Side 1, to reduce seeking, and that probably would have been my first guess too for the same reason. However it turns out that both were supported -- "NigDos 420K Double Sided" was "all side 0 then all side 1" (in order, as above), with sectors 0x91 to 0x9a, and "CPM 420K Double Sided" was "interleaved side 0 and side 1" as we first guessed, with sectors 0xa1 to 0xaa (see end of blog post for some more detail).

Having actually installed cpmtools (above), it turned out that one of its disk definitions (in /etc/cpmtools/diskdefs) included the Nigdos format:

diskdef nigdos
  seclen 512
  # NigDos double sided disk format, 42 tracks * 2 sides
  tracks 84
  sectrk 10
  blocksize 2048
  maxdir 128
  skew 1
  boottrk 0
  # this format wastes half of the directory entry
  logicalextents 1
  os 3
end

which seemed to match what I had detected (512 bytes * 10 sectors/track * 42 tracks/side * 2 sides = 430080 bytes; 512 bytes * 10 sectors/track * 84 tracks = 430080). That means that assuming the tracks are in the right order in the disk image file cpmtools should be able to extract the files out of the backups. (And I would assume for both "NigDos 420K" and "CPM 420K" assuming the tracks off the floppy disk end up in the image file in logical order rather than physical order.)

I believe the comment that "this format wastes half of the directory entry" relates to using 16-bit block numbers, rather than the 8-bit block numbers which would be sufficient for a 420KiB floppy disk using 2KiB blocks. This appears to have happened for backwards compatibility, and compatibility with the larger formats (eg, 716KiB and 816KiB above) which had over 256 * 2 KiB blocks and thus did need the 16-bit block numbers. In practice there were sufficiently many directory entires, and sufficiently little space, that running out of directory entries was not a problem in practice; it is just an odd data recovery quirk to be aware of later.

Reading the backup disk images

According to the notes from the disk labels (which fortunately I also transcribed back in 2004) the backups contained:

  • User 0: disks 1-5

  • User 1: disks 5-9

  • User 2: disk 9

  • User 3: disk 9-10

  • User 4: disk 10-11 (and implicitly 12/13, but I could not find them back in 2004 when I imaged the backup floppy discs)

This tends to suggest that some files will be missing from the backup :-( (At least unless I eventually find the missing floppies, and they are still readable 25 years later.) However the last disk is not full (and still contains "deleted" data) so possibly these were old backup disks that were overwritten but the labels never updated to indicate that fewer disks were required.

Anyway I could list the files on the backup disks with:

for DISK_IMAGE in *.nigdos; do
    cpmls -f nigdos -F "${DISK_IMAGE}"
done

which gave sensible output, although only about 400 files in total. However 11 disks at 420KiB each works out to about 4.5MiB, which is roughly the portion of the hard drive I made available (because some information from the hard drive tables needed to be held in RAM, and holding more in RAM took up too much of the Amstrad CPC RAM space -- so I only ever made 5MiB available). Certainly the disk images seemed to all have 370-400KiB used.

I believe that cpmls will by default list the files in all users areas. That, plus scanning the CP/M directories in the "hexdump -C" output suggested that all the files on the backups were in User 0 of the floppy disks.

So I extracted all the files into a specific directory per disk with:

for DISK in $(seq -f "%02.0f" 1 11); do
    (mkdir -p "${DISK}" && cd "${DISK}" &&
     cpmcp -f nigdos -p ../"cpc-hd-backup-1993-06-19-disk-${DISK}-of-15.nigdos" "0:*" .)
done

and then zipped that into a more modern archive for future reference:

zip -9r cpc-hd-backup-1993-06-19.zip [01]*

Looking through the backup it appears that it is mostly CP/M programs which were the main thing I would have been likely to store on the hard drive rather than the floppy drive. But there are few other gems included in there.

It appears that the files have been extracted properly, but the only easy way to tell is by reading text files, and looking for files stored with checksums (.ark/.arc, .zoo, .zip, etc).

Reading .ark/.arc files (the original System Enhancement Associates archive format) on Linux is possible with nomarch:

sudo apt-get install nomarch
nomarch -l ARCHIVE    # List files
nomarch -t ARCHIVE    # Test checksums of files in archive

Confirming the NigDos Double Sided Format (sectors 0x91 to 0x9a)

One of the gems from looking around was the source code to Bevan Arp's format, extdisc, etc, that he had apparently sent to me in 1989. So I sent that back to him. It seemed only fair :-)

The "extdisc" source file contained the definitions of the extra floppy disk formats:

DataFormat      equ   0
Block2K         equ 128
Sectors10       equ  32
Tracks8082      equ  64
NigDos          equ  16
Interlace       equ   4
TwoSideRev      equ   8
TwoSideNorm     equ  12
Reserve1Track   equ   1
Reserve2Track   equ   2

; Format Of Data

;               byte FirstSectorNumber
;               byte Flags
;               word DiscSpace

[...]

; NigDos 420K Double Sided

                byte &91
                byte DataFormat+Sectors10+NigDos+TwoSideNorm+Block2K
                word &d1

; CPM 420K Double Sided

                byte &a1
                byte DataFormat+Sectors10+Interlace+Block2K
                word &d1

The "SetFormat" routine in that source ends up rotating the flags values twice ("rra; rra" in Z80 assembly), and then storing them in the Amstrad Extended Disk Parameter Block (XDPB) as the "sidedness" field -- which is called "SID" in the source code.

As described in the seasip.info XDPB documentation)

DEFB        sidedness
                        ;Bits 0-1:  0 => Single sided
                        ;           1 => Double sided, flip sides
                        ;              ie track   0 is cylinder   0 head 0
                        ;                 track   1 is cylinder   0 head 1
                        ;                 track   2 is cylinder   1 head 0
                        ;               ...
                        ;                 track n-1 is cylinder n/2 head 0
                        ;                 track   n is cylinder n/2 head 1
                        ;           2 => Double sided, up and over
                        ;              ie track   0 is cylinder 0 head 0
                        ;                 track   1 is cylinder 1 head 0
                        ;                 track   2 is cylinder 2 head 0
                        ;               ...
                        ;                 track n-2 is cylinder 2 head 1
                        ;                 track n-1 is cylinder 1 head 1
                        ;                 track   n is cylinder 0 head 1

which means:

  • "4" (Interlace) turns into "1" when rotated right twice, and so is interleaved/interlaced;

  • "8" (TwoSideRev) turns into "2" when rotated right twice, and so is up/over/back

  • "12" (TwoSideNorm) turns into "3" when rotated twice, and is specially handled in the patched "TranslateTrack in the source code, to implement "out and out", the format that I had guessed (it skips over the Interleave routine, and then skips over the "Up and Over" track number reversal logic, resulting in just "track = track - tracks_per_side").

This confirms my guesses that the disk format that I had uses all of side 0 in order, then all of side 1 in order, which means my readnigdos.c implements the right logic for the sector values it is reading.

Conclusion

Overall this was a long adventure (14+ years if you count from when the floppy disks were read in; 25+ years from when they were written), but I appear to have successfully recovered about 4.5MB of data backed up from my CP/M hard drive, to explore later. I have also confirmed the format of the "NigDos 420K Two Sided" floppy disks that I created, which gives me confidence to recover data from the other floppy disk images I made in 2004. And I have several tools to extract files from Amstrad CPC .dsk files produced in various eras with various tools. Overall a very successful quest, but rather time consuming!

ETA, 2018-03-05: History of Amstrad CPC464 creation -- the first model, that led to the Amstrad CPC6128 a year or so later. (Archived Version Page 1 and Page 2)

ETA, 2018-03-11: Added date Amstrad CPC was packed away.

ETA, 2018-07-12: This blog post was mentioned on the CPC Wiki Forum. The amstrad tag has links to other posts about recovering Amstrad CPC data.

Posted Sat Mar 3 21:25:47 2018 Tags:

Introduction

Last year (2017) I bought a Numato Mimas v2 with the intention of running MicroPython on it. After a bit of guess work and some assistance on IRC, I managed to get MicroPython running on Mimas V2 (Spartan-6) FPGA.

A lot has changed in the 9 months since that original blog post, including:

So it seemed worth summarising the current build process for both FPGA MicroPython (FμPy) on the Numato Mimas v2 and also the Digilent Arty A7 in one place.

The process below has been tested on an Ubuntu 16.04 LTS x86_64 system, as with the previous build process last year.

USB Mode Switch udev rules

The Digilent Arty A7 benefits from the timvideos USB Mode Switch udev rules. So it is useful to install these before starting if you will be targetting the Arty A7:

git clone https://github.com/timvideos/HDMI2USB-mode-switch.git
cd HDMI2USB-mode-switch/udev
make install
sudo udevadm control --reload-rules

And also to ensure that your user has permission to access the relevant device files:

sudo adduser $USER video
sudo adduser $USER dialout
sudo reboot

(it may be sufficient to log out/log in again to pick up the relevant additional group permissions, but rebooting will ensure udev, etc, also have the updated configuration files active).

(The Numato Mimas v2 has a physical switch to move instead of relying on USB mode switching, so the above udev rules should not be needed if you are only working with a Numato Mimas v2.)

Downloading the build environment repository

To get the top level repository used for building, run, eg:

cd /src
git clone https://github.com/timvideos/litex-buildenv.git

(If you check it out in a different directory use that directory instead of cd /src/litex-buildenv below.)

This repository is common to both the Numato Mimas v2 (Spartan-6) and the Digilent Arty A7 (Artix-7), but contains both several git submodules referencing other git repositories (some slightly different between the two platforms) as well as a lot of conda install instructions to install relevant tools (again some slightly different between the two platforms).

Numato Mimas v2

The Numato Mimas v2 is a based around a Xilinx Spartan-6 FPGA, which uses the Xilinx ISE WebPACK proprietary FPGA synthesis software. The zero-cost "ISE WebPACK" license is sufficient for this process.

These instructions Assam you have:

Preparation

To get started building FPGA MicroPython for the Numato Mimas v2, start in the timvideos/litex-buildenv cloned above:

cd /src/litex-buildenv

and then configure the build targets:

CPU=lm32
PLATFORM=mimasv2
TARGET=base
FIRMWARE=micropython

export CPU PLATFORM TARGET FIRMWARE

Once this is done, download the other bits of the build environments required for the Numato Mimas v2:

scripts/download-env.sh

And then when that finishes, enter the build environment ready to build gateware and firmware for the Numato Mimas v2:

source scripts/enter-env.sh

All going well you should now have a prompt which begins with "(LX P=mimasv2 F=micropython)" (since the lm32 CPU is the default, as is the base TARGET). If anything is reported missing you may need to run scripts/download-env.sh again, or ensure that you have the Xilinx ISE WebPACK installed and reachable from /opt/Xilinx.

Once you have completed scripts/download-env.sh one time, you can then skip that step in later sessions, just doing:

cd /src/litex-buildenv
CPU=lm32
PLATFORM=mimasv2
TARGET=base
FIRMWARE=micropython
export CPU PLATFORM TARGET FIRMWARE
source scripts/enter-env.sh

to get started in a new terminal session.

Building the gateware and micropython for the Numato Mimas v2

To build the required gateware (FPGA configuration, including a lm32 soft CPU) and micropython run:

make gateware
scripts/build-micropython.sh

and after several minutes (depending on the speed of your build machine; the ISE WebPACK synthesis is very CPU intensive) you should have both a gateware/top.bin file, and also a software/micropython/firmware.bin file, inside build/mimasv2_base_lm32:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$ ls -l build/mimasv2_base_lm32/gateware/top.bin
-rw-rw-r-- 1 ewen ewen 340884 Jan 17 12:32 build/mimasv2_base_lm32/gateware/top.bin
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$ ls -l build/mimasv2_base_lm32/software/micropython/firmware.bin
-rwxrwxr-x 1 ewen ewen 167960 Jan 17 12:32 build/mimasv2_base_lm32/software/micropython/firmware.bin
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$

Loading the gateware and micropython onto the Numato Mimas v2

On the Numato Mimas v2, the gateware (hardware configuration, including the lm32 soft CPU), BIOS (boot loader, etc), and an application ("firmware", eg micropython) all need to be loaded together in one large combined image file.

This combined image file is loaded in the programming mode of the Mimas v2, which is controlled by the slide switch SW7 near the power and USB connectors.

To load the combined gateware/BIOS/firmware (application) image:

  • Slide the Mimas v2 SW7 switch to "programming" mode (switch in position closest to the USB connector), and connect the USB cable between the Mimas v2 and the build machine if it is not already connected

  • Run:

    make image-flash
    

    to load the combined image file. This loads over a 19200 bps serial link, so it takes many seconds to load.

You should see something like:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$ make image-flash
[...]
python $(which MimasV2Config.py) /dev/ttyACM0 build/mimasv2_base_lm32//image-gateware+bios+micropython.bin
****************************************
* Numato Lab Mimas V2 Configuration Tool *
****************************************
Micron M25P16 SPI Flash detected
Loading file build/mimasv2_base_lm32//image-gateware+bios+micropython.bin...
Erasing flash sectors...
Writing to flash 100% complete...
Verifying flash contents...
Flash verification successful...
Booting FPGA...
Done.
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$

Then when it is installed, to reach the micropython REPL:

  • Slide the Mimas v2 SW7 switch to "operation" mode (switch in position furthest from the USB connector)

  • Run:

    make firmware-connect
    

    and press enter a few times to get the MicroPython REPL prompt (or press the SW6 lm32 CPU reset on the Mimas v2 to see the boot prompts).

This looks something like:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$ make firmware-connect
flterm --port=/dev/ttyACM0 --speed=19200
[FLTERM] Starting...

>>>
>>>
>>>
>>>
LiteX SoC BIOS (lm32)
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Jan 17 2018 12:29:35

BIOS CRC passed (5338fc86)
Initializing SDRAM...
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Loading 167960 bytes from flash...
Executing booted program at 0x40000000
MicroPython v1.8.7-465-g7a4245d on 2018-01-17; litex with lm32
>>>

when the SW6 button was pressed after hitting enter a few times.

The advantage of this approach is that both the lm32 soft CPU gateware and micropython are stored in the SPI flash on the Numato Mimas v2, and thus will start automatically when power is applied to the Numato Mimas v2, so the only steps to start interacting with the micropython REPL is

cd /src/litex-buildenv
CPU=lm32
PLATFORM=mimasv2
TARGET=base
FIRMWARE=micropython
export CPU PLATFORM TARGET FIRMWARE
source scripts/enter-env.sh
make firmware-connect

and then hitting enter a few times to bring up the micropython REPL prompt.

The disadvantage is that the "make image-flash" step can take a very long time to upload the combined image.

Testing micropython updates with serialboot:

Because "make image-flash" takes such a long time, you may prefer to test interactive changes to the micropython application by loading them via serial boot, as then only the micropython application needs to be transferred with each change. To do this:

  • Slide the Mimas v2 SW7 switch to "operation" mode (switch in position furthest from the USB connector)

  • Run:

    make firmware-load
    

    to prepare to serialboot the lm32 software CPU into micropython.

  • Press the Mimas v2 lm32 "reset" switch, which is SW6 (near the SD card connector) to reset the software CPU and kick off a serialboot.

You should see something like:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$ make firmware-load
flterm --port=/dev/ttyACM0 --kernel=build/mimasv2_base_lm32//software/micropython/firmware.bin --speed=19200
[...]
[FLTERM] Starting...

LiteX SoC BIOS (lm32)
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Jan 17 2018 12:29:35

BIOS CRC passed (5338fc86)
Initializing SDRAM...
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[FLTERM] Received firmware download request from the device.
[FLTERM] Uploading kernel (167960 bytes)...
[FLTERM] Upload complete (1.6KB/s).
[FLTERM] Booting the device.
[FLTERM] Done.
Executing booted program at 0x40000000
MicroPython v1.8.7-465-g7a4245d on 2018-01-17; litex with lm32
>>> print("Hello World!")
Hello World!
>>>
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/litex-buildenv$

The serialboot of micropython will still take many seconds, as loading about 170KB at 19.2kbps is relatively slow! But at least you do not also need to load the gateware/firmware too.

ETA, 2018-02-10: An earlier version of this blog post suggested using "make gateware-flash" and then "make firmware-load" as a quicker method; unfortunately (a) "make gateware-flash" does not include the BIOS or firmware (application), and (b) make firmware-load relies on the BIOS to work, so that combination only works if the Mimas v2 board in question happens to have been previously programmed with "make image-flash", so the BIOS is already on the board. (In which case "make gateware-flash" is redundant unless the hardware definition has changed, and still fits in the gap left for it in the flash.)

Digilent Arty A7

The Digilent Arty A7 is a based around a Xilinx Artix-7 FPGA, which uses the Xilinx Vivado HL WebPACK proprietary FPGA synthesis software. The zero-cost "Vivado HL WebPACK" license is sufficient for this process.

These instructions assume you have:

  • a Digilent Arty A7 Artix-7 based FPGA board

  • an USB A to USB Micro cable, to connect Digilent Arty A7 to the Ubuntu 16.04 LTS build system

  • the Xilinx Vivado HL WebPACK synthesis tool installed, reachable from /opt/Xilinx (eg, via a symlink) with at least:

    • Design Tools / Vivado Design Suite / Vivado

    • Devices / Production Devices / 7 Series / Artix-7

    features installed; the other features are not needed and do not need to be installed.

Preparation

To get started building FPGA MicroPython for the Digilent Arty A7, start in the timvideos/litex-buildenv cloned above:

cd /src/litex-buildenv

and then configure the build targets:

CPU=lm32
PLATFORM=arty
TARGET=base
FIRMWARE=micropython

export CPU PLATFORM TARGET FIRMWARE

Once this is done, download the other bits of the build environment required for the Digilent Arty A7:

scripts/download-env.sh

(If you have already done this for the Numato Mimas v2 above, then most of the extras required will be already installed, but there are a few board-specific items, so be sure to run scripts/download-env.sh again to be sure.)

When that finishes, enter the build environment ready to build gateware and firmware for the Numato Mimas v2:

source scripts/enter-env.sh

All going well you should now have a prompt which begins with "(LX P=arty T=base)" (since the lm32 CPU is the default CPU). If anything is reported missing you may need to run scripts/download-env.sh again, or ensure that you have the Xilinx Vivado HL WebPACK installed and reachable from /opt/Xilinx.

Once you have completed scripts/download-env.sh one time, you can then skip that step in later sessions, just doing:

cd /src/litex-buildenv
CPU=lm32
PLATFORM=arty
TARGET=base
FIRMWARE=micropython
export CPU PLATFORM TARGET FIRMWARE
source scripts/enter-env.sh

to get started in a new terminal session.

Building the gateware and micropython for the Digilent Arty A7

To build the required gateware (FPGA configuration, including a lm32 soft CPU) and micropython run:

make gateware
scripts/build-micropython.sh

and after several minutes (depending on the speed of your build machine; the Vivado HL WebPACK synthesis is very CPU intensive) you should have both a gateware/top.bin file, and also a software/micropython/firmware.bin file, inside build/arty_base_lm32:

(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$ ls -l build/arty_base_lm32/gateware/top.bin
-rw-rw-r-- 1 ewen ewen 2192012 Jan 17 16:06 build/arty_base_lm32/gateware/top.bin
(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$ ls -l build/arty_base_lm32/software/micropython/firmware.bin
-rwxrwxr-x 1 ewen ewen 167816 Jan 17 16:07 build/arty_base_lm32/software/micropython/firmware.bin
(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$

Loading the gateware and micropython onto the Digilent Arty A7

There are two stages to getting micropython running on the Digilent Arty A7: loading the gateware (hardware configuration, including the lm32 soft CPU), and loading the micropython firmware.

On the Digilent Arty A7 these two steps can be done one after another, thanks to the USB mode switching:

To load the gateware:

  • Run:

    make gateware-load
    

    to load the hardware configuration. This loads over the JTAG path to the Digilent Arty A7, so loads in a few seconds (much faster than the equivalent step with the Numato Mimas v2).

You should see something like:

(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$ make gateware-load
openocd -f board/digilent_arty.cfg -c "init; pld load 0 build/arty_base_lm32//gateware/top.bit; exit"
Open On-Chip Debugger 0.10.0+dev-00267-gf7836bbc7-dirty (2018-01-15-07:40)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
none separate
adapter speed: 10000 kHz
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
jtagspi_program
Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi_tdo_sample_edge falling"
Info : clock speed 10000 kHz
Info : JTAG tap: xc7.tap tap/device found: 0x0362d093 (mfg: 0x049 (Xilinx), part: 0x362d, ver: 0x0)
Info : Listening on port 3333 for gdb connections
loaded file build/arty_base_lm32//gateware/top.bit to pld device 0 in 1s 833353us
(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$

To load micropython via serialboot:

  • Run:

    make firmware-load
    

    to prepare to serialboot the lm32 software CPU into micropython.

  • You may (or may not) then need to press the red "reset" button on the Digilent Arty A7 board (in the corner diagonally opposite the power connector) to get it to start serial booting (or if you have just powered the board on and loaded the gateware, it might be sitting waiting to serial boot).

You should see something like:

(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$ make firmware-load
[...]
flterm --port=/dev/ttyUSB1 --kernel=build/arty_base_lm32//software/micropython/firmware.bin --speed=115200
[FLTERM] Starting...

LiteX SoC BIOS (lm32)
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Jan 17 2018 16:03:05

BIOS CRC passed (df2e52a7)
Initializing SDRAM...
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[FLTERM] Received firmware download request from the device.
[FLTERM] Uploading kernel (167816 bytes)...
[FLTERM] Upload complete (7.7KB/s).
[FLTERM] Booting the device.
[FLTERM] Done.
Executing booted program at 0x40000000
MicroPython v1.8.7-465-g7a4245d on 2018-01-17; litex with lm32
>>> print("Hello World!")
Hello World!
>>>
(LX P=arty T=base F=micropython) ewen@parthenon:/src/litex-buildenv$

The serialboot of micropython will take several seconds, as loading about 170KB even at 115.2kbps is slow (but a lot faster than loading at 19.2kbps on the Mimas v2!).

Avoiding serialbooting

Unlike the Numato Mimas v2 there does not appear to be an easy way to avoid the serialboot process of micropython; but fortunately the Arty A7 serialboot (at 115.2kbps) is much faster than the Mimas v2 serialboot (at 19.2kbps).

However if the Arty A7 is already powered on with the gateware and micropython loaded, you can reconnect to the micropython REPL by running:

cd /src/litex-buildenv
CPU=lm32
PLATFORM=arty
TARGET=base
FIRMWARE=micropython
export CPU PLATFORM TARGET FIRMWARE
source scripts/enter-env.sh
make firmware-connect

and then hitting enter a few times to bring up the micropython REPL prompt.

Not that the Digilent Arty A7 does not store the gateware configuration in a SPI flash (unlike the Numato Mimas v2) so if the Arty A7 is power cycled you will need to go through the:

cd /src/litex-buildenv
CPU=lm32
PLATFORM=arty
TARGET=base
FIRMWARE=micropython
export CPU PLATFORM TARGET FIRMWARE
source scripts/enter-env.sh
make gateware-load
make firmware-load

cycle from the beginning, most likely including hitting the Arty A7 reset button to initiate serial booting. Fortunately this only takes a few tens of seconds to do.

ETA, 2018-01-19: Swapped instructions to using make firmware-load, now that it supports alternate firmware on Mimas v2 and Arty A7, as it is much easier to type :-)

Posted Wed Jan 17 18:29:01 2018 Tags:

Introduction

On modern storage media, and storage subsystems, file system alignment to "larger than 512 byte" boundaries is increasingly important for achieving good write performance, and on some media for avoiding excessive wear on the underlying media (due to additional write amplification). For about the last 8 years, the Linux kernel has supported an "/sys/block/*/alignment_offset" metric which indicates the number of bytes needed to get a particular layer back into alignment with the underlying storage media, and Linux distributions have included tools that attempt to do automatic alignment, where possible, for the last few years. Which helps newer systems, particularly new installations, but cannot automatically fix older systems.

I had one older system (originally installed at least 15 years ago, and "grandfather's axe" upgraded through various versions of hardware), that ended up having both:

  • 4KB (4096 byte) physical sectors (on a pair of WDC WD20EFRX-68A 2TB drive)

    ewen@linux:~$ cat /sys/block/sda/device/model 
    WDC WD20EFRX-68A
    ewen@linux:~$ cat /sys/block/sda/queue/physical_block_size 
    4096
    ewen@linux:~$ cat /sys/block/sda/queue/logical_block_size 
    512
    ewen@linux:~$
    

    although curiously the second apparently identical drive detects with 512-byte physical sectors, at least at present:

    ewen@tv:~$ cat /sys/block/sdb/device/model 
    WDC WD20EFRX-68A
    ewen@tv:~$ cat /sys/block/sdb/queue/physical_block_size 
    512
    ewen@tv:~$ cat /sys/block/sdb/queue/logical_block_size 
    512
    ewen@tv:~$ 
    

    for reasons I do not understand (both drives were purchased approximately the same time, as far as I can recall)

  • 1990s-style partition layout, with partitions starting on a "cylinder" boundary:

    (parted) unit s
    (parted) print
    Model: ATA WDC WD20EFRX-68A (scsi)
    Disk /dev/sda: 3907029168s
    Sector size (logical/physical): 512B/4096B
    Partition Table: msdos
    Disk Flags:
    
    Number  Start        End          Size         Type      File system  Flags
     1      63s          498014s      497952s      primary   ext4         raid
     2      498015s      4498199s     4000185s     primary                raid
     3      4498200s     8498384s     4000185s     primary                raid
     4      8498385s     3907024064s  3898525680s  extended
     5      8498448s     72501344s    64002897s    logical                raid
     6      72501408s    584508959s   512007552s   logical                raid
     7      584509023s   1096516574s  512007552s   logical                raid
     8      1096516638s  1608524189s  512007552s   logical                raid
     9      1608524253s  2120531804s  512007552s   logical                raid
    10      2120531868s  2632539419s  512007552s   logical                raid
    11      2632539483s  3144547034s  512007552s   logical                raid
    12      3144547098s  3656554649s  512007552s   logical                raid
    13      3656554713s  3907024064s  250469352s   logical                raid
    
    (parted)
    
  • Linux MD RAID-1 and LVM with no adjustments for the partition offsets to the physical block boundaries (due to being created with old tools), and

  • Linux file systems (ext4, xfs) created with no adjustments to the physical block boundaries (due to being created with old tools)

I knew that this misalignment was happening at the time I swapped in the newer (2TB) disks a few years ago, but did not have time to try to manually figure out the correct method to align all the layers, so I decided to just accept the lower performance at the time. (Fortunately being magnetic storage rather than SSDs, there was not an additional risk of excessive drive wear caused by the misalignment.)

After upgrading to a modern Debian Linux version, including a new kernel, this misalignment was made more visible again, including in kernel messages on every boot:

device-mapper: table: 253:2: adding target device (start sect 511967232 len 24903680) caused an alignment inconsistency
device-mapper: table: 253:4: adding target device (start sect 511967232 len 511967232) caused an alignment inconsistency
device-mapper: table: 253:4: adding target device (start sect 1023934464 len 511967232) caused an alignment inconsistency
device-mapper: table: 253:4: adding target device (start sect 1535901696 len 36962304) caused an alignment inconsistency
device-mapper: table: 253:4: adding target device (start sect 1572864000 len 209715200) caused an alignment inconsistency
device-mapper: table: 253:5: adding target device (start sect 34078720 len 39321600) caused an alignment inconsistency

so I planned to eventually re-align the partitions on the underlying drives to match the modern "optimal" conventions (ie, start partitions at 1 MiB boundaries). I finally got time to do that realignment over the Christmas/New Year period, during a "staycation" that let me be around periodically for all the steps required.

Overall process

The system in question had two drives (both 2TB), in RAID-1 for redundancy. Since I did not want to lose the RAID redundancy during the process my general approach was:

  • Obtain a 2TB external drive

  • Partition the 2TB external drive with 1 MiB aligned partitions matching the desired partition layout, marked as "raid" partitions

  • Extend the Linux MD RAID-1 to cover three drives, including the 2TB external drive, and wait for the RAID arrays to resync.

  • Then for each drive to be repartitioned, remove the drive from the RAID-1 sets (leaving the other original drive and the external drive), repartition it optimally, and re-add the drive back into the RAID-1 sets and wait for the RAID arrays to resync (then repeat for the other drive).

  • Remove the external 2TB drive from all the RAID sets

  • Reboot the system to ensure it detected the original two drives as now aligned.

This process took about 2 days to complete, most of which was waiting for the 2TB of RAID arrays to sync onto the external drive, as the system in question had only USB-2 (not USB-3), and thus copies onto the external drive went at about 30MB/s and took 18-20 hours. The last stage of repartioning and resyncing the original drives went much faster as the copies onto those drives went at over 120MB/s (reasonable for SATA-1.5 Gbps connected drives: the drives are SATA-6Gbps capable, but the host controller is only SATA 1.5 Gbps).

NOTE: If you are going to attempt to follow this process yourself, I strongly recommend that you have a separate backup of the system -- other than on the RAID disks you are modifying -- as accidentally removing or overwriting the wrong thing at the wrong time during this process could easily lead to a difficult to recover system or permanent data loss.

Partition alignment

Modern parted is capable of creating "optimal" aligned partitions if you give it the "-a optimal" flag when start it up. However, due to the age of this system I needed to recreate a MBR partition table with 13 partitions on it -- necessitating several logical partitions -- which come with their own alignment challenges (it turns out that the Extended Boot Record logical partitions require a linked list of partition records between the logical partitions, thus requiring some space between each partition); on a modern system using a GUID Partition Table avoids most of these challenges. (Some choose not to align the extended partition, as there is no user-data in the extended partition so only the logical partition alignment matters; but this is only a minor help if you have multiple logical partitions.)

After some thought I concluded that my desired partitioning had:

  • The first partition starting at 1MiB (2048s)

  • Every other partition starting on a MiB boundary

  • Every subsequent partition as close as possible to the previous one

  • All partitions at least a multiple of the physical sector size (4KiB)

  • All partitions larger than the original partitions on the disk, so that the RAID-1 resync would trivially work

  • Minimise wasted "rounding up" space

  • The last partition on the disks absorbing all of the reductions in disk space needed to meet the other considerations

Unfortunately it turned out that the need for multiple "logical" partitions, the desire to start every partition on a MiB boundary, parted most easily supporting creating partitions that were N MiB long, and a desire not to waste space between partitions, ended up conflicting with the need to have an Extended Boot Record between every logical partition. I could either accept that I would lose 1 MiB between each logical partition -- to hold the 512 byte EBR and have everything start/end on a MiB boundary -- or use more advanced methods to describe to parted what I needed. I chose to use more advanced methods.

Creating the first part of the partition table was pretty easy (here /dev/sdc was the 2TB external drive; but I followed the same partitioning on the original drives when I got to rebuilding them):

ewen@linux:~$ sudo parted -a optimal /dev/sdc
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdc will be destroyed and all data on
this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) quit

ewen@linux:~$ sudo parted -a optimal /dev/sdc
(parted) unit s
(parted) print
Model: WD Elements 25A2 (scsi)
Disk /dev/sdc: 3906963456s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start  End  Size  Type  File system  Flags

(parted) mkpart primary 1MiB 245MiB
(parted) set 1 raid on
(parted) mkpart primary 245MiB 2199MiB
(parted) set 2 raid on
(parted) mkpart primary 2199MiB 4153MiB
(parted) set 3 raid on
(parted) mkpart extended 4153MiB 100%

This gave a drive with three primary partitions starting on MiB boundaries, and an extended partition which covered the remainder (majority) of the drive. Each primary partition was marked as a RAID partition.

After that it got more complicated. I needed to start each logical partition on a MiB boundary, and then finishing them just before the MiB boundary, to allow room for the Extended Boot Record to sit in between. For the first logical partition I chose to sacrifice 1 MiB, and simply start it on the next MiB boundary, but for the end position I needed to figure out "4KiB less than next MiB" (ie, one physical sector) so as to leave room for the EBR and then starting the next logical partition on a MiB boundary -- with minimal wasted space.

I calculated the first one by hand, as it needed a unique size, and specified it in sectors (ie, 512-byte units -- logical sectors):

(parted) mkpart logical 4154MiB 72511484s      # 35406MiB - 4 sectors
(parted) set 5 raid on

then for most of the rest, they were all the same size, and so the pattern was quite repetitive. To solve this I wrote a trivial, hacky Python script to generate the right parted commands:

base=35406
inc=250004

for i in range(7):
  start = base + (i * inc)
  end   = base + ((i+1) * inc)
  last  = (end * 2 * 1024) - 4
  print("mkpart logical {0:7d}MiB {1:10d}s    # {2:7d}MiB - 4 sectors".format(start, last, end))

and then fed that output to parted:

(parted) mkpart logical   35406MiB  584519676s    #  285410MiB - 4 sectors
(parted) mkpart logical  285410MiB 1096527868s    #  535414MiB - 4 sectors
(parted) mkpart logical  535414MiB 1608536060s    #  785418MiB - 4 sectors
(parted) mkpart logical  785418MiB 2120544252s    # 1035422MiB - 4 sectors
(parted) mkpart logical 1035422MiB 2632552444s    # 1285426MiB - 4 sectors
(parted) mkpart logical 1285426MiB 3144560636s    # 1535430MiB - 4 sectors
(parted) mkpart logical 1535430MiB 3656568828s    # 1785434MiB - 4 sectors

to create all the consistently sized partitions (this would have been easier if parted had supported start/size values, which is what is actually stored in the MBR/EBR -- but it requires start/end values, which need more manual calculation; seems like a poor UI to me).

After that I could create the final partition to use the remainder of the disk, which is trivial to specify:

(parted) mkpart logical 1785434MiB 100%

and then mark all the other partitions as "raid" partitions:

(parted) set 6 raid on
(parted) set 7 raid on
(parted) set 8 raid on
(parted) set 9 raid on
(parted) set 10 raid on
(parted) set 11 raid on
(parted) set 12 raid on
(parted) set 13 raid on

which gave me a final partition table of:

(parted) unit s
(parted) print
Model: WD Elements 25A2 (scsi)
Disk /dev/sdc: 3906963456s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start        End          Size         Type      File system  Flags
 1      2048s        501759s      499712s      primary                raid, lba
 2      501760s      4503551s     4001792s     primary                raid, lba
 3      4503552s     8505343s     4001792s     primary                raid, lba
 4      8505344s     3906963455s  3898458112s  extended               lba
 5      8507392s     72511484s    64004093s    logical                raid, lba
 6      72511488s    584519676s   512008189s   logical                raid, lba
 7      584519680s   1096527868s  512008189s   logical                raid, lba
 8      1096527872s  1608536060s  512008189s   logical                raid, lba
 9      1608536064s  2120544252s  512008189s   logical                raid, lba
10      2120544256s  2632552444s  512008189s   logical                raid, lba
11      2632552448s  3144560636s  512008189s   logical                raid, lba
12      3144560640s  3656568828s  512008189s   logical                raid, lba
13      3656568832s  3906963455s  250394624s   logical                raid, lba

(parted)

As a final double check I also used the parted "align-check" to check the alignment (and manually divided each starting sector by 2048 to ensure it was on a MiB boundary -- 2048 = 2 * 1024, as the values are in 512-byte sectors):

(parted) align-check optimal 1
1 aligned
(parted) align-check optimal 2
2 aligned
(parted) align-check optimal 3
3 aligned
(parted) align-check optimal 4
4 aligned
(parted) align-check optimal 5
5 aligned
(parted) align-check optimal 6
6 aligned
(parted) align-check optimal 7
7 aligned
(parted) align-check optimal 8
8 aligned
(parted) align-check optimal 9
9 aligned
(parted) align-check optimal 10
10 aligned
(parted) align-check optimal 11
11 aligned
(parted) align-check optimal 12
12 aligned
(parted) align-check optimal 13
13 aligned
(parted)

And then exited to work with this final partition table:

(parted) quit

For more on partition alignment, particularly with MD / LVM layers as well see Thomas Krenn's post on Partition Alignment, and a great set of slides on partition / MD LVM alignment. Of note, both Linux MD RAID-1 metadata (1.2) and LVM Physical Volume metadata will take up some space at the start of the partition if you accept the modern defaults.

For Linux MD RAID-1, metadata 1.2 is at the start of the partition, and then the Data will begin at the "Data Offset" within the partition. (Linux MD RAID metadata 0.9 is at the end of the disk, so there is no offset, which is sometimes useful including for /boot partitions.) You can see the offset in use by examining the individual RAID-1 elements: on metadata 1.2 RAID sets there is a "Data Offset" value reported, which is typically 1 MiB (2048 * 512-byte sectors):

ewen@linux:~$ sudo mdadm -E /dev/sda2 | egrep "Version|Offset"
        Version : 1.2
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
ewen@linux:~$ 

although RAID sets created with more modern mdadm tools might have larger offsets (possibly for bitmaps to speed up resync?):

ewen@linux:~$ sudo mdadm -E /dev/sda13 | egrep "Version|Offset"
        Version : 1.2
    Data Offset : 131072 sectors
   Super Offset : 8 sectors
ewen@linux:~$ 

These result in unused space in the MD RAID-1 elements which can be seen:

ewen@linux:~$ sudo mdadm -E /dev/sda2 | egrep "Unused"
   Unused Space : before=1960 sectors, after=1632 sectors
ewen@linux:~$ sudo mdadm -E /dev/sda13 | egrep "Unused"
   Unused Space : before=130984 sectors, after=65536 sectors
ewen@linux:~$ 

although in this case the unused space at the end is most likely due to rounding up the partition sizes from those in the originally created RAID array. (The "--data-offset is computed automatically, but may be overridden from the command line when the array is created -- on a per-member-device basis. But presumably if the data offset is too small, various metadata -- such as bitmaps -- cannot be stored.)

By default, it appears that modern LVM will default to 192 KiB of data at the start of its physical volumes (PV), which can be seen by checking the "pe_start" value:

ewen@linux:~$ sudo pvs -o +pe_start /dev/md26
  PV         VG Fmt  Attr PSize   PFree 1st PE 
  /dev/md26  r1 lvm2 a--  244.12g    0  192.00k
ewen@linux:~$ sudo pvs -o +pe_start /dev/md32
  PV         VG Fmt  Attr PSize   PFree   1st PE 
  /dev/md32     lvm2 ---  244.14g 244.14g 192.00k
ewen@linux:~$ 

and controlled at pvcreate time with the --metadatasize and --dataalignment values (as well as an optimal manual override of the offset).

Fortunately all of these values (1MiB == 2048s, 64MiB == 131072s, 192 KiB) are all multiples of 4 KiB, so providing you are only aligning to 4 KiB boundaries you do not need to worry about additional alignment options if the underlying partitions are aligned. But if you need to align to, eg, larger SSD erase blocks or larger hardware RAID stripes, you may need to adjust the MD and LVM alignment options as well to avoid leaving the underlying file system misaligned. (If you are using modern Linux tools with all software layers -- RAID, LVM, etc -- then the alignment_offset values will probably help to ensure the defaults help with alignment; if there are any hardware layers you will need to provide additional information to ensure the best alignment.)

RAID rebuild

Having created a third (external 2TB) disk with suitably aligned partitions I could then move on to resyncing all the RAID arrays 3 times (once onto the external drive, and then once onto each of the internal drives). A useful Debian User post outlined the process for extending the RAID-1 array onto a third disk, and then removing the third disk again, which provided the basis of my approach.

The first step was to extend all but the last RAID array onto the new disk (the last one needed special treatment as it was getting smaller, but fortunately it did not have any data on it yet). Growing onto the third disk is fairly simple:

sudo mdadm --grow /dev/md21 --level=1 --raid-devices=3 --add /dev/sdc1
sudo mdadm --grow /dev/md22 --level=1 --raid-devices=3 --add /dev/sdc2
sudo mdadm --grow /dev/md23 --level=1 --raid-devices=3 --add /dev/sdc3
sudo mdadm --grow /dev/md25 --level=1 --raid-devices=3 --add /dev/sdc5
sudo mdadm --grow /dev/md26 --level=1 --raid-devices=3 --add /dev/sdc6
sudo mdadm --grow /dev/md27 --level=1 --raid-devices=3 --add /dev/sdc7
sudo mdadm --grow /dev/md28 --level=1 --raid-devices=3 --add /dev/sdc8
sudo mdadm --grow /dev/md29 --level=1 --raid-devices=3 --add /dev/sdc9
sudo mdadm --grow /dev/md30 --level=1 --raid-devices=3 --add /dev/sdc10
sudo mdadm --grow /dev/md31 --level=1 --raid-devices=3 --add /dev/sdc11
sudo mdadm --grow /dev/md32 --level=1 --raid-devices=3 --add /dev/sdc12

although as mentioned above it did take a long time (most of a calendar day) due to the external drive being connected via USB-2, and thus limited to about 30MB/s.

The second step was to destroy the RAID array for the last partition on the disks, discarding all data on it (fortunately none in my case), as that partition had to get smaller as described above. If you have important data on that last RAID array you will need to copy it somewhere else before proceeding.

ewen@linux:~$ head -4 /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md33 : active (auto-read-only) raid1 sdb13[0]
      125233580 blocks super 1.2 [2/1] [U_]

ewen@linux:~$ sudo mdadm --stop /dev/md33
[sudo] password for ewen:
mdadm: stopped /dev/md33
ewen@linux:~$ sudo mdadm --remove /dev/md33
ewen@linux:~$ grep md33 /proc/mdstat
ewen@linux:~$ sudo mdadm --zero-superblock /dev/sda13
ewen@linux:~$ sudo mdadm --zero-superblock /dev/sdb13
ewen@linux:~$

After this, check the output of /proc/mdstat to ensure that all the remaining RAID sets are happy, and show three active disks ("UUU") -- the two original disks, and the temporary external disk. If you are not sure everything is perfectly prepared, sort out the remainining issues before proceeding, as the next step will break the first original drive out of the RAID-1 arrays.

The third step, when everything is ready, is to remove the first original drive from the RAID-1 arrays:

sudo mdadm /dev/md21 --fail /dev/sda1  --remove /dev/sda1
sudo mdadm /dev/md22 --fail /dev/sda2  --remove /dev/sda2
sudo mdadm /dev/md23 --fail /dev/sda3  --remove /dev/sda3
sudo mdadm /dev/md25 --fail /dev/sda5  --remove /dev/sda5
sudo mdadm /dev/md26 --fail /dev/sda6  --remove /dev/sda6
sudo mdadm /dev/md27 --fail /dev/sda7  --remove /dev/sda7
sudo mdadm /dev/md28 --fail /dev/sda8  --remove /dev/sda8
sudo mdadm /dev/md29 --fail /dev/sda9  --remove /dev/sda9
sudo mdadm /dev/md30 --fail /dev/sda10 --remove /dev/sda10
sudo mdadm /dev/md31 --fail /dev/sda11 --remove /dev/sda11
sudo mdadm /dev/md32 --fail /dev/sda12 --remove /dev/sda12

and then repartition the drive following the instructions above (ie, to be identical to the 2TB external drive, other than the size of the final partition).

When the partitioning is complete, run:

ewen@linux:~$ sudo partprobe -d -s /dev/sda
/dev/sda: msdos partitions 1 2 3 4 <5 6 7 8 9 10 11 12 13>
ewen@linux:~$ sudo partprobe  -s /dev/sda
/dev/sda: msdos partitions 1 2 3 4 <5 6 7 8 9 10 11 12 13>
ewen@linux:~$

to ensure the new partitions are recognised, and then also compare the output of:

ewen@linux:~$ sudo parted -a optimal /dev/sda unit s print
Model: ATA WDC WD20EFRX-68A (scsi)
Disk /dev/sda: 3907029168s
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start        End          Size         Type      File system  Flags
 1      2048s        501759s      499712s      primary                raid
 2      501760s      4503551s     4001792s     primary                raid
 3      4503552s     8505343s     4001792s     primary                raid
 4      8505344s     3907028991s  3898523648s  extended               lba
 5      8507392s     72511484s    64004093s    logical                raid
 6      72511488s    584519676s   512008189s   logical                raid
 7      584519680s   1096527868s  512008189s   logical                raid
 8      1096527872s  1608536060s  512008189s   logical                raid
 9      1608536064s  2120544252s  512008189s   logical                raid
10      2120544256s  2632552444s  512008189s   logical                raid
11      2632552448s  3144560636s  512008189s   logical                raid
12      3144560640s  3656568828s  512008189s   logical                raid
13      3656568832s  3907028991s  250460160s   logical                raid

ewen@linux:~$

with the start/size sectors recognised by Linux as active:

ewen@linux:/sys/block/sda$ for PART in 1 2 3 5 6 7 8 9 10 11 12 13; do echo "sda${PART}: " $(cat "sda${PART}/start") $(cat "sda${PART}/size"); done
sda1:  2048 499712
sda2:  501760 4001792
sda3:  4503552 4001792
sda5:  8507392 64004093
sda6:  72511488 512008189
sda7:  584519680 512008189
sda8:  1096527872 512008189
sda9:  1608536064 512008189
sda10:  2120544256 512008189
sda11:  2632552448 512008189
sda12:  3144560640 512008189
sda13:  3656568832 250460160
ewen@linux:/sys/block/sda$

to ensure that Linux will copy onto the new partitions, not old locations on the disk.

The fourth step is to add the original first drive back into the RAID-1 sets, and wait for them to all resync:

sudo mdadm --manage /dev/md21 --add /dev/sda1
sudo mdadm --manage /dev/md22 --add /dev/sda2
sudo mdadm --manage /dev/md23 --add /dev/sda3
sudo mdadm --manage /dev/md25 --add /dev/sda5
sudo mdadm --manage /dev/md26 --add /dev/sda6
sudo mdadm --manage /dev/md27 --add /dev/sda7
sudo mdadm --manage /dev/md28 --add /dev/sda8
sudo mdadm --manage /dev/md29 --add /dev/sda9
sudo mdadm --manage /dev/md30 --add /dev/sda10
sudo mdadm --manage /dev/md31 --add /dev/sda11
sudo mdadm --manage /dev/md32 --add /dev/sda12

which in my case took about 6 hours.

Once this is done, the same steps can be repeated to remove the /dev/sdb* partitions, repartition the /dev/sdb drive, re-check the partitions are correctly recognised, and then re-add the /dev/sdb* partitions into the RAID sets.

Of note, when I started adding the /dev/sda* partitions back in after repartitioning, I got warnings saying:

ewen@linux:/sys/block$ sudo dmesg -T | grep misaligned
[Mon Jan  1 09:22:20 2018] md21: Warning: Device sda1 is misaligned
[Mon Jan  1 09:22:35 2018] md22: Warning: Device sda2 is misaligned
[Mon Jan  1 10:07:26 2018] md27: Warning: Device sda7 is misaligned
[Mon Jan  1 10:49:12 2018] md28: Warning: Device sda8 is misaligned
[Mon Jan  1 10:49:21 2018] md29: Warning: Device sda9 is misaligned
[Mon Jan  1 11:30:04 2018] md30: Warning: Device sda10 is misaligned
[Mon Jan  1 12:26:56 2018] md31: Warning: Device sda11 is misaligned
[Mon Jan  1 12:45:08 2018] md32: Warning: Device sda12 is misaligned
ewen@linux:/sys/block$

and when I went checking I found that the "alignment_offset" values had been set to "-1" in the affected cases:

ewen@tv:/sys/block$ grep . md*/alignment_offset
md21/alignment_offset:-1
md22/alignment_offset:-1
md23/alignment_offset:0
md25/alignment_offset:0
md26/alignment_offset:0
md27/alignment_offset:-1
md28/alignment_offset:-1
md29/alignment_offset:-1
md30/alignment_offset:-1
md31/alignment_offset:-1
md32/alignment_offset:-1
ewen@tv:/sys/block$ grep . md*/alignment_offset

Those alignment offsets should normally be bytes to adjust by to achieve alignment again -- I saw values like 3072, 3584, etc, in them prior to aligning the underlying physical partitions properly, and "0" indicates that it is natively aligned already.

After some hunting it turned out that "-1" was a special magic value meaning basically "alignment is impossible":

 *    Returns 0 if the top and bottom queue_limits are compatible.  The
 *    top device's block sizes and alignment offsets may be adjusted to
 *    ensure alignment with the bottom device. If no compatible sizes
 *    and alignments exist, -1 is returned and the resulting top
 *    queue_limits will have the misaligned flag set to indicate that
 *    the alignment_offset is undefined.

My conclusion was that because the RAID-1 sets had stayed active throughout what was happening was that the previous alignment offset of /dev/sda* partitions was non-zero, and the RAID-1 sets were attempting to find an alignment offset for the new /dev/sda* partitions that would match the physical sectors and use those same offsets -- but there was no such valid offset that matched both the old and new /dev/sda* partition offsets, hence it failed. So I chose to ignore those warnings and carry on.

Once both original drives had been repartitioned and resync'd, the next step was to recreate the /dev/md33 RAID partition again, on the smaller partitions:

ewen@tv:~$ sudo mdadm --zero-superblock /dev/sda13
ewen@tv:~$ sudo mdadm --zero-superblock /dev/sdb13
ewen@tv:~$ sudo mdadm --zero-superblock /dev/sdc13
ewen@tv:~$ sudo mdadm --create /dev/md33 --level=1 --raid-devices=3 --chunk=4M /dev/sda13 /dev/sdb13 /dev/sdc13
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md33 started.
ewen@tv:~$

(because I was not booting from that partition metadata 1.2 was fine, and gave more options -- this one was created with recovery bitmaps).

Note that in this case I chose to create the RAID-1 set including three drives, because the external 2TB drive was slightly smaller, and I wanted the option of later resync'ing it onto that drive as an offsite backup.

At this point it is useful to update /etc/mdadm/mdadm.conf with the new UUID of the new RAID set, to ensure that it stays in sync and RAID arrays can be auto-started.

When that new RAID set completed resync'ing, I then removed the 2TB external drive from all the RAID sets, and set them back to "2-way" RAID to avoid the RAID sets sitting there partly failed:

sudo mdadm /dev/md21 --fail /dev/sdc1  --remove /dev/sdc1
sudo mdadm /dev/md22 --fail /dev/sdc2  --remove /dev/sdc2
sudo mdadm /dev/md23 --fail /dev/sdc3  --remove /dev/sdc3
sudo mdadm /dev/md25 --fail /dev/sdc5  --remove /dev/sdc5
sudo mdadm /dev/md26 --fail /dev/sdc6  --remove /dev/sdc6
sudo mdadm /dev/md27 --fail /dev/sdc7  --remove /dev/sdc7
sudo mdadm /dev/md28 --fail /dev/sdc8  --remove /dev/sdc8
sudo mdadm /dev/md29 --fail /dev/sdc9  --remove /dev/sdc9
sudo mdadm /dev/md30 --fail /dev/sdc10 --remove /dev/sdc10
sudo mdadm /dev/md31 --fail /dev/sdc11 --remove /dev/sdc11
sudo mdadm /dev/md32 --fail /dev/sdc12 --remove /dev/sdc12
sudo mdadm /dev/md33 --fail /dev/sdc13 --remove /dev/sdc13

sudo mdadm --grow /dev/md21 --raid-devices=2
sudo mdadm --grow /dev/md22 --raid-devices=2
sudo mdadm --grow /dev/md23 --raid-devices=2
sudo mdadm --grow /dev/md25 --raid-devices=2
sudo mdadm --grow /dev/md26 --raid-devices=2
sudo mdadm --grow /dev/md27 --raid-devices=2
sudo mdadm --grow /dev/md28 --raid-devices=2
sudo mdadm --grow /dev/md29 --raid-devices=2
sudo mdadm --grow /dev/md30 --raid-devices=2
sudo mdadm --grow /dev/md31 --raid-devices=2
sudo mdadm --grow /dev/md32 --raid-devices=2
sudo mdadm --grow /dev/md33 --raid-devices=2

and then checked for any remaining missing drive references or "sdc" references:

ewen@linux:~$ cat /proc/mdstat | grep "_"
ewen@linux:~$ cat /proc/mdstat | grep "sdc"
ewen@linux:~$

Then I unplugged the 2TB external drive, to keep for now as an offline backup.

To make sure that the system still booted, I reinstalled grub and updated the initramfs to pick up the new RAID UUIDs:

ewen@linux:~$ sudo grub-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
ewen@linux:~$ sudo grub-install /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.
ewen@linux:~$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.9.0-4-686-pae
ewen@linux:~$
ewen@linux:~$ sudo update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.9.0-4-686-pae
Found initrd image: /boot/initrd.img-4.9.0-4-686-pae
Found linux image: /boot/vmlinuz-4.9.0-3-686-pae
Found initrd image: /boot/initrd.img-4.9.0-3-686-pae
Found linux image: /boot/vmlinuz-3.16.0-0.bpo.4-686-pae
Found initrd image: /boot/initrd.img-3.16.0-0.bpo.4-686-pae
Found memtest86 image: /memtest86.bin
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
ewen@linux:~$

and then rebooted the system to make sure it could boot cleanly by itself. Fortunately it rebooted automatically without any issues!

After reboot I checked for reports of misalignment:

ewen@linux:~$ uptime
 11:53:28 up 4 min,  1 user,  load average: 0.05, 0.42, 0.25
ewen@linux:~$ sudo dmesg -T | grep -i misaligned
ewen@linux:~$ sudo dmesg -T | grep alignment
ewen@linux:~$ sudo dmesg -T | grep inconsistency
ewen@linux:~$

and was pleased to find that none were reported. I also checked all the alignment_offset values, and was pleased to see all of those were now "0" -- ie "naturally aligned" (in this case to the 4KiB physical sector boundaries):

ewen@linux:~$ cat /sys/block/sda/sda*/alignment_offset
0
0
0
0
0
0
0
0
0
0
0
0
0
ewen@linux:~$ cat /sys/block/sdb/sdb*/alignment_offset
0
0
0
0
0
0
0
0
0
0
0
0
0
ewen@linux:~$ cat /sys/block/md*/alignment_offset
0
0
0
0
0
0
0
0
0
0
0
0
ewen@linux:~$ cat /sys/block/dm*/alignment_offset
0
0
0
0
0
0
0
ewen@linux:~$

It is too soon to tell if this has any actual practical benefits in performance due to improving the alignment. But not being reminded that I "did it wrong" several years ago when putting the disks in -- due to the fdisk partition defaults at the time being wrong for 4 KiB physical sector disks -- seems worth the effort anyway.

Posted Tue Jan 2 14:13:30 2018 Tags: