Fundamental Interconnectedness

This is the occasional blog of Ewen McNeill. It is also available on LiveJournal as ewen_mcneill, and Dreamwidth as ewen_mcneill_feed.

Introduction

The "big news" of 2020 has been the first mass spreading infectious disease with a non-trivial death rate, for which there is no current vaccination, in many decades -- Covid-19. Cases were diagnosed in New Zealand from 2020-02-28 onwards -- initially in those who had contracted Covid-19 overseas and then returned/travelled to New Zealand.

As the number of Covid-19 cases both overseas and in New Zealand continued to rise, the New Zealand Government launched the covid19.govt.nz site, and the 4-level Alert System. The website was launched before 2020-03-18 (which is the first archive), and the 4-level Alert System was launched 2020-03-21, initially at one URL on 2020-03-21 (with a brief New Zealand Covid-19 Alert Levels PDF for reference); and then a few weeks later at a different URL, from 2020-04-07 with more detail. (There were some other PDFs with details of the levels, in between, referenced below as relevant.)

The stated meanings of the levels are:

  • Alert Level 1 -- Prepare

  • Alert Level 2 -- Reduce

  • Alert Level 3 -- Restrict

  • Alert Level 4 -- Lockdown

but the actual implementation of these levels has varied over time. In particular the definition of Level 2 and Level 3 before Level 4, and the definition of Level 3 and Level 2 after Level 4 are noticeably different.

Timeline

NZ Covid-19 Alert Level 1 -- Prepare

In practice New Zealand has never actually been at Covid-19 Alert level 1: by the time the Alert Level System was announced (2020-03-21) we were already at Alert Level 2, and realistically had been since about 2020-03-19 when larger indoor events were restricted. So the actual definition of Covid-19 Alert Level 1 does not matter at present as it is inevitable (given the changes noted below) that it will change before we ever reach Covid-19 Alert Level 1.

In theory Alert Level 1 means (compare with 2020-05-21 version:

  • Border restrictions

  • Intensive Covid-19 testing

  • Contact tracing

  • Self isolation when relevant

  • Gatherings over 500 cancelled

It seems likely that New Zealand will reach some version of the Covid-19 Alert Level 1 at some point later in 2020 (ie, border restrictions and large gatherings prohibited, but routine day to day interactions largely as they were in 2019 and earlier). But the timing of that is currently unknown and unknowable.

NZ Covid-19 Alert Level 2 -- Reduce

At the time of writing, officially New Zealand has had two periods of "Covid-19 Alert Level 2":

  • 2020-03-21 xx:xx NZDT to 2020-03-23 13:49 NZDT ("March")

  • 2020-05-13 23:59 NZST onwards ("May")

In practice the 2020-03-21 announcement of the "Alert Levels" and the current position at Alert Level 2 simply recorded where New Zealand already was, and realistically New Zealand had been at something equivalent to Alert Level 2 from 2020-03-19 23:59 NZDT when the borders closed and events over 100 people were restricted.

There was not a separate "Alert Level 2" page for the March version so those details have been taken from the overall Alert Level Summaries.

2020-03-21 view of March Alert Level 2

  • Maximum border restrictions

  • Restrictions on mass gatherings (implied: over 100, by 2020-03-19 announcement)

  • Physical distancing on public transport

  • Limit non-essential travel around New Zealand

  • Alternative working (remote working, staggered working, etc); activate Business Continuity Plans

  • High risk people to remain at home (over 70 / other medical conditions)

"Level 2" stayed with the same restrictions through 2020-03-23 when we moved to Alert Level 3, and on into 2020-03-26 when we were in Alert Level 4.

"Level 2" stayed the same through the change in URL for the Covid-19 Alert System on 2020-04-07. And the same up through 2020-04-15.

2020-04-16 definition of Alert Level 2

The first substantial change of Alert Level 2 happened around 2020-04-16, changing to:

  • Physical distancing of 1 metre

  • Gatherings of 100 people indoors, 500 people outdoors

  • Most business open for staff and customers (but alternative ways of working encouraged)

  • Schools open

  • High risk people encouraged to remain at home

This definition remained up through 2020-05-05.

See also detailed Alert Levels PDF from 2020-05-04.

2020-05-07 definition of Alert Level 2

A second substantial change of "Alert Level 2" happened around 2020-05-07:

  • Physical distancing of 2 metres, except in controlled environments (eg workplace) of 1 metre

  • Gatherings of 100 people indoors or outdoors

  • Most business open for staff and customers, if following physical distancing and contact tracing rules

  • Schools open

  • High risk people encouraged to remain at home

Around 2020-05-07 also marked the introduction of a dedicated Alert Level 2 page (supposedly "edited 2020-05-04"), with more detail, including:

  • Indoor gatherings should be seated, and be approximately two hours long

  • Take extra care if you interact with people you do not know (because it is harder to do contact tracing)

  • Specific guidance to come for close personal contact work (eg, hair dressers)

These rules stayed the same until around 2020-05-10.

2020-05-11 definition of Alert Level 2

The first major change to the May Alert Level 2 rules happened around 2020-05-11, a couple of days before they were due to come into effect:

  • Physical distancing of 2 metres, except in controlled environments (eg workplace) of 1 metre

  • Gatherings of up to 10 people in your house

  • Gatherings outside your house of up to 10 people for up to 2 hours, with attendees recorded

  • Most business open for staff and customers, with contact registers/ contact tracing records for "everyone who you interact with on your premises". Up to 10 people in a group, up to 2 hours.

  • Specific guidance to come for close personal contact work (eg, hair dressers)

  • Limit use of public transport, avoid sitting next to someone you do not know, or standing

  • Schools open

See also detailed Alert Level 2 PDF from 2020-05-11

Earlier on 2020-05-11 gatherings in your own home were limited to "up to 2 hours", but that restriction was removed by the next morning.

2020-05-12 definition of Alert Level 2

Similar to 2020-05-11, with the notable changes being:

  • Keep your distance when out and about (ideally 2 metres), not just from people you do not know

  • Schools to open from 2020-05-18 (ie, the Monday)

  • Detailed guidance for public venues like libraries and museums to be released.

  • Duck Shooting Season starts 2020-05-23

  • 2 Hour limit for customers on your premises (not just groups)

plus a bunch of rewording of text for clarity (but with the same meaning).

2020-05-13 definition of Alert Level 2

By 2020-05-13 there was a change to the definition that permitted up to 50 people to attend a funeral or tangihanga.

2020-05-14 definition of Alert Level 2

By the 2020-05-14 version of Alert Level 2:

  • retailers no longer needed to "keep records of customers to enable contact tracing" (but it appears quite a few of them are requiring some form of contact tracing for entry by customers anyway, even as late as 2020-05-20); and

  • "There is no maximum number of customers allowed in a store, as long as they can keep 2 metres apart at all times" (but nearly all stores were still limiting numbers entering as of 2020-05-20, leading to queues outside).

  • Businesses requiring close personal contact (hairdressers, home help, etc) can operate providing they keep contact tracing registers, keep customers 1 metre apart, and disinfect between customers. They do not have to wear personal protective equipment (PPE) but workers/customers may prefer them to do so.

  • Alcohol can only be served to people eating a meal (up to 2020-05-21)

  • If more than 10 people normally live in your house, no more can join.

Plus there were a lot of edits to change from future tense to present tense, as by 2020-05-14 we were in Alert Level 2, May edition.

These rules remained unchanged up through 2020-05-20.

The rule around alcohol was silently removed somewhere around 2020-05-21/2020-05-22 now that it no longer applies, without changing the "Last Modified" date (and unfortunately that change is not being reflected in the Internet Archive Wayback captures yet); the page was modified 2020-05-22 02:04:22 NZST according to the live headers. But that appears to be the only change since 2020-05-20 other than some additional links silently added.

Alert Level 3 and Alert Level 4

There were changes to the definitions of Alert Level 3 and Alert Level 4 while they were implemented, but they were less substantial than the changes to Alert Level 2 between March/April/May, so they are not analysed in detail here.

The main change of significance in Alert Level 4 was clarifications around "Shared Bubbles" -- shared custody, or individuals living alone who could join up with another household for support.

Final note

This blog post would have been impossible to write without the Internet Archive Wayback Machine; if you found this blog post useful please consider donating to the Internet Archive to help support archiving the web.

ETA 2020-05-29:

On 2020-05-25 it was announced that the Level 2 gathering limit would rise to 100 people from 2020-05-29 12:00 (noon, NZST).

There appear to be new rules for social gatherings in your home

A business or responsible individual, in charge of a social gathering, must ensure records are kept for contact tracing purposes, except where every person in a gathering knows each other. This includes in your home, community hall, or other such space.

Ie, there is again a requirement to keep contact tracing records for gatherings in your home "except where every person in a gathering knows each other". (So, presumably, if you invite two friends who do not already know each other, then the contact tracing requirement kicks in.)

Compare capture from 2020-05-29 11:05 NZST with capture from 2020-05-29 12:05 NZST. There is no longer a distinction between "in home" and "outside home" gatherings: it seems like the same rules, including contact tracing records, apply to both.

And there are a large number of other detailed rule changes too, presumably to try to make the "100" practical for various venues, including "separate spaces" within a building that might be one business but basically function in isolation.

ETA 2020-08-14:

Predictibly, there was another (currently small) community outbreak in New Zealand (in Auckland), leading to a change to Alert Level 3 in Auckland, and Alert Level 2 in the rest of New Zealand, soon followed by a public health direction relating to two businesses, a requirement for all businesses to display Covid19 Tracing QR code posters, and an emphasis on face coverings (which given the short notice led to pre-made masks being sold out most places).

The definition of Level 2 moved to yet another different URL (one deeper in the URL hierachy), and changed again, eg, the QR code requirement; see the Wayback Archive of the new Level 2 page.

It looks like the definition of Level 2 in detail just vanished from the menu until 2020-08-12; only Level 1 was in the menu; but there was a early August Level 2 definition at yet another URL.

ETA 2020-09-01:

On Sunday 2020-08-30, when the Auckland Region moved down levels somewhat from Level 3, but not as far as the "rest of New Zealand" Level 2, the Prime Minister invented a Level 2.5 for Auckland, between their Level 3 and the Level 2 of the rest of the country (archive; see also specific instructions for Auckland).

The Auckland "Level 2.5" is roughly the same restrictions as the May 2020 Level 2 (ie, smaller groups, limits on funerals etc), just with mandatory face coverings in some situations and mandatory contact tracing QR code display.

The "Level 2" for the rest of New Zealand is unchanged from earlier in August, but now includes mandatory contact tracing QR code display (active since mid August), and strongly encouraged face coverings in situations where physical distancing is not possible (eg, public transport).

The "Level 2.5" is strong proof that in hindsight the NZ Level system really needed one more level -- between Level 2 and Level 3. It would have allowed far less "constantly redefining what Level 2 means", and much more certainty.

Posted Fri May 22 14:15:07 2020 Tags:

Introduction

I have wanted a 3D printer ever since Vik Olliver spoke at Linux.Conf.Au 2006 about the RepRap project -- a 3D printer that "prints itself" -- and then showed working models at later Linux.Conf.Au conferences. Vik went on to spend years working on the RepRap project, helping open up 3D printing to makers.

The whole idea of a "self replicating machine" was inspiring,and over the years I have seen many makers produce lots of custom one off designs using 3D printers, as the technology has matured. However I had no practical use for a 3D Printer, particular at the early innovation stage, and so I just looked on in admiration from a distance.

Over the past couple of years FDM 3D printing ("Fused Deposition Modelling", sometimes called "Fused Filament Fabrication" or "FFF") has matured to the point that there are numerous 3D printing machines, many of which descend directly or indirectly from the RepRap project.

The "Original Prusa I3 MK3s" is one of the best known for maximum refinement of the RepRap deisgn, starting with the Prusa Mendel, and has great reviews. But it is also US$750 even as a DIY kit, plus shipping, and it's costly enough to require paying GST on the way into New Zealand as well (so all up NZ$1350 or so). Which still was more than I could justify for an something to use occasionally to make custom parts, and "experiment with 3D design/printing".

Creality Ender 3

After a bit of a rocky start (with controversy around source license compliance, and quality control/cost optimisation choices), the Creality Ender 3 has emerged as one of the most popular, highly reviewed budget 3D printers, and considered a "good printer for the price" particularly when it is on sale (Ender 3 Unboxing).

I have been aware of Creality for a couple of years, ever since Naomi Wu started working with Creality to improve their relationship with the western maker community (both Creality and Naomi are based in Shenzhen, Guangdong, China). Naomi's work led to the Creality Ender 3 being one of the first Open Source Hardware Association Certified projects out of China (Ender 3 OSHWA certificate -- CN000003; see also Naomi's announcement video), with the firmware source code, PCB design, and mechanical hardware all on GitHub. (Naomi is also responsible for OSHWA CN000001, and encouraging OSHWA CN000004 and OSHWA CN000005 -- the first for her sino:bit project, and the other two for the Creality CR-10 3D Printer.)

A few weeks ago, reminded by the USA "4th of July" sales, I happened to see the Creality Ender 3 on sale at Kiwi 3D for NZ$500, including "free" shipping and GST, and finally ordered one -- spending the $NZ$100 discount on some PLA filament to have something to start testing the printer with. (While the printer was available from overseas, including direct from Creality for cheaper -- US$189.99 is a common sale price -- by the time that shipping and GST at import were added, the sale price in New Zealand was was less than 20% more than the likely landed price importing it myself -- and by buying from a New Zealand retailer it arrived just over one business day after ordering, with much less hassle.)

The Creality Ender 3 arrives as a "partially assembled" kit -- the base print bed / Y axis, and the extruder unit are assembled, and the remainder is flat packed for end user assembly. I spent an afternoon carefully assembly the kit, following the Tomb of 3D Printed Horrors "Creality Ender 3 assembly and pro build tips" video guide, as well as the "Creality Ender-3 Official Assembly Instruction" PDF (downloaded from Creality's site); the PDF guide has more instructional text than the provided "one large sheet" diagram.

Other than challenges identifying the correct parts needed at each step (there are a lot of subtly different bolt sizes for instance and extras supplied for many of them), the assembly went fairly smoothly -- and the "Tomb of 3D Printed Horrors" build video helped identify some things to check during the build at times when they were easy to change, avoiding finding those issues later when they were more difficult to correct. (Of note, my base arrived so that it was slightly wobbly on a flat surface, and a couple of the bed levelling knobs and springs had come loose during shipping; but putting the assembled unit on a flat surface and slightly loosening the two bolts holding the right -- LCD panel -- side base on to the unit was enough to allow everything to drop into alignment with no wobble, and the bed levelling springs and knobs were easy to put back onto the base. The only other surprise I found was that the control box fan does not spin all the time, seemingly not even at the start of a print -- but it definitely spins once the printing is under way, so presumably there is a thermal sensor for the control box fan now too. My XT60 power supply connectors appear to be genuine, which was not true of all earlier Creality Ender 3 printers due to a quality control issue.)

After assembly, and manual bed levelling, my Ender 3 printed the "Tomb of 3D Printed Horrors" bed levelling test model (downloaded from their DropBox) successfully, and I declared success on the initial assembly. (The "bed levelling test" was supplied as G-Code, which was convenient for just getting started -- but fortunately also short enough I could manually review it to determine it was safe before sending it to my printer; as Thomas Sanladerer points out G-Code can include instructions which make changes that persist over power cycling the device, or attempt to cause mechanical damage, so verifying G-Code is important if it comes from an unknown origin.)

Ultimaker Cura

The usual exchange format for 3D printing models is STL, and this is what is available from, eg, Thingiverse and other 3D model sites. STL is a widely supported 3D model format that describes 3D objects in terms of meshes of triangles (and is encoded either in ASCII, or in a binary format; not all STL files are 3D printable but most 3D printable models are provided as STL files).

For 3D FDM printing, the STL model needs to be translated by a "slicer", which decides how to move the X / Y / Z axes of the printer to produce a physical representation of the model, and produces a GCode file to instruct the printer on what to do. In software terms the STL file is the source code, the slicer is the compiler, and the GCode is the executable file (which is one reason why GCode from unknown sources cannot be completely trusted; see above).

There are several Open Source 3D slicers, including:

I decided to start with Cura because Maker's Muse found that the Cura slicer chose movement paths that better reduced filament stringing on the Creality Ender 3, Thomas Pietrowski maintains an Ubuntu PPA for Cura (stable version), and several makers published Cura profiles for the Ender 3 that worked for them, so it seemed like a good place to start. While writing this blog post, I also found another guide to tuning Cura for the Ender 3 -- 6mm extruder retraction at 25mm/s, without z-hop, seems to be the magic setting to reduce stringing, resulting in a slight "nozzle wipe" over the part. (These days Creality ship a Microsoft Windows only "Creality Slicer" which I have not used; but as far as I can tell Creality used to ship with Cura, which is another reason to start with Cura for the Ender 3.)

Unfortunately the latest releases of Cura 4.x will not run on older Ubuntu versions, including Ubuntu 16.04 LTS and Ubuntu 18.04 LTS, because they rely on newer QT library functions. However with a bit of tweaking, I was able to get Cura running in an Ubuntu 19.04 ("Disco") Docker image, using Thomas's Ubuntu PPA packages, following the approach of Steve Baker, but updated to a later Ubuntu version and a later Cura version, and with some startup tweaks. (My cura-docker repository.)

For now I have simply used the built in Creality Ender 3 profile provided with Cura 4.1, for PLA filament, which prints at 200C with a 60C bed. It seems to work reasonably well for me, with the "Kiwi 3D PLA Filament". (I did notice that Cura wanted to "phone home" with usage information, which I dislike, but it is possible to disable the feature early in the application setup by clicking on "more information" when the panel advising it will "send anonymized data" and then choosing to disable that feature, and that setting appears to be persisted in the Cura preferences.

OpenSCAD

In order to have the whole 3D printing experience, I wanted to try designing my own 3D model and printing it. As with most beginning 3D printers, in the true RepRap fashion, the first one does is print something to improve the printer. Since the manual bed levelling is a common issue in Ender 3 printers, and the relatively extended, relatively thin springs are commonly identified as something to upgraded, I decided to tackle that issue first by creating a shim to ensure the springs were more compressed. This had the advantage of being a trivial 3D object to model -- a small washer -- which was a great "first project". (While writing this blog post I also found someone created an Ender 3 Bed Spring Stabliser, which combines the shim with an inner spring support to fill the gap between the inner bolt and the outer spring; which I might investigate further later.)

Since I am a programmer at heart, and I was designing a simple mechanical part, I decided to use OpenSCAD to model the part (conveniently it was packaged for Ubuntu 16.04 LTS, in the Ubuntu repository, so I could just install it; that is a somewhat old version but sufficient for my simple initial needs). OpenSCAD is a compiler for a parametric [modelling] language -- it translates .scad source files containing combinations of simple objects into (ASCII) STL files. Following the first couple of parts of a four part OpenSCAD tutorial got me started pretty quickly (note: there is no fifth part; by the time they finished the fourth part they decided a fifth part was not needed...).

After some experimentation I came up with this simple OpenSCAD model:

$fa=2;   // default facet angle
$fs=1;   // default facet size

// Parametric shim washer
//
// Measurements are diameter.  We divide them in half to get required radius.
//
module shim_washer(od, id, height, offset=0.5) {
    difference() {
        cylinder(r=(od/2), h=height);
        translate([0,0,-offset]) cylinder(r=(id/2),  h=height+(2*offset));
    }
}

shim_washer(od=13, id=6, height=2.75);

which I could load into OpenSCAD (using vim for editing, rather than the built in editing pane which can be closed), preview it, then render it (F6), validate it, and then export it as a STL file (File -> Export -> Export as STL...), and then load into Cura for "slicing" to convert it to GCode. (Conveniently, both OpenSCAD and Cura default to auto-reload of the model when their relevant files are written, which means having vim, OpenSCAD, and Cura all open, and saving changes as they are ready works quite well; the one issue I found was that while Cura will auto reload models, if you have duplicated a model for printing, it appears to only reload one of them :-( So I spent a while clearing the build bed and re-duplicating / re-laying out three of them for printing efficiently at once.)

It took a few attempts to get reasonable sized shims to put under the springs -- my first attempt confused diameter and radius resulting in washers much too wide, my second had the inner hole a little too small, my third had the shim a little too high (at 5mm), and the fourth worked. But each attempt only took about 20 minutes, even with beginner stumbling blocks, so it was fairly quick to iterate to a useful solution. I ended up printing only three shim washers, for the three corners without the bed power support bar -- and settled on the 2.75mm as being approximately the same height as that bed support bar. (If I were doing it again I might pick a 3mm height, and would probably also try a 5mm inner diameter; 4mm was definitely too tight, but 6mm is a little loose.)

Overall I was pleased to be able to go from conception to something that is now installed -- printed in black PLA, with 0.2mm layers and 70% infill for strength -- on my 3D printer. (I installed the shims at the bottom of the springs, immediately above the print bed support base, which should leave them more than far away from the bed heater that they do not get hot; and besides the bed will usually only be 60C, which is well under the PLA melting temperature.)

Next steps

Most likely my next step will be to install OctoPrint, probably via OctoPi on a Raspberry Pi 3B that I have sitting around, to enable "network printing". SneakerNetting images to print back and forth between my computer and the Ender 3 gets old quickly, and the Ender 3 "TF" ("TransFlash", aka MicroSD) slot is a bit fiddly to reach anyway, in addition to the need to put it into/take it out of a USB adapter.

Possibly then followed by 3D printing a few more of the "recommended upgrades" for the Ender 3, such as a filament guide (one of the main design flaws of the Ender 3 is that, as supplied, the filament is practically certain to drag along the Z axis lead screw in normal use, picking up oil -- and removing needed oil from that lead screw).

I might also upgrade the Marlin Firmware (on GitHub) on my Ender 3 with a more recent version, as I believe Creality are still shipping firmware based on the Marlin 1.0.1 firmware, but there is Marlin 1.1.x firmware including Ender 3 example configuration available now. The Creality Ender 3 mainboard apparently does not include an Arduino Bootloader (for space reasons), but it is possible to use another Arduino as an In-circuit Serial Programmer and I have a couple of suitable Arduino boards available. Being able to upgrade the firmware is one of the advantages of Open Source hardware :-)

Posted Sun Jul 21 19:05:07 2019 Tags:

Introduction

About 2.5 years ago I bought a Dell XPS 9360 laptop, so that I had a modern Linux laptop to take to conferences. At the time I was optimising for light weight, and relatively low cost, so went with an i7 system, with 8 GiB RAM, and 256 GiB M.2 storage system.

It came with Windows 10 Home (Pro cost extra :-) ), and I configured it to dual boot Windows 10 and Ubuntu Linux 16.04 LTS because I wanted to mostly run Ubuntu Linux, but still have the opportunity to run Windows 10 occasionally (since I have no other Windows systems).

Because I have mostly used the Ubuntu Linux 16.04 install for FPGA development, across a variety of vendors (Xilinx, Lattice) and FPGA models (Spartan6, Artix7, iCE40, iCE40 UP), all of which need different FPGA development tools -- and because most FPGA vendor tools are huge (many GiB for each Xilinx tool set) -- I ran out of storage space on my Linux partition pretty frequently.

While -- like most small thin laptops -- the Dell XPS 9360 CPU, RAM, etc is not upgradable, as they are directly soldered onto the motherboard, it turns out that the storage is a regular M.2 2280 NVMe drive, and can be swapped out for another M.2 NVMe drive. So I decided to replace the M.2 NVMe drive in my Dell XPS 9360 with a larger one to give the laptop a longer lease of life. (I would like to have more RAM, but in practice for what I do 8 GiB of RAM is sufficient even if it is not exactly ample RAM. So I can probably live with 8 GiB of RAM indefinitely. And the i7 CPU is still relatively fast for what I do with the laptop.)

After looking around for a while I settled on a 1TB Samsung 970 EVO Plus because:

  • It was available in stock from my local retailer, for a reasonable price

  • 1TB was a large increase in storage space, which would reduce the impact of solid state drive overwrite limits

  • It reviewed pretty well: AnandTech, TomsHardware, PCWorld, StorageReview, Guru 3D, etc.

  • While it is a TLC drive (3-bits per cell), it has both a TurboWrite feature (initial write to SLC (2-bits per cell) section) and a RAM cache to reduce the speed impact. This means it is well tuned to the sort of bursty write you get on a laptop (and not well tuned for sustained "enterprise" writes). Also the existing Toshiba drive supplied with the Dell XPS 9360 was also a TLC drive, and performed good enough for my laptop use.

  • People had reported putting other Samsung 9xx EVO drives into Dell XPS 93x0 models (eg, on Reddit, iFixIt, and the Dell forums, etc).

  • Samsung make their own storage chips, and have a pretty reasonable reputation (and so the 5 year warranty offered is likely to actually indicate the quality).

Of note, while the Samsung 970 EVO Plus is capable of up to about 3 GB/s transfer speeds via M.2, the Dell XPS 9360 runs the M.2 interface in a power saving mode, which limits the maximum transfer speed to about 1.8 GB/s. Which means that one could choose to buy a slower drive and still get the same performance -- but I chose to pay a little bit more now for hopefully a more reliably drive, that could potentially be moved to another system later.

Swapping the drive

Physically swapping the M.2 drive requires disassembling the laptop, but I found lots of guides to replacing the M.2 drive, so I was fairly confident that I could physically replace the M.2 drive itself. The major challenges was going to be getting the data from one drive to the other, because the Dell XPS 9360 has only one M.2 slot and M.2 external adapters are not common, so copying directly from the old drive to the new drive was not possible.

In terms of physically replacing the drive I suggest you look at some of the other guides. My only extra hints would be:

  • I used a Torx T5 bit to remove most of the screws, and a Philips #00 to remove the one under the bottom flap and on the M.2 drive itself.

  • There are a lot of plastic clips around all sides of the rear of the laptop, which need to be unclipped with a spudger or similar (I used a plastic spudger that came in a "phone repair" kit, which worked but was not ideal).

  • The clips at the front of the laptop and the sides are smaller, and should be unclipped first.

  • There are large clips across the rear in the hinge area, so the best option is to unclip all the others, and then lift the bottom forwards (away from the hinge area towards the front of the laptop) to unclip those clips (and remember to reinstall the base in the reverse manner: large clips by the hinge first, then around the sides).

  • Make certain you have everything copied off the old drive before opening the laptop to swap out the M.2 drive, as you will want to avoid having to open the laptop more than once.

Other than getting the bottom of the case off (possible, but fiddly and time consuming), swapping the M.2 drive physically is fairly easy, and anyone used to working with PC internals would be able to swap the drive. (Lots of extra care is required in opening the case, though, due to all the plastic clips; fortunately there is no glue.)

Transferring data

Because the hardware limitations prevented a direct copy between the drives my approach was:

  • Make a backup of everything on the laptop (copying everything onto my NAS, both the Windows 10 and Ubuntu Linux partitions, and all the other partitions)

  • Boot into Windows 10 and create both a Recovery Boot USB stick (small, about 1GiB), and a Recovery Install USB stick (about 16GiB) just in case. (I needed the Recovery Boot USB stick to get Windows 10 booting again, so do not skip that one; fortunately I did not need to use the Recovery Install USB stick again.)

  • Boot off a Ubuntu 18.04 Live USB stick, and then use dd to copy each individual partition into its own file on an external hard drive. (To boot from the USB stick, the easiest option is to plug in the USB stick, and then press F12 repeatedly when the Dell logo is displayed after power on until it says "Preparing One Time Boot Menu"; note that on the Dell XPS 9360 the Fn key should not be pressed, as unlike a Mac laptop, those keys are function keys by default, and Fn is needed for the other features.)

  • Use md5sum to verify that the original drive partition contents and the dd copies were bit for bit identical.

  • Make multiple printouts of the partition table of the old drive (in various units), using parted list, so I could recreate it again on the new drive (by hand).

Then once I was happy that I had a fully copy of the old drive, I powered off the laptop, opened it up (as above), and installed the new Samsung 970 EVO Plus drive.

Once the laptop was back together I:

  • Plugged the Ubuntu 18.04 Live USB stick back in again, and used F12 to bring up the One Time Boot Menu, and booted back into the Live CD.

  • Manually partitioned the new M.2 drive with a gpt partition table, with the partitions the same size as the previous drive, and with the same flags, etc.

  • Used dd to copy the partition contents off the external drive onto the new Samsung drive.

  • Used md5sum to verify that those copies on the Samsung drive partitions were bit for bit identical to what had been copied off the old Toshiba drives.

And then I rebooted, and it failed to boot at all :-(

As best I can tell, even though I maintained the partition contents identically (and the first time, the partition locations and flags identically), the UEFI booting in both Ubuntu Linux 16.04 LTS and in Windows 10 (neither would boot), was relying at least in part on something else -- the Partition UUID which is part of the gpt format, perhaps -- which changed, and that threw off the booting process. (Plus changing the drive clearly caused the UEFI BIOS to forget the boot order sequence it previously had.)

Getting Ubuntu Linux 16.04 LTS to boot again

To fix the booting of Ubuntu Linux 16.04 LTS (via grub) I:

  • Booted off the Ubuntu Linux 18.04 live CD again

  • Mounted the Ubuntu Linux 16.04 root file system (which is in a LVM volume group, inside a LUKS encrypted container) with:

    sudo cryptsetup luksOpen /dev/nvme0n1p4 dell
    sudo pvscan
    sudo mkdir /install
    sudo mount /dev/vg/root /install
    

    which requires the password for the LUKS volume at the luksOpen stage.

  • Mounted /proc, /sys, dev, etc inside that:

    sudo mount -t sysfs sys /install/sys
    sudo mount -t proc proc /install/proc
    sudo mount -t devtmpfs udev /install/dev
    sudo mount -t devpts devpts /install/dev/pts
    
  • Changed into the chroot, and used that to mount the remaining volumes:

    sudo chroot /install
    mount -a
    

    and checked that /boot and /boot/efi had mounted:

    df -Pm | grep /boot
    
  • Then updated/reinstalled grub:

    update-grub
    grub-install
    

    from inside the chroot.

  • Then exited the chroot, and unmounted everything:

    exit
    sudo umount /install/dev/pts
    sudo umount /install/dev
    sudo umount /install/proc
    sudo umount /install/sys
    sudo umount /install/boot/efi
    sudo umount /install/boot
    sudo umount /install
    

And then I rebooted again. This time, to my relief, the laptop booted into grub normally, and then booted into Ubuntu Linux 16.04 LTS normally.

Unfortunately it still could not boot Windows 10 :-( It just kept coming up with the "Recovery" screen ("Your PC Device needs to be repaired"). Even after running sudo update-grub again from within the working Ubuntu Linux 16.04 LTS environment, to ensure that Windows 10 was found in the boot environment. Since I knew the file system contents was bit for bit identical, I figured the boot process had become confused (possibly due to the new partition UUIDs).

Getting Windows 10 to boot again

I first tried booting off the Windows 10 Boot Recovery (1GiB) USB stick I had made (using F12 to get a One Time Boot Menu to boot the USB stick), then navigating into Advanced Options / Startup Repair / Windows 10 but that just reported "Startup Repair couldn't repair your PC". So clearly Windows 10 was very confused.

Next I found a Dell guide to Repairing the Windows EFI bootloader which I tried, by going into Advanced Options / Command Prompt from the Windows 10 Boot Recovery USB stick. Unfortunately I got stuck at step 7 of that guide, because ESP (EFI boot partition) was hidden for some reason, which meant those instructions would not allow me to assign a drive letter to reinstall the Windows 10 boot config. (My guess is the same issue caused the "Startup Repair" to fail.) I do not know why it was shown up as Hidden, without a drive letter, as it did not have the hidden flag in the gpt partition table (and I could be certain it was the ESP partition by the exact size).

Fortunately I found another way to assign a drive letter to the ESP partition:

diskpart
list disk
sel disk 0
list partition
select partition 1
assign

which let me move on.

Unfortunately, the next step bootrec /FixBoot then failed with "Access is Denied". Some guides recommend reformatting the ESP partition at this point but I was reluctant to do that because there were both Ubuntu Linux 16.04 UEFI boot files on there and Dell UEFI boot files on there (eg, for recovery tools), so I kept looking.

Another guide suggested, bootrec /REBUILDBCD which I tried next, but after scanning the system that then reported "The system cannot find the path specified." :-(

With some more hunting online, I found someone who had encountered and fixed that issue, by doing:

cd /d H:\EFI\Microsoft\Boot
ren BCD BCD.bak
bcdboot c:\Windows /l en-nz /s h: /f ALL

where c:\Windows is the Windows 10 directory on the drive letter found as the main Windows 10 Drive, en-nz is the preferred local (en-us seems likely to be the default), /s h: specifies the drive letter assigned to the EFI partition, and /f ALL specifies that the UEFI, BIOS, and NVRAM boot settings should all be updated. For good measure I also tried:

bootrec /fixboot

again, but that still failed ("Access is Denied").

However after exiting out the recovery shell and rebooting the laptop, it actually automatically booted into the Windows 10 environment using the Windows boot manager. At this point it only booted Windows 10, but I was able to get into the boot manager (eg, F12), ask it to boot Ubuntu Linux 16.04 LTS, and then do:

sudo update-grub
sudo grub-install

inside a Ubuntu Linux 16.04 LTS terminal window, and then the laptop was booting normally, with grub able to boot both Ubuntu Linux 16.04 LTS and Windows 10 as it did on the old drive. Phew.

Expanding the Linux root partition

Once everything was copied onto the new Samsung 970 EVO Plus 1TB drive, and booting successfully, that just left the original purpose: expanding the Linux file system. (Because I hardly use Windows 10 on the laptop, and had not run into space problems on that partition -- about 90 GiB -- I chose to dedicate all the extra space on the new drive to Linux.)

My original plan was just to create an additional partition on the end of the drive (with the remaining 700 GiB of space), and then use LVM to join the two partitions together, given that the root file system was already on LVM -- and that was how I laid out the drive when I first copied the data over. However I realised that with the Linux filesystem inside a LUKS encrypted partition, that would both be more fragile (two LUKS containers, or some data in a LUKS container and the rest outside it), and potentially require entering two passwords on boot (to unlock each partitions).

So I spent a while shuffling partitions around so that all the ESP / Windows / Dell partitions were at the start of the drive, followed by the Ubuntu /boot partition (which needs to be outside the LUKS encryption unless you do special EFI boot tricks), followed by the main LUKS / LVM partition at the end of the drive. (It was fiddly to shuffle things around, but fortunately when you have a drive that is four times as big as the original content it is easy to make more partitions to temporarily hold copies of the data you want to move: so it was just more use of dd and md5sum to make sure everything was copied correctly, into the right places. I even had to delete some of the partitions, quit parted, and then recreate the partitions at the identical spot and size to get them to show up in the right partition order that I wanted.)

Once all the partitions were in the right order, and the right size, and the laptop was booting Ubuntu Linux 16.04 LTS and Windows 10 correctly, I was ready to carry on with expanding the Linux root drive. My Linux root drive is:

  • In an ext4 file system (Ubuntu 16.04 LTS default)

  • On a LVM logical volume (LV)

  • Inside a LVM volume group (VG)

  • Inside a LVM physical volume (PV)

  • Inside a LUKS encrypted container

  • Inside a partition on /dev/nvme0n1 (/dev/nvme0n1p7 by the end of all the partition shuffling).

Which means in order to expand the root file system, all of those layers need to be expanded, in the opposite order. That's a lot of layers to potentially go wrong.

Since I still had a couple of recent backups of the laptop drive, as well as the original M.2 drive, which I had recently tested, and I knew how to recover from booting issues, I felt it was worth giving these steps ago. (Seriously have known tested good backups before trying to do this: there is a lot to go wrong, and typos or interrupted operations could wreak havoc.)

(Several of these instructions suggest creating a temporary partition after the one you are expanding, and writing random data to that partition before expanding the LUKS volume into it: I did not do that because (a) it takes a bunch of time to do, (b) it forces the SSD to assume the entire disk is in use, and copy more data around thus using up SSD drive life due to write amplification, (c) the encrypted partition is already large enough for my level of paranoia about this particular volume being recovered, and (d) the data on this laptop is not that sensitive -- it's mostly just a FPGA development laptop at this point, and almost all of that development is open source on GitHub anyway, so I do not feel the need to do that much to protect it: it is just encrypted because that is what I do with all my computers that I might travel with, to make data recovery by someone else non-trivial.)

Expanding the partition

To expand the partition I booted off the Ubuntu 18.04 Live USB install, and then used parted to do:

resizepart 7 953869MiB

where 953869MiB was 1MiB lower than the maximum size of the drive displayed by parted at the top of the partition table list result (the exact size does not work, I think due to rounding and/or the partition elements starting at 0).

Then I rebooted back into Ubuntu 18.04 LTS Live CD to expand the LUKS container, while it was open but not mounted.

Expanding the LUKS container

Finding some guides to enlarging a LVM on LUKS install was what convinced me that rearranging the disk partitions to have a single LUKS container was the best option. (Previously I had assumed expanding the LUKS container was not possible, even though I knew all the other steps were possible.)

Once the partition is expanded, and you have rebooted back into the Ubuntu Linux Live environment, to ensure that Linux consistently sees the new partition as expanded but not mounted, then open the LUKS container and expand it to the new size of the partition (note: new partition number as I rearranged the partitions, above, to have the LUKS / LVM partition at the end):

sudo cryptsetup luksOpen /dev/nvme0n1p7 dell
ls -l /dev/mapper/dell
sudo vgscan
sudo vgchange -ay
sudo cryptsetup resize dell

That command completes pretty much immediately, and the default new size is "the size of the disk partition" which is exactly what we want here.

Verify the new size of the LUKS container with:

sudo cryptsetup -v status dell

which reports the size in "512 byte sectors" (one of the most useless units for modern drives :-( ); but fortunately dividing by 2048 (2 * 1024) gives us MiB, and we can verify the new size is very close to the partition size (in my case 2 MiB smaller; it also revealed the LUKS container was injecting a 4096 sector offset, which is exactly 2 MiB: 4096 * 512 = 2097152 = 2 * 1024 * 1024; I am not sure if that is a requirement, a default, or something I chose when I first set it up).

Expanding the LVM physical volume (PV)

Expanding the LVM physical volume is just a matter of asking LVM to recognise the additional space:

sudo pvresize /dev/mapper/dell

and it should return almost immediately reporting "1 physical volume(s) resized / 0 physical volume(s) not resized)". We can verify the new physical volume size with:

sudo pvdisplay /dev/mapper/dell

and that should show a "PV Size" around 832 GiB, as well as lots of "Free PE" now the physical volume is much larger.

Expanding the LVM volume group (VG)

The volume group is automatically expanded when it has physical volumes with spare space in them, which we can verify with:

sudo vgdisplay

That should also show a "VG Size" around 832 GiB, as well as lots of "Free PE / Size".

Expanding the LVM logical volume (LV)

My existing install had two logical volumes, created during the original Ubuntu Linux 16.04 LTS:

  • /dev/vg/root

  • /dev/vg/swap

and unfortunately they were in that order on the disk, as shown by:

sudo lvdisplay

Since I preferred to have my root LV contiguous, I chose to remove the swap logical volume, then expand the root logical volume, then create a new swap logical volume and initialise that again. (Because we are booted into an Ubuntu Live USB environment, none of these are mounted, and the swap usage is obviously transitory anyway, so the contents did not need to be retained.)

To do this I did:

sudo lvchange -an /dev/vg/swap
sudo lvremove /dev/vg/swap

which prompts for confirmation of removing the swap logical volume.

Then I expanded the root logical volume to 512 GiB (chosen to not completely use up the extra disk space to allow more flexibility, but to be about 4 times as big as the existing Linux root filesystem):

sudo lvresize -L 512G /dev/vg/root

and verified that worked as expected with:

sudo lvdisplay /dev/vg/root

which should show a "LV Size" of 512.00 GiB as a result.

Then I made a new swap logical volume, of 2 GiB again:

sudo lvcreate -n swap -L 2G /dev/vg
sudo lvdisplay /dev/vg/swap

and reinitialised the swap space:

sudo mkswp -L swap /dev/vg/swap

(Note that this process changes the UUID of the swap partition, which might need to be fixed up, if you are mounting the swap by UUID rather than LV path or volume label.)

Resizing the ext4 root file system

Now that everything below is resized, we can resize the ext4 filesystem. With ext4 this could actually be done online (while mounted), but because I was still booted into the Ubuntu Linux live environment, I did the resize offline, starting by checking the file system:

sudo e2fsck -f /dev/vg/root
sudo resize2fs -p /dev/vg/root

where the -p is for progress messages, but in practice the resize only took a few seconds on the Samsung 970 EVO Plus drive (as it only moves metadata around). The new file system size is reported in 4KiB blocks (another not very useful unit :-( ), as 134217728 4KiB blocks, which we can check is correct with 134217728 / 4 * 1024 * 1024 = 512 GiB.

After that I did another e2fsck -f /dev/vg/root check out of abundance of precaution, and then mounted the partition to check the expected contents were there (and verify the way the swap partition was mounted to reduce boot issues):

sudo mkdir /install
sudo mount /dev/vg/root /install
grep swap /install/etc/install

Fortunately the swap was being mounted by LV path (/dev/mapper/vg-swap) so it should survive being recreated elsewhere on the disk.

While it was mounted, I also checked the /dev/vg/root filesystem usage, to make sure I now had lots of free space:

sudo df -Pm /install

and that showed I had gone from about 98% used on the root partition to about 27% used. So I unmounted the file system again:

sudo umount /install

Running:

sudo vgdisplay

showed I had a bit over 300GiB of unallocated space in the volume group saved for later. (And if I do want to expand the root logical volume I would probably remove the swap again, and then recreate it, to keep the root logical volume in one LVM extent for simplicity.)

Conclusion

Once all the expansion steps were done from the Ubuntu Linux live environment, I simply rebooted, and Ubuntu 16.04 LTS and Windows 10 booted fine -- and I had lots more space in my Linux environment.

With a couple of days of effort, and a few hundred dollars for a new M.2 drive, I have managed to change my Dell XPS 9360 laptop from a persistently almost full root file system, to one which is about 27% full (and has about 300 GiB still available to allocate later). That makes it much more useful, potentially for several more years.

About a day of that time was consumed by:

  • Making backups

  • Copying the file system partitions around (especially to/from a USB 3 spinning disk)

  • Checking those backups, and copies

  • Waiting for Windows 10 to create USB Recovery drives (the 16GiB system install recovery drive took over 2.5 hours to write!)

and the remainder was research, getting Ubuntu Linux 16.04 LTS and Windows 10 booting again, etc. (Actually physically swapping the M.2 drive inside the Dell XPS 9360 took maybe half an hour including all the disassembly and reassembly.) I expect if I did it again the process would be faster, as I could have avoided some of the file system rearrangement I did by going directly to the final partition layout, knowing that I was going to have to make everything bootable again anyway.

Of note:

  • One potential advantage of not using the whole SSD, is that writes to the drive will be constrained to about the first 60% of the drive, which should reduce the amount of data that the SSD firmware feels it needs to shuffle around (particularly important because by default LUKS does not pass on TRIM/DISCARD requests for security reasons, so any block written to will then be copied around by the SSD firmware forever).

  • It turned out almost impossible to find the SSD erase block size, and align anything to those erase blocks (cf XFS Storage Layout Considerations). As best I could tell erase block sizes seem to be trade secrets now, certainly not appearing in data sheets and in some cases not even available by requests; and everything seems to be defaulting to aligning to 1 MiB blocks as being sufficient. Aligning to 1 MiB seems likely to be reasonable (especially after 5+ years of OSes doing that automatically, and vendors designing for those OS), but possibly not the most optimal choice in theory. So for single drive systems it probably makes sense just to let everything auto-align to 1 MiB boundaries, and ignore the problem.

Even modern local (internal) storage is basically a "network attached storage server" of its own, with its own ideas about how to store data, and its own storage API. It just happens to speak SATA or SAS or NVMe or similar, rather than Ethernet and TCP/IP.

ETA 2019-04-30: After doing all of this, when I next booted into Windows 10 for an extended period, I found that Windows Update was going to install (no option) the update:

Dell, Inc - Firmware - 9/27/2018 12:00:00 AM - 2.10.0
Status: Pending Install

(with the lovely message "We'll automatically install updates when you aren't using your device, or you can install them now if you want.").

It was not clear to me what it is. By searching on the Microsoft Catalog for 2.10.0 I could find three versions, with the matching date (and three 2.10.0 versions with a later update date of 3/25/2019); I think the three versions are for different versions of Windows 10, and my guess is this version is the one that would be installed on my Dell XPS 9360, since I think I have already updated to the latest Windows 10 release. Unfortunately there were no other details for what it is. And it was unclear if the install is being prompted by replacing the drive, or just by the date.

Since Windows 10 was not giving me an option, and I hate having things break randomly in the background, I chose to plug the laptop into the mains power, and let it "Install Now". Naturally it wanted to restart, and when it restarted it then proceeded to update the BIOS and firmware of everything in the laptop :-(

I am unclear whether this user hostile behaviour of forcing compulsory unscheduled firmware updates is Dell behaviour or new Microsoft behaviour (or both), why it was forced to a September 2018 version, and whether it was triggered by replacing the Toshiba M.2 drive that came with the laptop with the Samsung 970 EVO Plus drive -- or just triggered by the date / first sufficiently long Windows boot that something decided it had time to treat my laptop as its own.

Hopefully this unplanned firmware update does not adversely impact is on the Dell XPS 9360 that I had just spent a couple of days upgrading the drive :-( Fortunately Windows 10 and Ubuntu Linux 16.04 LTS do seem to boot up again.

(After rebooting into Windows again, Dell Update -- not Windows Update -- decided it had a further 12 updates it wanted to install, including a BIOS released 2019-04-22; but fortunately I could choose "Remind Later" to those, so that is what I did.)

Posted Tue Apr 30 14:25:13 2019 Tags: