Replacing my Dell XPS 9360 drive

Introduction

About 2.5 years ago I bought a Dell XPS 9360 laptop, so that I had a modern Linux laptop to take to conferences. At the time I was optimising for light weight, and relatively low cost, so went with an i7 system, with 8 GiB RAM, and 256 GiB M.2 storage system.

It came with Windows 10 Home (Pro cost extra :-) ), and I configured it to dual boot Windows 10 and Ubuntu Linux 16.04 LTS because I wanted to mostly run Ubuntu Linux, but still have the opportunity to run Windows 10 occasionally (since I have no other Windows systems).

Because I have mostly used the Ubuntu Linux 16.04 install for FPGA development, across a variety of vendors (Xilinx, Lattice) and FPGA models (Spartan6, Artix7, iCE40, iCE40 UP), all of which need different FPGA development tools -- and because most FPGA vendor tools are huge (many GiB for each Xilinx tool set) -- I ran out of storage space on my Linux partition pretty frequently.

While -- like most small thin laptops -- the Dell XPS 9360 CPU, RAM, etc is not upgradable, as they are directly soldered onto the motherboard, it turns out that the storage is a regular M.2 2280 NVMe drive, and can be swapped out for another M.2 NVMe drive. So I decided to replace the M.2 NVMe drive in my Dell XPS 9360 with a larger one to give the laptop a longer lease of life. (I would like to have more RAM, but in practice for what I do 8 GiB of RAM is sufficient even if it is not exactly ample RAM. So I can probably live with 8 GiB of RAM indefinitely. And the i7 CPU is still relatively fast for what I do with the laptop.)

After looking around for a while I settled on a 1TB Samsung 970 EVO Plus because:

It was available in stock from my local retailer, for a reasonable price
1TB was a large increase in storage space, which would reduce the impact of solid state drive overwrite limits
It reviewed pretty well: AnandTech, TomsHardware, PCWorld, StorageReview, Guru 3D, etc.
While it is a TLC drive (3-bits per cell), it has both a TurboWrite feature (initial write to SLC (2-bits per cell) section) and a RAM cache to reduce the speed impact. This means it is well tuned to the sort of bursty write you get on a laptop (and not well tuned for sustained "enterprise" writes). Also the existing Toshiba drive supplied with the Dell XPS 9360 was also a TLC drive, and performed good enough for my laptop use.
People had reported putting other Samsung 9xx EVO drives into Dell XPS 93x0 models (eg, on Reddit, iFixIt, and the Dell forums, etc).
Samsung make their own storage chips, and have a pretty reasonable reputation (and so the 5 year warranty offered is likely to actually indicate the quality).

Of note, while the Samsung 970 EVO Plus is capable of up to about 3 GB/s transfer speeds via M.2, the Dell XPS 9360 runs the M.2 interface in a power saving mode, which limits the maximum transfer speed to about 1.8 GB/s. Which means that one could choose to buy a slower drive and still get the same performance -- but I chose to pay a little bit more now for hopefully a more reliably drive, that could potentially be moved to another system later.

Swapping the drive

Physically swapping the M.2 drive requires disassembling the laptop, but I found lots of guides to replacing the M.2 drive, so I was fairly confident that I could physically replace the M.2 drive itself. The major challenges was going to be getting the data from one drive to the other, because the Dell XPS 9360 has only one M.2 slot and M.2 external adapters are not common, so copying directly from the old drive to the new drive was not possible.

In terms of physically replacing the drive I suggest you look at some of the other guides. My only extra hints would be:

I used a Torx T5 bit to remove most of the screws, and a Philips #00 to remove the one under the bottom flap and on the M.2 drive itself.
There are a lot of plastic clips around all sides of the rear of the laptop, which need to be unclipped with a spudger or similar (I used a plastic spudger that came in a "phone repair" kit, which worked but was not ideal).
The clips at the front of the laptop and the sides are smaller, and should be unclipped first.
There are large clips across the rear in the hinge area, so the best option is to unclip all the others, and then lift the bottom forwards (away from the hinge area towards the front of the laptop) to unclip those clips (and remember to reinstall the base in the reverse manner: large clips by the hinge first, then around the sides).
Make certain you have everything copied off the old drive before opening the laptop to swap out the M.2 drive, as you will want to avoid having to open the laptop more than once.

Other than getting the bottom of the case off (possible, but fiddly and time consuming), swapping the M.2 drive physically is fairly easy, and anyone used to working with PC internals would be able to swap the drive. (Lots of extra care is required in opening the case, though, due to all the plastic clips; fortunately there is no glue.)

Transferring data

Because the hardware limitations prevented a direct copy between the drives my approach was:

Make a backup of everything on the laptop (copying everything onto my NAS, both the Windows 10 and Ubuntu Linux partitions, and all the other partitions)
Boot into Windows 10 and create both a Recovery Boot USB stick (small, about 1GiB), and a Recovery Install USB stick (about 16GiB) just in case. (I needed the Recovery Boot USB stick to get Windows 10 booting again, so do not skip that one; fortunately I did not need to use the Recovery Install USB stick again.)
Boot off a Ubuntu 18.04 Live USB stick, and then use dd to copy each individual partition into its own file on an external hard drive. (To boot from the USB stick, the easiest option is to plug in the USB stick, and then press F12 repeatedly when the Dell logo is displayed after power on until it says "Preparing One Time Boot Menu"; note that on the Dell XPS 9360 the Fn key should not be pressed, as unlike a Mac laptop, those keys are function keys by default, and Fn is needed for the other features.)
Use md5sum to verify that the original drive partition contents and the dd copies were bit for bit identical.
Make multiple printouts of the partition table of the old drive (in various units), using parted list, so I could recreate it again on the new drive (by hand).

Then once I was happy that I had a fully copy of the old drive, I powered off the laptop, opened it up (as above), and installed the new Samsung 970 EVO Plus drive.

Once the laptop was back together I:

Plugged the Ubuntu 18.04 Live USB stick back in again, and used F12 to bring up the One Time Boot Menu, and booted back into the Live CD.
Manually partitioned the new M.2 drive with a gpt partition table, with the partitions the same size as the previous drive, and with the same flags, etc.
Used dd to copy the partition contents off the external drive onto the new Samsung drive.
Used md5sum to verify that those copies on the Samsung drive partitions were bit for bit identical to what had been copied off the old Toshiba drives.

And then I rebooted, and it failed to boot at all :-(

As best I can tell, even though I maintained the partition contents identically (and the first time, the partition locations and flags identically), the UEFI booting in both Ubuntu Linux 16.04 LTS and in Windows 10 (neither would boot), was relying at least in part on something else -- the Partition UUID which is part of the gpt format, perhaps -- which changed, and that threw off the booting process. (Plus changing the drive clearly caused the UEFI BIOS to forget the boot order sequence it previously had.)

Getting Ubuntu Linux 16.04 LTS to boot again

To fix the booting of Ubuntu Linux 16.04 LTS (via grub) I:

Booted off the Ubuntu Linux 18.04 live CD again
Mounted the Ubuntu Linux 16.04 root file system (which is in a LVM volume group, inside a LUKS encrypted container) with:
```
sudo cryptsetup luksOpen /dev/nvme0n1p4 dell
sudo pvscan
sudo mkdir /install
sudo mount /dev/vg/root /install
```
which requires the password for the LUKS volume at the luksOpen stage.

Mounted /proc, /sys, dev, etc inside that:

sudo mount -t sysfs sys /install/sys
sudo mount -t proc proc /install/proc
sudo mount -t devtmpfs udev /install/dev
sudo mount -t devpts devpts /install/dev/pts

Changed into the chroot, and used that to mount the remaining volumes:
```
sudo chroot /install
mount -a
```
and checked that /boot and /boot/efi had mounted:
```
df -Pm | grep /boot
```
Then updated/reinstalled grub:
```
update-grub
grub-install
```
from inside the chroot.

Then exited the chroot, and unmounted everything:

exit
sudo umount /install/dev/pts
sudo umount /install/dev
sudo umount /install/proc
sudo umount /install/sys
sudo umount /install/boot/efi
sudo umount /install/boot
sudo umount /install

And then I rebooted again. This time, to my relief, the laptop booted into grub normally, and then booted into Ubuntu Linux 16.04 LTS normally.

Unfortunately it still could not boot Windows 10 :-( It just kept coming up with the "Recovery" screen ("Your PC Device needs to be repaired"). Even after running sudo update-grub again from within the working Ubuntu Linux 16.04 LTS environment, to ensure that Windows 10 was found in the boot environment. Since I knew the file system contents was bit for bit identical, I figured the boot process had become confused (possibly due to the new partition UUIDs).

Getting Windows 10 to boot again

I first tried booting off the Windows 10 Boot Recovery (1GiB) USB stick I had made (using F12 to get a One Time Boot Menu to boot the USB stick), then navigating into Advanced Options / Startup Repair / Windows 10 but that just reported "Startup Repair couldn't repair your PC". So clearly Windows 10 was very confused.

Next I found a Dell guide to Repairing the Windows EFI bootloader which I tried, by going into Advanced Options / Command Prompt from the Windows 10 Boot Recovery USB stick. Unfortunately I got stuck at step 7 of that guide, because ESP (EFI boot partition) was hidden for some reason, which meant those instructions would not allow me to assign a drive letter to reinstall the Windows 10 boot config. (My guess is the same issue caused the "Startup Repair" to fail.) I do not know why it was shown up as Hidden, without a drive letter, as it did not have the hidden flag in the gpt partition table (and I could be certain it was the ESP partition by the exact size).

Fortunately I found another way to assign a drive letter to the ESP partition:

diskpart
list disk
sel disk 0
list partition
select partition 1
assign

which let me move on.

Unfortunately, the next step bootrec /FixBoot then failed with "Access is Denied". Some guides recommend reformatting the ESP partition at this point but I was reluctant to do that because there were both Ubuntu Linux 16.04 UEFI boot files on there and Dell UEFI boot files on there (eg, for recovery tools), so I kept looking.

Another guide suggested, bootrec /REBUILDBCD which I tried next, but after scanning the system that then reported "The system cannot find the path specified." :-(

With some more hunting online, I found someone who had encountered and fixed that issue, by doing:

cd /d H:\EFI\Microsoft\Boot
ren BCD BCD.bak
bcdboot c:\Windows /l en-nz /s h: /f ALL

where c:\Windows is the Windows 10 directory on the drive letter found as the main Windows 10 Drive, en-nz is the preferred local (en-us seems likely to be the default), /s h: specifies the drive letter assigned to the EFI partition, and /f ALL specifies that the UEFI, BIOS, and NVRAM boot settings should all be updated. For good measure I also tried:

bootrec /fixboot

again, but that still failed ("Access is Denied").

However after exiting out the recovery shell and rebooting the laptop, it actually automatically booted into the Windows 10 environment using the Windows boot manager. At this point it only booted Windows 10, but I was able to get into the boot manager (eg, F12), ask it to boot Ubuntu Linux 16.04 LTS, and then do:

sudo update-grub
sudo grub-install

inside a Ubuntu Linux 16.04 LTS terminal window, and then the laptop was booting normally, with grub able to boot both Ubuntu Linux 16.04 LTS and Windows 10 as it did on the old drive. Phew.

Expanding the Linux root partition

Once everything was copied onto the new Samsung 970 EVO Plus 1TB drive, and booting successfully, that just left the original purpose: expanding the Linux file system. (Because I hardly use Windows 10 on the laptop, and had not run into space problems on that partition -- about 90 GiB -- I chose to dedicate all the extra space on the new drive to Linux.)

My original plan was just to create an additional partition on the end of the drive (with the remaining 700 GiB of space), and then use LVM to join the two partitions together, given that the root file system was already on LVM -- and that was how I laid out the drive when I first copied the data over. However I realised that with the Linux filesystem inside a LUKS encrypted partition, that would both be more fragile (two LUKS containers, or some data in a LUKS container and the rest outside it), and potentially require entering two passwords on boot (to unlock each partitions).

So I spent a while shuffling partitions around so that all the ESP / Windows / Dell partitions were at the start of the drive, followed by the Ubuntu /boot partition (which needs to be outside the LUKS encryption unless you do special EFI boot tricks), followed by the main LUKS / LVM partition at the end of the drive. (It was fiddly to shuffle things around, but fortunately when you have a drive that is four times as big as the original content it is easy to make more partitions to temporarily hold copies of the data you want to move: so it was just more use of dd and md5sum to make sure everything was copied correctly, into the right places. I even had to delete some of the partitions, quit parted, and then recreate the partitions at the identical spot and size to get them to show up in the right partition order that I wanted.)

Once all the partitions were in the right order, and the right size, and the laptop was booting Ubuntu Linux 16.04 LTS and Windows 10 correctly, I was ready to carry on with expanding the Linux root drive. My Linux root drive is:

In an ext4 file system (Ubuntu 16.04 LTS default)
On a LVM logical volume (LV)
Inside a LVM volume group (VG)
Inside a LVM physical volume (PV)
Inside a LUKS encrypted container
Inside a partition on /dev/nvme0n1 (/dev/nvme0n1p7 by the end of all the partition shuffling).

Which means in order to expand the root file system, all of those layers need to be expanded, in the opposite order. That's a lot of layers to potentially go wrong.

Since I still had a couple of recent backups of the laptop drive, as well as the original M.2 drive, which I had recently tested, and I knew how to recover from booting issues, I felt it was worth giving these steps ago. (Seriously have known tested good backups before trying to do this: there is a lot to go wrong, and typos or interrupted operations could wreak havoc.)

(Several of these instructions suggest creating a temporary partition after the one you are expanding, and writing random data to that partition before expanding the LUKS volume into it: I did not do that because (a) it takes a bunch of time to do, (b) it forces the SSD to assume the entire disk is in use, and copy more data around thus using up SSD drive life due to write amplification, (c) the encrypted partition is already large enough for my level of paranoia about this particular volume being recovered, and (d) the data on this laptop is not that sensitive -- it's mostly just a FPGA development laptop at this point, and almost all of that development is open source on GitHub anyway, so I do not feel the need to do that much to protect it: it is just encrypted because that is what I do with all my computers that I might travel with, to make data recovery by someone else non-trivial.)

Expanding the partition

To expand the partition I booted off the Ubuntu 18.04 Live USB install, and then used parted to do:

resizepart 7 953869MiB

where 953869MiB was 1MiB lower than the maximum size of the drive displayed by parted at the top of the partition table list result (the exact size does not work, I think due to rounding and/or the partition elements starting at 0).

Then I rebooted back into Ubuntu 18.04 LTS Live CD to expand the LUKS container, while it was open but not mounted.

Expanding the LUKS container

Finding some guides to enlarging a LVM on LUKS install was what convinced me that rearranging the disk partitions to have a single LUKS container was the best option. (Previously I had assumed expanding the LUKS container was not possible, even though I knew all the other steps were possible.)

Once the partition is expanded, and you have rebooted back into the Ubuntu Linux Live environment, to ensure that Linux consistently sees the new partition as expanded but not mounted, then open the LUKS container and expand it to the new size of the partition (note: new partition number as I rearranged the partitions, above, to have the LUKS / LVM partition at the end):

sudo cryptsetup luksOpen /dev/nvme0n1p7 dell
ls -l /dev/mapper/dell
sudo vgscan
sudo vgchange -ay
sudo cryptsetup resize dell

That command completes pretty much immediately, and the default new size is "the size of the disk partition" which is exactly what we want here.

Verify the new size of the LUKS container with:

sudo cryptsetup -v status dell

which reports the size in "512 byte sectors" (one of the most useless units for modern drives :-( ); but fortunately dividing by 2048 (2 * 1024) gives us MiB, and we can verify the new size is very close to the partition size (in my case 2 MiB smaller; it also revealed the LUKS container was injecting a 4096 sector offset, which is exactly 2 MiB: 4096 * 512 = 2097152 = 2 * 1024 * 1024; I am not sure if that is a requirement, a default, or something I chose when I first set it up).

Expanding the LVM physical volume (PV)

Expanding the LVM physical volume is just a matter of asking LVM to recognise the additional space:

sudo pvresize /dev/mapper/dell

and it should return almost immediately reporting "1 physical volume(s) resized / 0 physical volume(s) not resized)". We can verify the new physical volume size with:

sudo pvdisplay /dev/mapper/dell

and that should show a "PV Size" around 832 GiB, as well as lots of "Free PE" now the physical volume is much larger.

Expanding the LVM volume group (VG)

The volume group is automatically expanded when it has physical volumes with spare space in them, which we can verify with:

sudo vgdisplay

That should also show a "VG Size" around 832 GiB, as well as lots of "Free PE / Size".

Expanding the LVM logical volume (LV)

My existing install had two logical volumes, created during the original Ubuntu Linux 16.04 LTS:

/dev/vg/root
/dev/vg/swap

and unfortunately they were in that order on the disk, as shown by:

sudo lvdisplay

Since I preferred to have my root LV contiguous, I chose to remove the swap logical volume, then expand the root logical volume, then create a new swap logical volume and initialise that again. (Because we are booted into an Ubuntu Live USB environment, none of these are mounted, and the swap usage is obviously transitory anyway, so the contents did not need to be retained.)

To do this I did:

sudo lvchange -an /dev/vg/swap
sudo lvremove /dev/vg/swap

which prompts for confirmation of removing the swap logical volume.

Then I expanded the root logical volume to 512 GiB (chosen to not completely use up the extra disk space to allow more flexibility, but to be about 4 times as big as the existing Linux root filesystem):

sudo lvresize -L 512G /dev/vg/root

and verified that worked as expected with:

sudo lvdisplay /dev/vg/root

which should show a "LV Size" of 512.00 GiB as a result.

Then I made a new swap logical volume, of 2 GiB again:

sudo lvcreate -n swap -L 2G /dev/vg
sudo lvdisplay /dev/vg/swap

and reinitialised the swap space:

sudo mkswp -L swap /dev/vg/swap

(Note that this process changes the UUID of the swap partition, which might need to be fixed up, if you are mounting the swap by UUID rather than LV path or volume label.)

Resizing the `ext4` root file system

Now that everything below is resized, we can resize the ext4 filesystem. With ext4 this could actually be done online (while mounted), but because I was still booted into the Ubuntu Linux live environment, I did the resize offline, starting by checking the file system:

sudo e2fsck -f /dev/vg/root
sudo resize2fs -p /dev/vg/root

where the -p is for progress messages, but in practice the resize only took a few seconds on the Samsung 970 EVO Plus drive (as it only moves metadata around). The new file system size is reported in 4KiB blocks (another not very useful unit :-( ), as 134217728 4KiB blocks, which we can check is correct with 134217728 / 4 * 1024 * 1024 = 512 GiB.

After that I did another e2fsck -f /dev/vg/root check out of abundance of precaution, and then mounted the partition to check the expected contents were there (and verify the way the swap partition was mounted to reduce boot issues):

sudo mkdir /install
sudo mount /dev/vg/root /install
grep swap /install/etc/install

Fortunately the swap was being mounted by LV path (/dev/mapper/vg-swap) so it should survive being recreated elsewhere on the disk.

While it was mounted, I also checked the /dev/vg/root filesystem usage, to make sure I now had lots of free space:

sudo df -Pm /install

and that showed I had gone from about 98% used on the root partition to about 27% used. So I unmounted the file system again:

sudo umount /install

Running:

sudo vgdisplay

showed I had a bit over 300GiB of unallocated space in the volume group saved for later. (And if I do want to expand the root logical volume I would probably remove the swap again, and then recreate it, to keep the root logical volume in one LVM extent for simplicity.)

Conclusion

Once all the expansion steps were done from the Ubuntu Linux live environment, I simply rebooted, and Ubuntu 16.04 LTS and Windows 10 booted fine -- and I had lots more space in my Linux environment.

With a couple of days of effort, and a few hundred dollars for a new M.2 drive, I have managed to change my Dell XPS 9360 laptop from a persistently almost full root file system, to one which is about 27% full (and has about 300 GiB still available to allocate later). That makes it much more useful, potentially for several more years.

About a day of that time was consumed by:

Making backups
Copying the file system partitions around (especially to/from a USB 3 spinning disk)
Checking those backups, and copies
Waiting for Windows 10 to create USB Recovery drives (the 16GiB system install recovery drive took over 2.5 hours to write!)

and the remainder was research, getting Ubuntu Linux 16.04 LTS and Windows 10 booting again, etc. (Actually physically swapping the M.2 drive inside the Dell XPS 9360 took maybe half an hour including all the disassembly and reassembly.) I expect if I did it again the process would be faster, as I could have avoided some of the file system rearrangement I did by going directly to the final partition layout, knowing that I was going to have to make everything bootable again anyway.

Of note:

One potential advantage of not using the whole SSD, is that writes to the drive will be constrained to about the first 60% of the drive, which should reduce the amount of data that the SSD firmware feels it needs to shuffle around (particularly important because by default LUKS does not pass on TRIM/DISCARD requests for security reasons, so any block written to will then be copied around by the SSD firmware forever).
It turned out almost impossible to find the SSD erase block size, and align anything to those erase blocks (cf XFS Storage Layout Considerations). As best I could tell erase block sizes seem to be trade secrets now, certainly not appearing in data sheets and in some cases not even available by requests; and everything seems to be defaulting to aligning to 1 MiB blocks as being sufficient. Aligning to 1 MiB seems likely to be reasonable (especially after 5+ years of OSes doing that automatically, and vendors designing for those OS), but possibly not the most optimal choice in theory. So for single drive systems it probably makes sense just to let everything auto-align to 1 MiB boundaries, and ignore the problem.

Even modern local (internal) storage is basically a "network attached storage server" of its own, with its own ideas about how to store data, and its own storage API. It just happens to speak SATA or SAS or NVMe or similar, rather than Ethernet and TCP/IP.

ETA 2019-04-30: After doing all of this, when I next booted into Windows 10 for an extended period, I found that Windows Update was going to install (no option) the update:

Dell, Inc - Firmware - 9/27/2018 12:00:00 AM - 2.10.0
Status: Pending Install

(with the lovely message "We'll automatically install updates when you aren't using your device, or you can install them now if you want.").

It was not clear to me what it is. By searching on the Microsoft Catalog for 2.10.0 I could find three versions, with the matching date (and three 2.10.0 versions with a later update date of 3/25/2019); I think the three versions are for different versions of Windows 10, and my guess is this version is the one that would be installed on my Dell XPS 9360, since I think I have already updated to the latest Windows 10 release. Unfortunately there were no other details for what it is. And it was unclear if the install is being prompted by replacing the drive, or just by the date.

Since Windows 10 was not giving me an option, and I hate having things break randomly in the background, I chose to plug the laptop into the mains power, and let it "Install Now". Naturally it wanted to restart, and when it restarted it then proceeded to update the BIOS and firmware of everything in the laptop :-(

I am unclear whether this user hostile behaviour of forcing compulsory unscheduled firmware updates is Dell behaviour or new Microsoft behaviour (or both), why it was forced to a September 2018 version, and whether it was triggered by replacing the Toshiba M.2 drive that came with the laptop with the Samsung 970 EVO Plus drive -- or just triggered by the date / first sufficiently long Windows boot that something decided it had time to treat my laptop as its own.

Hopefully this unplanned firmware update does not adversely impact is on the Dell XPS 9360 that I had just spent a couple of days upgrading the drive :-( Fortunately Windows 10 and Ubuntu Linux 16.04 LTS do seem to boot up again.

(After rebooting into Windows again, Dell Update -- not Windows Update -- decided it had a further 12 updates it wanted to install, including a BIOS released 2019-04-22; but fortunately I could choose "Remind Later" to those, so that is what I did.)