Introduction
As a word of warning to other readers, this is basically a shaggy dog story, containing extremely extensively yak shaving to get to the point of installing a dual-boot system. Anyone hoping for a pithy guide to a dual-boot installation should look elsewhere. (But unfortunately those "elsewhere"s omit most of the hardships of the journey, hence this extensive yak shaving version.)
Almost every step in this process contained a risk of leaving the computer unbootable, or overwriting data. I do not recommend following anything in this blog post unless you are certain that you know exactly what it will do, and that you can get yourself out of any trouble you get yourself into. For basically all of it, my recovery strategy was basically "well it is a new machine, and I could always just do a factory install and start again". If you have data you care about on the system I would recommend experimenting somewhere else -- perhaps in a throw away VM? Proceed at your own risk. Even reading further may harm your sanity...
Background
Since I have been using OS X laptops for a while, the only laptop I have which "runs Linux" (without an Apple logo on it) is now pretty ancient -- the HP NC6220 is around 11 years old. Using an Apple laptop at a Linux Conference is frowned upon, and the NC6620 fairly rapidly became unsuitable in the last couple of years due to the switch from 1024x768 VGA for projection to 720p HDMI for projection. (The HP NC6220 is also... rather heavy to lug around by modern standards. At 2.3kg it is roughly twice the weight of modern "travel" laptops.)
I had been considering getting a "conference" laptop to run Linux for a while, and Apple's recent hardware moves (fewer expansion ports, removing the Esc key (!!), etc) definitely encouraged having another attempt at "Linux on the Desktop". (I had previously run "Linux on the Desktop" for about 10 years, from around 2000-2009, through actual desktops and two separate laptops -- with the HP NC6220 being the last Linux laptop before 6+ years of OS X on the Desktop.)
Looking at my options, the Dell XPS 13 range was getting a fair number of recommendations as a reasonable choice, particularly since there was a "Dell XPS 13 Developer Edition" pre-installed with Ubuntu Linux. Sadly that "Developer Edition" is only available in North America and certain European countries -- and definitely not available in New Zealand. For recent models it appears the hardware is basically identical between the Windows edition and the "Developer" edition; older models seemed to have some hardware differences in things like the Wifi cards. (I guess "Developer" is a code for "wants a Linux OS" :-) And hence Linux compatibility was important. FWIW, I ruled out a Chromebook as an option pretty early on because I am not keen on being tethered to the Googleverse, and terminals are not always enough; beside most of the Chromebooks are painfully low on resources. Also Dell's Chromebook was not available in New Zealand at the time I was buying.)
Late last month, Dell had a sale on selected Dell XPS 13 (9360) laptops, including the Z510891NZ model which was on sale for about NZ$2125 -- about 15% off -- with a roughly two week delivery time. It is a FHD (1920x1080p) model with a reasonably good CPU (i7-7500U, TurboBoost to 3.5GHz), 8GB of RAM and 256GB of SSD -- ie the bare minimum RAM/disk to be of any real use (same as the MacBook Air I bought earlier in the year to have a travel Mac for photography software, but obviously I was not going to dual boot that one).
After a bunch of investigation of other models, including the more expensive QHD (3k across) models (which had options for more RAM/SSD) and older "end of line" models (eg, 9350 - the late 2015 model), I eventually decided I could not justify the higher end models unless it was going to be a "permanent" desktop -- and if I was going to spend that much money it might as well be for the latest model, which would hopefully have the longest useful life as a "travel" laptop, as well as a better supported Wifi card. That may or may not have been the best choice, as it appears going with an Intel SkyLake processor means that USB-2 devices are not supported; where as they were supported on all previous models :-( I think this is just removing the EHCI hub and leaving only the xHCI hub, in which case USB 2.0 devices should work in backwards compatibility mode -- and so far that seems to be the case. But operating systems without USB 3 (xHCI controller) support will not work. Fortunately Linux has supported USB 3 (xHCI controller) for years.
I ordered the Z510891NZ model, with all the base options -- including Windows 10 Home, rather than the "Pro" upgrade. It is a Dell XPS (9360 - Late 2016) model, with 1920x1080p, 8GB of RAM and 256 GB SSD. (If it were my main work laptop, 16GB of RAM and 512GB-1TB of SSD would have been the bare minimum; my existing work laptop has that, and the storage is still pretty tight. But with a travel laptop I can leave most of my work behind -- and running Linux natively means I can mostly avoid the need to run VMs on my laptop, which are usually there to run Linux...)
The only extra I bought at order time was the Dell DA200 USB-C to HDMI/VGA/Ethernet/USB 3 adapter, since it was relatively inexpensive and potentially useful. However it is unclear how well the Dell DA200 adapter is actually supported under Linux, so I may need some different adapters to use with Linux.
For reference:
Dell XPS 13 (9360) Setup and Specifications (online webpage only)
Dell XPS 13 (9360) Service Manual (online webpage only)
Dual Boot
Unlike my previous "Linux on Laptop" installs, where I booted straight off the Linux install media the first time I powered the laptop on and overwrite everything on the disk, this time around I plan to retain Windows 10 Home as a dual boot option. The main reasons are:
Modern vendor laptops do not come with Windows reinstallation media in the box, so it is harder to change your mind later and reinstall (at present it seems Dell provide a means to download recovery media given a Dell Service Tag; but presumably that is valid only for the warranty period or similar; they also have an at-purchase option to get reinstall media, but it is a substantial fraction of the cost of the laptop so I did not do that).
The Microsoft of 2016 is less hostile than the Microsoft of 10 years ago -- they even joined the Linux Foundation and release software on GitHub; it appears Linux everywhere has the newer Microsoft leadership thinking about cooperation more than domination.
Occasionally I run into tasks for which having a Microsoft Windows system would be helpful -- and I have no other Microsoft Windows systems (and mostly never have, apart from one "for games" computer many many years ago).
Having been bought with a Windows license, the warranty covers only Windows and not Ubuntu Linux, so in the event of a hardware fault being able to demonstrate it on Windows may help.
So while I do not expect to use Microsoft Windows much, I am inclined to retain it on the smallest reasonable portion of the drive that I can, at least until I determine what I am doing with this laptop long term.
On the Linux side, installing Ubuntu 16.04 LTS is a matter of pragmatism -- I prefer Debian-like Linux operating systems, and Ubuntu Linux is the one that has been used by Dell for the "Developer Edition", which means in theory it should be easier to get the hardware working. I am not really a fan of the direction that Canonical has taken Ubuntu Linux Desktop, but for a first install shortly before travel, pragmatism is winning. Dell even have a guide to Ubuntu and Windows 10 dual boot on their hardware, so saying I am "running Ubuntu" at least stands a chance of being understood by the support people.
So my dual boot will be Windows 10 Home and Ubuntu 16.04 LTS.
Windows First
Microsoft Windows needs to be first on the hard drive; it also includes
no ability to dual boot anything else. (I guess Microsoft cooperation
with Linux, etc, has not got that far yet!) Since it came pre-installed
on the hard drive that is fairly easily accomplished -- and just requires
shrinking the Microsoft Windows disk partition to leave room at the end
to install Linux, then have Linux (grub) take control of the boot menu.
Windows Setup
On first power on, Microsoft/Dell's Windows 10 pre-install guides you through finishing setting up the operating system, like most other modern OS installs. There is nothing especially surprising to watch out for other than:
Gigantic legal agreements, displayed in two half-width columns with Microsoft on the left and Dell/others on the right (it appears the Microsoft legal agreement is roughly half length of the Dell/others agreements); these are naturally "all or nothing"...
The "Get going fast" page which tries to get you to skip past your ability to opt out of leaking information to Microsoft/Dell; at that point you really want the "Customise" button. There are three pages of customisations (despite the first one having a scroll bar they appear to all fit on the first screen), and I turned all but one of them off (I left "Use SmartScreen online services" on because it seems the most likely to be useful to protect Windows from the regular drive-by attacks).
Skip the Microsoft Account page (click on "Skip this step" link at the bottom)
If you enter a (full) name with a space in it at the "Who is going to use this PC?" prompt then you will get a home directory with a space in it. Joy. (Cf Apple and Linux which will usually try to give you, eg, a first name as your username/home directory and/or ask for a full name separately.)
Choose "Not Now" for Cortana to try to avoid an always-on microphone... (No idea if they leave the microphone on anyway :-( I too would prefer a computer/laptop without a microphone; or at least one that can be definitively disabled. If I need a microphone then I would prefer to plug in a headset, or a decent recording setup.)
You can leave the "Support and Protection" page completely empty and just click "Next", which seems to have the effect of skipping it (and/or turning that into "remind me later").
I also chose "remind me later" at the Dell XPS registration page (I bought it direct from Dell; they should know I have it, so I do not feel compelled to give Dell more information). If you get to the end of the Dell XPS Welcome App it tries to get you to activate McAfee/Dropbox; it appears the only way out is to close the app, so that is what I did. (They keep reappearing periodically, eg, after rebooting; and the only opt out option remains to be to close them. No wonder Windows users are annoyed by computers...)
Windows updates
It is probably worth installing all the Windows updates before proceeding.
A while after booting the notification window tells you "we are adding some new features to windows. This could take a few minutes", which I assumed indicates that auto-update defaults to turned on. And checking in Windows Settings -> Update & Security -> Update Settings that is true -- they are on by default. (It claims "Available updates will be downloaded and installed automatically, except over metered connections (where charges may apply)" -- but I doubt its ability to determine that, as many NZ connections are metered including the one I was installing on. People have had horror stories of thousand dollar bills from Windows update over mobile tethering, etc.) So far I have not gone digging to try to find a way to avoid automatic updates/downloads using up my bandwidth... but fortunately I do not plan to run Windows very often, so mostly I get to choose when it updates by when I run Windows :-)
That same "Update Settings" screen initially told me that no updates were available, but unsurprisingly for a "new out of the box" machine when I told it to actually check for updates now, it found several to download and install. Including an Adobe Flash update (surprise!) and a cumulative Windows security update (surprise!). So I let those install and rebooted before continuing.
Update Dell Firmware
Dell firmware updates are released fairly regularly and have helped with Linux support in the past. They are most easily installed from within Windows -- so it is worth checking the installed firmware and updating it if required before proceeding further.
The Dell Update Utility pre-installed in Windows prompted to install
a BIOS Update, while I was preparing a USB drive for the making the
Windows Recovery Media (below), so I let it go ahead and (try to)
install the BIOS Update. I believe it was trying to install the
Dell XPS 9360 BIOS version 1.2.3,
1.2.3,
which was released 2016-12-14 (in North America presumably, so
literally today as I was setting the system up; yesterday when I
checked the latest BIOS available was 99.3.24, 1.0.7 released
2016-09-26). But it did not say which version it was trying to
install.
For some reason that automatic BIOS update did not seem to do anything (it just restarted without an error message): I keep getting repeatedly prompted to do the BIOS update by Dell Update Manager (eg, each boot), despite trying a few times and even shutting down; and when I go into setup (F2 when the Dell logo is displayed; there's no prompt to do so) I can see that BIOS version 1.0.7 is still installed.
Since I did not actually seem to need to install a new BIOS the day after it was released, I elected to leave actually installing this BIOS update for later, and carry on with creating Windows recovery media.
With the benefit of hindsight that was a mistake, as one of the fixes appears to have been relevant to much of my frustration with creating the Windows Recovery disk: the right USB port not being activated for safe removal -- ie devices plugged in there not showing up as "removable" -- and it took me a long time to accidentally try plugging it in the left hand USB port (which worked) by mistake, and longer still to realise that was important.
Given that the automatic BIOS update never worked (apparently never actually triggered the firmware update process in the BIOS at all), after creating multiple Windows Recovery disks, I went back and explicitly upgraded the BIOS by manually downloading the 1.2.3 BIOS update, and manually running it. It prompted for all the various firmware it was going to update (clearly it is actually a "firmware bundle"), and then rebooted -- and on reboot updated the firmware one at a time, ending with a cold restart of the system (even power light went off briefly). After it was done the BIOS had been updated to 1.2.3, as visible in the F2 setup screen.
The BIOS update included:
System BIOS with BIOS Guard: 1.2.3
Embedded Controller: 1.0.3
Intel Management Engine (VPro) Update: 11.6.1.1142
Intel Management Engine (Non-VPro) Update: 11.6.1.1142
Main System TI Port Controller 0: 1.2.6 (unchanged from 1.0.7 BIOS)
Dino2MLK Board Map: 1.0.1 (unchanged from 1.0.7 BIOS)
PCR0 XML: 0.0.0.1 (unchanged from 1.0.7 BIOS)
so there were four separate firmware update processes run, from within the UEFI environment.
(For reference supposedly one should disable BitLocker when updating the BIOS, presumably in case the TPM loses the encryption key for it... but in my case it seems to have worked without doing so. Windows Disk Management claims my drive is "Bitlocker Encrypted" -- out of the box, I assume -- but the encryption panel tells me that I "need a Microsoft Account" to complete the encryption -- presumably to escrow the decryption key. Personally I would prefer to just have to reinstall. So I chose not to "Turn Off" encryption, because it was unclear if I would be able to turn it on again. Also it is unclear whether or not BitLocker encryption is actually fully active, since the system apparently completely boots without prompting for a password -- unlike what happens on a Mac with an encrypted drive, where you get a pre-boot prompt for a password.)
ETA, 2017-08-23: On further investigation it turns out that Windows 10 Home does not include BitLocker at all (it is only available in Windows 10 Professional and above -- a NZ$169 upgrade if purchased after the machine purchase). Since my Dell XPS 13 was purchased with Windows 10 Home (to save money since it was mostly intended for use with Linux), that means it does not have BitLocker functionality enabled. Which probably explains why I did not need to do anything to disable BitLocker before installing Ubuntu Linux 16.04 LTS dual boot.
Windows recovery media
Pre-installed Microsoft Windows systems generally do not ship with install media by default any longer -- if it is available, it is a "value add" extra cost (around 3-5% of the computer cost in my case).
Instead Microsoft/Dell provide a means to (attempt to) create your own recovery media from the running pre-installed system. The process seems to be fairly error prone (it took me at least a dozen attempts to get started creating a recovery drive with the system files included trying solutions at random; and that first attempt that I got started failed very quickly without an indication why).
The error messages are extremely opaque, leaving the user
to try to guess what might have gone wrong, and find obscure commands
(eg, "sfc /scannow" to check for file system corruption; reagentc
/info to check the Windows Recovery
Environment
is enabled) to try to see if they will fix the issue (neither found
any problem in my case). The pathological desire to never show a
user any error message other than "sorry it did not work" leaves
users guessing what might be the problem, and causes much more
irritation than simply displaying "incomprehensible" error messages
-- at least users can search for the "incomprehensible" error message
and find more specific information than searching for "it did not
work" will return. I would have given up much earlier if it were
not for the fact that I planned to do multiple things that risked breaking
the Windows install: change the drive mode to AHCI; resize the main
Windows partition to make room for Linux; install Linux onto partitions
specified by number.
With the benefit of hindsight, it appears the main issue that I ran into was that in the Dell XPS 13 (9360) BIOS 1.0.7, the right USB port had the issue "no 'safe remove' on taskbar when plug-in USB key to right USB port" (supposedly fixed in the just released BIOS 1.2.3) which meant that devices plugged in there were not seen as removable, and thus not possible candidates for creating a recovery image. But it did not occur to me to try the left USB port until well after I had already concluded that 16GB was actually not large enough; I figured this out only after spending a lot of time (slow) formatting a larger drive so that it would work. Even a message saying "no removable drives found" would have helped, as would a message saying "the removable drives found is too small"; but the tool has neither message. (Ironically I did actually try to update the BIOS to a version with this fix before trying the tool; but the BIOS update failed -- see above.)
What worked for me:
Find a 16GB or larger USB drive (in practice about 10GB seems to be actually used on the Dell XPS 13 (9360) with Microsoft Windows 10 Home at present; so a "16GB" USB drive which is actually only 14.5GiB or 14.8GiB should actually work...)
Ensure nothing is needed from the USB drive. If it has previously been used elsewhere (eg on a Mac) you may have to explicitly ensure that it has (a) a single primary partition, and (b) that primary partition has the partition type "
C" (W95 FAT32), otherwise it may not be recognised (I think this was the case for multiple frustrating attempts with a previously used "32GB" USB drive, previously formatted on OS X, which was not recognised by the recovery tool until I explicitly changed the partition type under Linux, and made a fresh Fat32 filesystem on it).Connect the USB drive to the left USB 3 port on the Dell XPS 13, not the right USB 3 port, so that it is detected as a removable drive (perhaps unless you have successfully updated the BIOS to 1.2.3 -- see above -- but I have not tried again since; the hint that it is not detected as removable is the words "Local Disk" displaying in the File Manager...)
Use Disk Management (right-click on the Windows logo) to format the USB drive as a FAT32 file system, using a full format (ie not the quick format) unless it is brand new out of the box. That will take quite some time for a larger USB 2 drive (apparently multiple hours in my case). Among other things the full format forces all sectors to be written and reallocated, and ensures any old file system data does not remain on the drive to cause confusion. The full format should not be necessary for a drive that is brand new; just a quick format.
Leave the drive plugged in! There is both no need to eject/unplug it, it also seems to work better if stays plugged in after the format. (Despite the instructions telling you to plug it in after starting the recovery tool.)
Open the "Create a recovery drive" control panel, eg, by searching windows for "
create recovery drive"; it appears this will eventually run the "Create USB Recovery" app, but apparently one should not run that directly.Permit "Recovery Media Creator" (published by "Microsoft Windows") to make changes to your device (ie, "yes" to the User Access Control).
Make sure "Back up system files to the recovery drive" stays checked, so that the recovery media can be used to do a full reset/reinstall from the drive (rather than just the recovery partition; if it is unchecked, you basically get a separate boot disk from which you can attempt to repair the existing install, or using the factory recovery partition).
Once the stars align so that your USB drive is recognised as an option you will see your USB drive as an "available drive", and "Next" will be selectable; select your USB drive and click "Next".
Confirm that the drive can be overwritten and you are okay with all content being lost (hopefully nothing left on it before this point), by clicking on "Create".
All going well, the recovery drive should be created, with a progress bar covering preparing the drive, formatting the drive, copying utilities, backing up system files, and copying system (most of the progress bar). It should end with "The recovery drive is ready" and a "Finish" button.
Click "Finish" to exit, and the app will close. You should end up with a "RECOVERY" drive. In my case the first RECOVERY drive I successfully made had 21GB free out of 29GB (ie, it took about 8-10GB), so I made a second one on a smaller smaller "16GB" (14.5GiB) USB drive once I had figured out the USB drive had to be placed in the left USB 3 port on the Dell XPS 9360 with BIOS 1.0.7.
Some versions of the the tool apparently offer an option to delete the recovery partition from the drive, if it is copying from the Recovery Partition but this did not appear in my case (good, as I had planned not to delete the recovery partition: on my system the various recovery partitions take about 10GB, which is space that would be useful, but not so useful as to be worth making it impossible to use that as a recovery mechanism later on).
Generally it seems creating the Recovery Drive without the system files (ie, bootable only; no reset/reinstall options) works for many people, and takes relatively little space (under 1GB it seems; it asked for a 512MB USB drive, and worked for me with a "1GB" one). That Recovery drive without system files is basically equivalent to a 1990s emergency boot floppy -- just scaled up to larger software sizes.
But the version with the system files seems to be very problematic for lots of people, with very little indication of the specific cause of the problem, due to the pathological desire never to display a specific error message. There is no good reason for a tool that almost every user is "recommended to run" to be this error prone and user hostile.
For reference, TenForums has a good illustrated guide to creating the recovery drive, which indicates that "Back up system files to the recovery drive" is actually creating a system backup. It also has another article on using the recovery drive, and an explanation of the Windows 10 advanced startup options. (Also of note, an article on how to find your Windows Product Code -- although it appears to also be displayed in the "System" control panel, which would seem easier to find...)
The other plausible "bare metal" recovery option is to download Dell's Windows Recovery Image for the Service Tag, which in my case is a 6.4GB file. Then follow the instructions to create Windows 10 install media -- which are Windows specific but appear to be basically making a FAT32 file system from the command line, and copying the contents of the downloaded ISO onto it. So I have downloaded that file too (about a 6 hour download...), for safe keeping (since my faith it will remain available for a long time is pretty low).
For later reference, using the recovery drive, which involves pressing F12 during the Dell logo display (for the "one time boot menu"), and choosing to boot off the USB drive (most likely the obscurely named UEFI Boot option that is not the "Windows Boot Manager"). Unless you have an activity indicator on the USB drive, the main indication is that the boot is much slower than normal... and you should arrive at a menu letting you choose your keyboard layout. From there the Troubleshoot menu will allow doing a Factory Image Restore (presumably from the restore partition), and offer other Advanced Options to repair the system, including a command prompt. (The recovery drive with the full system image adds "Recover from a drive" to the options at the top menu, in addition to "Factory Image Restore" which is from the recovery partition.)
In theory no product key is required to use the the recovery media providing the recovery media is made on an activated system (check via Start -> Settings -> Update & Security -> Activation). I am not sure if that is true of doing the factory restore from the recovery partition; hopefully it is, as the days of Microsoft Certificate of Authenticity being plastered over every laptop seem to be over. (The only sticker is an Intel i7 inside one.)
Changing the disk to AHCI mode
Linux will only install on the NVMe hard disk when it is in AHCI mode, not when it is in "RAID" mode. Dell defaults to setting it to "RAID" mode, and installing Microsoft Windows 10 Home in "RAID" mode. (This "RAID" mode uses Intel Rapid Storage Technology (IRST) which appears to use a RAID mode on the individual storage chips on the NVMe SSD; but a driver for this particular IRST seems to be only available for Microsoft Windows, so all the Linux install instructions for the Dell XPS describe changing to AHCI mode.)
Microsoft Windows is understandably surprised at having its disk device changed, and needs some reassurance before it will boot again -- but it can be persuaded to work in AHCI mode fairly easily. The most useful instructions on changing Microsoft Windows to AHCI mode list the steps:
Right click the Window icon and select to run the
Command Prompt (Admin)from among the various options, to start an Administrator Command Prompt.Invoke a Safe Mode boot with the command:
bcdedit /set {current} safeboot minimalRestart the PC.
When the Dell logo appears, press F2 to enter the BIOS.
Navigate to System Configuration -> SATA Operation
Change from
RAID On(Intel Rapid Restore Technology) toAHCImode.Read the warning about changing the SATA Operation mode (basically the OS may be confused -- hence asking for safe mode above)
Answer "Yes" to the "are you sure you want to continue?" question.
Hit "Apply" (at the bottom of the screen), ensure "Save as custom user settings" is ticked, then click "Ok" on the "Apply Settings Confirmation" window.
Exit by the BIOS settings by hitting the "Exit" button at the bottom right.
The system will do a cold restart (the powered on indicator goes off briefly then comes back on), then boot up Windows.
Windows 10 will launch in Safe Mode, as requested above. (There will be various warnings about things that cannot run in Safe Mode, while an Administrator is logged in, which can be temporarily ignored.)
Right click the Window icon and select to run the
Command Prompt (Admin)from among the various options, to start an Administrator Command Prompt.Cancel Safe Mode booting with the command:
bcdedit /deletevalue {current} safebootRestart your PC once more and this time it will boot up normally but with AHCI mode activated.
That uses a minimal "safe mode" boot to allow Windows to redetect
the hard drive device, after which it should then boot normally.
Note that the command includes the word "safeboot" not "safemode";
if the /deletevalue is refused then check that you have entered
the command correctly. (Some people noticed freezes on the SSD in
Windows,
for which there is apparently an update that helps -- a Samsung
950 Pro driver
update
I think; for now I am hoping that the update has made it into the
base system in the last 6 months. So far I have not seen any issues, but
the system has had only minimal usage to date.)
In theory the disk access will be somewhat slower in AHCI mode, but
it does have the benefit of actually working in both Microsoft Windows 10
and Ubuntu Linux 16.04 LTS.
Shrink Windows disk to make room for Ubuntu Linux
It is possible to shrink the Windows drive from within Windows these days, at least with Windows 10. (Other people have used other freeware tools.)
To do it with the built in Windows tools it is supposed to be possible to:
Right click on the Start button, and run "Disk Management"
Find
C:, the Active/Primary NTFS partitionRight click on that partition, and choose "Shrink Volume..."
Enter the amount to shrink by, ie make available for other use
Click on the "Shrink" button
But this did not work for me (see below).
In terms of choosing the amount to shrink, with only a "256 GB"
(238 GiB) drive installed, and dual booting, space is very tight.
Other than Microsoft Windows 10's partition (C:) there are several
other pre-installed partitions:
500MB UEFI boot partition at the start of the drive
450MB "Recovery" partition (UEFI boot for recovery, I think -- the label is "
WINRETOOLS"), after the Microsoft Windows 10 partition.9.85GB Recovery partition (I assume with the factory install image on it; the label is "
Image"), after that.1.08GB Recovery partition (with the label "
DELLSUPPORT, presumably some sort of diagnostics or similar) at the end of the drive.
So that is a total of 12GB used up by non-Windows/non-Linux things, leaving 226GB to be shared by Microsoft Windows 10 and Ubuntu Linux 16.04 LTS. That 226GB is the default size of the Microsoft Windows 10 partition. Of that 226GB, Microsoft Windows 10 is using 32GB basically out of the box, leaving 194GB free.
Given that I am mostly going to use Linux, I wanted most of the space for Linux, but I did want to leave some free space for Windows, in case I wanted to run third party applications there. So a reasonable option appeared to be about 33% to 40% of the drive for Microsoft Windows, leaving 60% to 66% for Linux. Fiddling with some numbers, the best round numbers seemed to be taking 140GB for Linux (about 62% of the drive) and leaving 86GB for Windows (about 38% of the drive), which leaves a bit over 50GB free for Windows. That is not dramatically smaller than the usable space for Microsoft Windows on a 128GB SSD (about 107GB for Windows) so it seems like it should be tight, but okay.
So I chose to shrink the Windows C: partition by 140GB, to leave
86GB (about 50GB free) for Windows.
Unfortunately when I tried to do so via the Disk Management GUI, all I got was an error from "Virtual Disk Manager" that "The parameter is incorrect" (without actually bothering to say which parameter is incorrect :-( ). The Windows Application Log suggest that the error is Event ID 257 (link is Windows 8.1, which got a hot fix, not Windows 10 though; there seems to have been an issue with SSD defrag on Windows 8). There was also a code 0x80070057 in the Event Log, which seems to appear in various other situations (mostly around upgrades to Windows 10 from earlier versions of Windows) without a clear resolution. For reference, this Error ID 257 seems to be different from the more common Event ID 259 error -- Event ID 259 seems to relate to unmovable files.
After a lot of stumbling around trying various other things, I eventually found that I could shrink the drive, but only if nothing ever tried to ask for the maximum amount the drive could shrink -- and the GUI version of Disk Manager always asks what the maximum it is that it can shrink the drive before even asking how much you want to shrink the drive.
The work around was to shrink the disk at the command line:
Right click on the Windows logo and run a "
Command Prompt (Admin)"Run "
diskpart"Do "
list volume" to see the volumesDo "
select volume N" to select the volume to shrink; in my case "select volume 0"Guess a safe amount to shrink and attempt to shrink that amount; I found that I could not shrink by the full amount that I wanted in one go (it was refused as above the minimum amount), but I could get there in three steps.
Shrink with: "
shrink desired=NNNNN" whereNNNNNis the amount in megabytes to shrink the existing volume by. The three steps I used were:shrink desired=51200 shrink desired=51200 shrink desired=40960to shrink by a total of 140GiB (those numbers being 50 * 1024 and 40 * 1024 respectively).
Use "
list volume" again (potentially after each shrink command as you go) to verify that it happened as planned. In my case the result was an 86GBC:as intended.Use "
exit" to leavediskpart
In particular it is vital not to run "shrink querymax" as that is the
command that fails with Error 257, 0x80070057, for reasons I still have
not established. Once "shrink querymax" it appears you have to quit
diskpart and start it again before it will do any shrinking, even of
a specified size.
When you are done in the command line, going back into the Disk
Management GUI should allow you to verify that you have an 86GB C:,
and 140GB of "Unallocated Space" immediately after it.
After shrinking the drive it is useful to reboot Windows a couple of times to make sure it is happy with the new, smaller, partition.
For the record, things that did not work to resolve the problem
of using the Disk Management GUI to shrink the drive (or diskpart's
shrink querymax):
"
sfc /scannow" (no errors, no change in symptoms)"
DISM.exe /Online / Cleanup-image /Scanhealth" (no errors, no change in symptoms)"
DISM.exe /Online / Cleanup-image /Restorehealth" (which failed early on with Error 1726 -- some sort of RPC failure -- the first time but worked the second time. However it made no difference to the ability to shrink the drive :-( See also more on running DISM checks.)To rule it out, as it was a common issue shrinking volumes, I also tried the suggestion of sancho.s on superuser.com, following the download3k instructions. These temporarily disable several key Windows features in the hope of getting them out of the way of shrinking the drive; may sure you have good backups of anything you want to keep first! Unfortunately they made no difference in my case, so I tried to turn everything back on again (although obviously the system restore points prior to turning system restore back on are now lost :-( ).
I also tried temporarily going back to "Raid on (IRST)" mode for the disk, but that also appeared to make no difference -- ie, same symptoms when attempting to shrink the drive. (But for what it is worth the command line shrinking was done in "Raid On (IRST)" mode, and I put it back to
AHCImode after the shrinking was complete. I do not think the disk mode made any difference.)Temporarily turning off McAffe AntiVirus (which Dell had force-installed :-( ) also did not change the symptoms (and it seems to turn itself on at each boot too).
For completeness I also tried to do the Shrink in Safe Mode, but got told "This service cannot be started in Safe Mode" -- a different message, but no more helpful. (It is also unintuitive why shrinking the disk should be blocked in safe mode; shrinking the disk is the sort of thing you would expect to want to do with fewer processes running...)
The only commonly suggested thing that I did not try was deleting the NTFS Journal, but mostly because I figured out how to make it work with the command line
diskpartfirst...
Depending on how I end up using the laptop, it is possible I may end up re-partitioning it again -- which will involve re-installing Ubuntu Linux, but hopefully should not require reinstalling Windows (as I think it can Expand/Shrink upwards). For the lack of other suggestions, I tried following the guide to shrink beyond where unmovable files are located -- basically by disabling the features that include those files.
(For reference, it is also possible to shrink the drive from within
the Ubuntu Linux installer, using gparted, but it seemed safer
to let Windows shrink
itself, on
a fairly fresh install, as Windows will understand NTFS
best.
In hindsight it seems like it would have been a lot faster to
shrink the drive in the Ubuntu installer, or maybe to use the
freeware Partition
Manager
suggested by this superuser.com
answer.)
ETA, 2017-03-30: Michael Wisniewski, who was following this
guide emailed to point out that it is worth jumping ahead to the
"Adding Windows back into the grub boot menu" (at the end of
this post) at this point, and explicitly disabling Fast Boot in
Windows
10
before carrying on, as that will make grub automatically recognise
the Windows partition during the install and save some of the extra
mucking around with rebooting that I needed to do (as detailed
below). If you do, be sure to shut down and power off Windows 10
completely before carrying on, to ensure the Fast Boot files are
flushed from the disk.
Ubuntu Linux 16.04 LTS
Creating an install USB drive
For reasons which are not completely obvious, Ubuntu Linux (and most other Linux distros) still distribute their installation media as CD/DVD ISO images, even though most installs these days are done on machines without physical CD/DVD drives. So modern installs are typically either from virtualised CD/DVD drives (eg, via Dell iDRAC) or from USB drives -- and the same features providing virtualised CD/DVD drives usually also provide virtualised USB drives now.
Possibly it is time for Linux distros to start providing install
media in a form which can just be directly copied onto a USB drive
(eg, with dd). However since that has not happened there are
literally
dozens
of tools to create a USB install
disk,
starting from an existing Ubuntu Linux
system,
a Windows system with third party
software,
or on a Mac
(but it is not clear if the media created on a Mac will only work
on a MacBook, as it is not clear what ends up in the .img
file, as the UDRW
image format seems to be Mac specific). What these tools actually
do seem to be a pretty well guarded secret, for non-obvious reasons.
Although some of them, like rufus
are actually open source, so presumably one could reverse engineer
the process... Unfortunately many of the recommendations online
seem not to work when the computer is in UEFI
mode (which you would think
was pretty common these days).
Fortunately for a UEFI install the steps actually required appear to be pretty simple:
Ensure your computer is booting in UEFI mode (very likely with a modern Windows 10 pre-install, especially if Secure Boot is enabled)
Find a 2GB+ USB drive that you can completely overwrite (even the partition table will be overwritten in the next step; so all contents will be lost). Plug the device into a Linux computer.
Create a removable USB drive with an
msdos(MBR or Master Boot Record) partition table, with a singleFAT32. (Supposedly the drive should have aGPT(GUID Partition table) with a singleFAT32file system on it, eg from another Linux system, but when I tried with agptpartition table, the USB drive was not recognised in the Dell XPS 13 UEFI boot menu. I do not know why.) These instructions are for anmsdospartition table which seemed to work for me even in UEFI boot mode:sudo apt-get install parted sudo parted /dev/sdN print # Check it is right device! sudo parted /dev/sdN mklabel msdos # Overwrite partition table sudo parted /dev/sdN print | grep msdos # Check it is a msdos table now sudo parted /dev/sdN mkpart primary fat32 0% 100% sudo parted /dev/sdN print # Verify partition created sudo mkfs.vfat -F 32 -n UBUNTU -v /dev/sdN1 # Make file system sudo parted /dev/sdN print # Check fat32 detected sudo parted /dev/sdc align-check opt 1 # Verify alignmentwhere
/dev/sdNis the device of your removable drive (be extra careful that you have identified your removable drive, and not one of your internal drives!)At the end, for the
msdospartition table you want something like:ewen@tv:~$ sudo parted /dev/sdc print Model: SanDisk Cruzer Blade (scsi) Disk /dev/sdc: 8003MB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 1049kB 8003MB 8002MB primary fat32 boot, lba ewen@tv:~$Note that sector 34 is the first available sector when using a GPT, which is 17408 bytes into the disk; but it may not be well aligned.
partedis very opinionated about partition alignment, but unwilling to share its recommendations beyond "the computer says no". Using the calculation tool posted here it seems like2048s(ie, 1MiB, but not 1MB) is likely to result in acceptable alignment, and the default when0%is specified;100%is a shortcut for "until end of the device". This0% 100%shortcut seems poorly documented (see eg attempt 4 of the blog post with the calculation tool). It is really unfortunate thatparted(a) requires you to enter the start/end values, and (b) does not bother to share its recommended alignment when telling you the provided start/end values are not to its taste. Using percentages seems to be the only way to trickpartedinto picking its favourite alignment rather than simply critiquing your own guesses with useless "no, guess again" messages.Download the latest Ubuntu 16.04 LTS 64-bit (
amd64) desktop install ISO, currentlyubuntu-16.04.1-desktop-amd64.iso, and ideally at least verify the SHA1SUM (in thereleases/16.04/SHA1SUMSfile).Copy the contents of the ISO, either by mounting it or by directly extracting it with something like
p7zip. I chose to usep7zipbecause it was available, and there were already plenty of yaks shaved so far.sudo apt-get install p7zip-full # Mount USB drive, if not auto-mounted; uid/gid values set to my own sudo mount -o uid=$(id -u),gid=$(id -g) /dev/sdN1 /mnt cd /mnt 7z x /var/tmp/ubuntu-16.04.1-desktop-amd64.iso # Verify that the EFI Boot files also got extracted ls EFI/BOOT/ # Unmount the drive cd sudo umount /mntMost of what is extracted is a squashfs file system container file; the rest is just ancillary files to get the system to boot in various environments.
For certainty, explicitly mark the partition as bootable:
sudo parted /dev/sdN set 1 boot on sudo parted /dev/sdN print | grep boot # Check for boot flag
Then remove the USB drive from the computer your are creating it on, ready to be used as an Ubuntu 16.04 LTS liveboot or install disk.
See also hints on dual booting with UEFI install, and working with Ubuntu secure boot; although I did not pursue these when trying to figure out why the GPT partition table version would not boot.
Booting the Ubuntu 16.04 LTS installer
Plug the USB drive created above into the Dell XPS 13 (9360). Because of superstition, I plugged it into the left USB 3 port (see above for why!). Power on the machine, or restart it.
When the Dell Logo appears press F12 to get the "one-time boot menu". All going well you should see:
"Boot mode is set to: UEFI; Secure Boot: ON"
"UEFI BOOT:" menu listing two items
The second item should be something like:
UEFI: SanDisk, Partition 1where the vendor matches the vendor of your USB drive.
(The first item is labelled "Windows Boot Manager", and appears to actually boot the Dell SupportAssist partition at the end of the drive; it is unclear why it does not have a better name than "Windows Boot Manager".)
Highlight the "UEFI: SanDisk, Partition 1" (second) entry and press return to boot it. You should get a GRUB menu offering "Try Ubuntu without installing" and "Install Ubuntu" as the first two options (in a minuscule font, in the top left corner, even on only a 1080p display -- apparently whoever wrote this has amazing eyesight, or really likes negative space!).
The "Install Ubuntu" (second entry) starts a simplified installer for Ubuntu Linux 16.04 LTS, which it turns out (see below) is too simple for my requirements. It is also possible to use the "Try Ubuntu without installing" to kick off an install to the hard drive, from within the live environment -- and that is what I needed to do (see below).
Ubuntu 16.04 LTS disk layout
My aim with this installation is to have:
Microsoft Windows 10 / Ubuntu Linux 16.04 LTS dual boot
UEFI Secure Boot enabled (so Microsoft Windows 10 will run)
Ubuntu 16.04 LTS installed on an encrypted partition (because it is a laptop, and will be used for travel)
Unfortunately it turns out that this complexity is beyond the standard Ubuntu 16.04 LTS Installer -- ie the one you get with "Install Ubuntu" -- so various manual setup is required. (The default installer will let you have either (a) encrypted LVM by itself on the drive (wiping everything else) or (b) dual boot with Windows -- but not both. This may be partially a limitation of the Desktop installer... as I found hints that maybe the other installers were more cooperative; but I had already downloaded the Desktop installer and did not feel like, eg, downloading the Alternative installer in the hope of more options.) Sadly hunting for answers turns up a lot of older information, so I tried to stick to Ubuntu Linux 14.04 guides on the assumption they would be more relevant to Ubuntu Linux 16.04.
The recommended approach is to boot Ubuntu in as a live environment -- "Try Ubuntu without installing" -- and then use that live environment to set up the disk before invoking the "Install Ubuntu 16.04 LTS" option. This requires a series of manual steps.
When taking this approach on the Dell XPS 13 (9360) the first thing to note carefully is:
/dev/sdawill be the USB drive you booted fromThere is no
/dev/sdbThe drive you want to install onto is
/dev/nvme0n1, because the modern Linux device naming convention aims for "stable device names" at the expense of ease of use.There are partitions on that
/dev/nvme0n1with the suffixpN, for each partition, eg,/dev/nvme0n1p1,/dev/nvme0n1p2and so on.The free space created above (by shrinking the Windows partition) will be in the middle of the drive -- between
/dev/nvme0n1p3and/dev/nvme0n1p4.The EFI boot partition is at the start, as
/dev/nvme0n1p1.The easiest way to check all of this is:
sudo parted --listwhich will discover and list all storage devices and their partitions. Once you find the right one, it is more useful to do:
sudo parted /dev/nvme0n1 printand avoid the confusion of the other devices.
The closest guide to what I
wanted
recommended creating two partitions for Linux -- one for /boot,
to be unencrypted, and one for an encrypted LVM volume that contains
everything else. The /boot partition needs to be 512MB-1GB, so that
there is room for multiple kernels; I chose 512MB because I was tight on
space already (and know from past experience that with Ubuntu's rate of
kernel version number churn, 256MB is too tight). The LVM partition
can be the remainder of the free space.
To actually partition the drive, it is easier to use the gparted GUI
rather than fight with the command line options. Run with:
sudo gparted /dev/nvme0n1
and verify that the disk partition map shown looks (essentially) the same as you saw in Windows 10.
Then create the partitions by:
Highlighting the "unallocated" 140GB
Choosing Partition -> New
For the first partition choose:
Free space preceding (MiB): 0
Free space following (MiB): 512 (leaving room for
/bootafter it)Create as: primary partition (yay GPT, and many primary partitions)
Partition name: Ubuntu
File system: unformatted
with the partition size auto-calculated to use all but 512 MiB of the unallocated space, and the
Label:left empty since we are not making a file system. Once it looks okay, click "Add" and verify the "New Partition #1" appears where you expected.For the second partition chose:
Free space preceding (MiB): 0
New size (MiB): 512
Free space following (MiB): 0
Create as: primary partition
Partition name: UbuntuBoot
File system: ext2
Label: UbuntuBoot
with the hope that the Partition Name/Label will help us identify this boot partition later. Click "Add" and verify that "New Partition #2" appears where you expected it.
At this point you should have a "New Partition #1" of 139.50 GiB, and a "New Partition #2" of 512 MiB, assuming you shrunk the Windows partition by the same amount described above.
Assuming everything looks okay, click on the green tick icon at the top
to "Apply All Operations", and then after double checking what will be
done confirm you are sure you want to Apply the changes. It should
fairly quickly tell you that all operations completed successfully, and
then you will have two more partitions: /dev/nvme0n1p7 (139.50GiB) and
/dev/nvme0n1p8 (512MiB). One more:
sudo parted /dev/nvme0n1 print
should confirm those partitions exist.
Now it is possible to set up the encrypted container:
sudo cryptsetup luksFormat --cipher aes-xts-plain --key-size 512 \
--hash sha512 --iter-time 2000 /dev/nvme0n1p7
being extra careful to choose the larger newly created partition (since otherwise it will overwrite data you want to keep). When you are certain it is the newly created partition you have specified type "YES" to overwrite it. (Here 512 bit means AES256 due to the way the XTS cipher mode works.)
You need to enter a passphrase for the container (twice), without which the container will be inaccessible. Be sure it is a passphrase you will remember, but will not be easily guessed by someone else. Long but memorable is good :-) (I have a feeling that the direct way that the passphrase turns into the encryption key means it is not possible to change the passphrase later; so choose carefully.)
The format phase completes pretty quickly, as it is effectively a "quick format". Before carrying on, we want to open the LUKS volume, and then overwrite the volume with zeros (forcing encrypted data to be written out to the underlying file system).
To open the volume:
sudo cryptsetup luksOpen /dev/nvme0n1p7 dell
and then enter your passphrase again (hopefully you still remember it!).
It is useful to use the pv (pipe viewer) tool to be able to monitor
the progress of overwriting the disk. That is not available on the
Live CD, but is available in the Ubuntu repository. Ideally one
would just turn on the Wifi, and install
it,
but for some reason the live CD was unable to connect to my Wifi
(yet the Install mode was able to connect fine). So I downloaded
the package manually,
picking the latest version which was before the 2016-04-xx release
date of Ubuntu 16.04 LTS. Then copied that file over on another
USB drive, and installed it with:
cd /media/ubuntu/... # USB drive auto-mounted location
sudo dpkg --install pv_1.5.7-2_amd64.deb
Then zero the volume in a root shell
pv -tprebB 16m /dev/zero | sudo dd bs=16M of=/dev/mapper/dell
where the pv options mean:
-t: enable timer mode-p: enable progress bar-r: enable the rate counter-e: enable guessing the estimated completion time-b: turn on bytes copied counter-B 16m: transfer in 16MB chunks (for efficiency)
and the target device is the open LUKS volume (not the raw
underlying partition), so that all the writes go through LUKS,
and thus get encrypted. Note that the 16m for pv is in lower
case and the 16M for dd is in upper case, to make both tools
happy.
On my system this seemed to proceed at about 250MiB/s, and thus would take several minutes but under an hour to complete; I went off to do something else at this point. It was done by the time I came back, reporting that it finished in under 9 minutes -- I guess there are some advantages to a fast processor, fast storage, and small disks. (In my day job I am more used to many terrabyte RAID arrays, which take literally days to initialise.)
Once that is done it is possible to set up LVM within the encrypted volume, with two logical volumes -- one for root, and one for swap. I allowed 2GiB for swap, which should be plenty to allow RAM overcommit, but if the system ends up swapping beyond that there's probably something badly wrong. (If you want to hibernate the system you would need more swap space I think.) To set up LVM:
sudo pvcreate /dev/mapper/dell
sudo vgcreate vg /dev/mapper/dell
sudo lvcreate -n root -L 137.5G vg
sudo lvcreate -n swap -l 511 vg # 2G - 1 extent: remaining extents
sudo vgdisplay -v vg
And then initialise those two new filesystems:
sudo mkfs.ext4 -b 4096 -j -L root -v /dev/vg/root
sudo mkswap -L swap /dev/vg/swap
Installing Ubuntu 16.04 LTS
Now it is possible to launch the Ubuntu Linux 16.04 LTS installer from within the Live CD, by double clicking on the "Install Ubuntu 16.0.4.1 LTS" icon (top left).
The Ubuntu 16.04 LTS installer asks for your install language, and your Wifi network/credentials -- the joys of computers without Ethernet ports bundled! Fortunately the password entered there actually works; it is unclear why the same password given to Network Manager does not work.
The installer then offers to let you install third party modules, but only if you turn off Secure Boot -- ah the joys of kernel/module signing :-(. I declined the third party modules, since my Wifi clearly works without third party modules and generally the Dell XPS Ubuntu Linux install guides suggest that by Ubuntu 16.04 LTS pretty much everything works "out of the box"; I would rather have Secure Boot enabled, to make Windows/Linux dual booting easier.
At the Installation Type (install disk) prompt, you need to choose "Something else", so that you can tell the installer about the partitions that you have carefully prepared for it. You want to select:
/dev/mapper/vg-rootas an ext4 file system, to use for/; which the installer is allowed to format (that seemed to be necessary before "Install Now" was activated)./dev/mapper/vg-swapas a swap disk, which does not need to be initialised./dev/nvme0n1p8(the unencrypted partition made earlier) as ext2, to be/boot, which the installer is allowed to format. (I had earlier tried to make that Fat32 on the assumption the UEFI booting will need a Fat32 partition to boot from, but the Ubuntu Linux installer will not accept/booton a Fat32 partition, so I changed it to ext2 based on the advice that journalling cost 30MB. In hindsight there is already an EFI boot partition on the disk, which is FAT32, so/bootjust needs to be readable bygrubwithout unlocking the encryption.)
To set the use for each one highlight the partition then click "Change". Be very careful not to select any of the NTFS partitions, or the raw partition underneath the LUKS encryption.
I left the "Device for boot loader installation" set to /dev/dm-0``
based on [this post recommending *not* overwriting the MBR with
grub` when booting via
UEFI](http://askubuntu.com/questions/666631/how-can-i-dual-boot-windows-10-and-ubuntu-on-a-uefi-hp-notebook).
(It also contains the advice to disable Windows Fast Startup, and
an explanation of how it works and why to disable it -- detail I
wish I had realised before getting this far into the Linux install.)
/dev/dm-0 is /dev/mapper/dell, ie inside the LUKS encrypted
container. That seems a safe enough place to install an MBR record,
but unlikely to have any effect on booting. (If necessary to
reinstall grub to the MBR, this post explains how to convert to
UEFI booting with
grub.)
After double checking the disk partitions, click on "Install Now". It
will prompt that it is going to overwrite the partitions you selected
to format (/dev/mapper/vg-root for /; /dev/mapper/vg-swap; and
/dev/nvme0n1p8 for /boot). Check they are the partitions you intended
and then click on "Continue". (In my case it also needed to update the
partition table to convert partition #8, ie /dev/nvme0n1p8 from fat32
to ext2, as I had set originally created it as fat32 earlier due to
a misunderstanding. Actually that fat32 partition is the EFI partition
which needs to end up mounted at /boot/efi, and the Ubuntu installer
seems to do that automatically.)
There is a pause while the disk is updated, and then it asks for your location (eg, for the timezone), which to my surprise guessed correctly (my assumption is based on network-driven geolocation). It then asked for the keyboard layout, before asking for the user account to create. There is an option to "encrypt home folder" which I did not select, because the entire install is on an encrypted disk.
Once you click "Continue" after the user account prompt, Ubuntu Linux 16.04 LTS will actually install. Complete with the usual "captive audience" in-installation advertising. At least some of the install appears to be downloaded from the Internet (rather than coming off the install CD).
One the installer finishes, click on "Continue Testing" to drop back to the live environment, so as to be able to configure it to actually boot.
Configuring Ubuntu Linux 16.04 LTS to boot
After the install finishes mount all the partitions used in the
install under /mnt:
sudo mount /dev/vg/root /mnt
sudo mount /dev/nvme0n1p8 /mnt/boot
sudo mount --bind /dev /mnt/dev
and then transition into that environment, and mount some more useful virtual filesystems:
sudo chroot /mnt
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devpts devpts /dev/pts
mount /boot/efi
Within the chroot environment, create /etc/crypttab:
UUID=$(sudo blkid /dev/nvme0n1p7 | cut -f 2 -d '"')
LUKS=dell
VG=vg
echo "# TARGET SOURCE KEYFILE OPTIONS" | tee /etc/crypttab
echo "${LUKS} UUID=${UUID} none luks,retry=1,lvm=${VG}" | tee -a /etc/crypttab
# Check your work
cat /etc/crypttab
And the file /etc/initramfs-tools/conf.d/cryptroot:
echo "CRYPTROOT=target=${LUKS},source=/dev/disk/by-uuid/${UUID}" | tee /etc/initramfs-tools/conf.d/cryptroot
# Check your work
cat /etc/initramfs-tools/conf.d/cryptroot
Then run:
update-initramfs -k all -c
to update the initial ramdisk to have those boot options.
Finally edit /etc/default/grub to update the kernel command line so that
it reads:
GRUB_CMDLINE_LINUX="cryptopts=target=${LUKS},source=/dev/disk/by-uuid/${UUID},lvm=${VG}"
with those values expanded out, and the double quotes present. Eg:
echo "cryptopts=target=${LUKS},source=/dev/disk/by-uuid/${UUID},lvm=${VG}" | tee /tmp/cmdline
vi /etc/default/grub
# Search for GRUB_CMDLINE_LINUX
# r /tmp/cmdline
# and adjust into place between the double quotes
Check the values make sense with:
grep GRUB_CMDLINE_LINUX /etc/default/grub
then run:
update-grub
to update the grub boot time configuration files. (There will be some
errors about /run/lvm/lvmetad.socket connection failures, which I think
can be ignored at this point.)
For completeness, check that there is actually an "ubuntu" entry on the EFI boot partition:
ls /boot/efi/EFI/ubuntu
and that it has sensible EFI files on it, including shimx64.efi, which
is the file needed to kick start UEFI Secure Boot into grub.
At this point, if you are very lucky, Ubuntu Linux will boot automatically; if it does not, you get to boot off the Live CD again, and try to fix up the boot environment.
Unmount the partitions you mounted, and exit out of the chroot:
umount /boot/efi
umount /boot
umount /dev/pts
umount /proc
umount /sys
exit
sudo umount /mnt/dev
sudo umount /mnt
Then use the cog in the upper right to choose Shutdown -> Restart.
All going well, the system should restart into grub, which should then
have an option to boot Ubuntu. When Ubuntu starts booting it should
prompt to unlock the LUKS device ("dell" in my case), and when that
password is entered, the system should boot to a graphical login window.
In my case, to my complete surprise, this just worked the first time.
(At minimum I was expecting it to boot Windows instead, due to not having
disabled Fast Boot.)
Assuming you reach the login window, log in and check that the system is running as expected:
mount | egrep "root|boot"
swapon -s
The "swapon" output probably says /dev/dm-2 if you followed the
instructions above; you can check which logical volume that is by:
ls -l /dev/mapper | grep dm-2
and it should turn out to be vg-swap.
Assuming it all looks okay, try rebooting again to make sure that works, and also try shutting down/powering off, and powering on again.
Thanks to the Ubuntu encrypted LUKS/LVM guide for all the directions in getting this going.
Adding Windows back into the grub boot menu
The main issue that I noticed with the way that I went through the install
is after installing Ubuntu Linux 16.04 LTS, there was no longer an option
to boot Microsoft Windows 10 -- instead grub is started automatically,
despite trying to avoid overwriting the boot record.
I was able to get back into Microsoft Windows 10 by going to the "one-time boot menu" (F12 when the Dell logo is displayed), and choosing the "ubuntu" option, which then booted Windows (surprise!). (If this did not happen then it probably would have been necessary to boot the Windows recovery USB drive made earlier.)
I believe both of the issues above happened due to having left
Windows Fast Boot
enabled
when I installed Linux (because I did not realise how important it
was to turn it off until half way through the Ubuntu install. Thus
(a) Microsoft Windows 10 did not end up in the grub boot menu,
and (b) the UEFI boot menu was set to boot up Microsoft Windows.
To disable Fast Boot in Windows 10:
Go to Control Panel -> Hardware and Sound -> Power Options
Click on "Choose what the power buttons do"
Click on "Change settings that are currently unavailable"
Untick "Turn on fast startup (recommended)", so it is disabled.
Click on "Save changes" at the bottom, then exit the Control Panel.
Then shut down the computer so that Windows does a full shutdown, and powers off.
Power the computer back on again. grub should launch automatically.
Boot Ubuntu Linux and log in. Once logged in, open a terminal and run:
sudo update-grub
and grub should find the Linux boot volumes, and also find the Windows
volume automatically this time (ie, now that Fast Shutdown is turned off).
Restart again. Let grub launch. There should be a "Windows Boot
Manager" option in the grub menu. Select that and verify that
it boots Windows. (Due to the Ubuntu/grub background being
non-black, it does not look quite as elegant as when booted directly
from the Dell prompt, but it does work.)
Restart again, and verify that Ubuntu Linux still boots. Power the system
off, and on again, and verify that Microsoft Windows 10 will boot from the
grub menu. Power the system off again and verify that Ubuntu Linux will
boot up.
At this point, the "dual boot" installation is basically complete. What remains is getting the two systems to share the hardware nicely... starting with the hardware clock (Windows assumes the hardware clock holds local time; Linux assumes it holds UTC :-( ). And installing/configuring all the applications in each OS. All of which is beyond the scope of this (very long) post :-)
ETA, 2017-05-13: To make Microsoft Windows 10 and Ubuntu Linux 16.04 share the hardware clock better, it helps to set Ubuntu Linux 16.04 to expect the hardware time (RTC) to be in local time, with:
timedatectl set-local-rtc 1
(You probably do not want the --adjust-system-clock argument, to set
the system time from the RTC again, unless NTP is not running, or has
not run yet.)
You can use:
timedatectl
by itself to verify the current values of the system clock, RTC, and whether the RTC is expected to be UTC or local time.
There are some issues around changes to/from daylight time when you
store the RTC in local time rather than UTC -- in particular possibly
neither or both operating systems will try to make the change --
but at least in theory they should be straightened out fairly soon
after each OS boot, by using NTP to set the time from the Internet.
The same is also true of changes in timezone around travelling --
if the timezone is different in Ubuntu Linux 16.04 and Microsoft
Windows 10, non-trivial confusion will result. There is a warning
when you run timedatectl by itself, and the RTC is set to local
time, for this reason. For now this feels managable to me, given
I basically only use Ubuntu Linux 16.04 when travelling, and in the
unlikely event I do boot Microsoft Windows 10 while travelling,
hopefully I will remember to change the timezone there as well.
(It is also possible to make Microsoft Windows 10 expect the RTC to be storing UTC, using a registry key setting, but apparently the Windows Time Service will still write Local Time to the RTC despite that setting, so you have to disable the Windows time service as well. That all feels even more fragile than storing local time in the RTC.)
ETA, 2017-07-16: After upgrading to the "Windows 10 Creators Update", I installed the Bash on Ubuntu on Windows feature, which gives me Ubuntu 16.04 functionality on both sides of the dual boot system :-)