Introduction
Back in the Before Times (tm) when international travel seemed safer, and the only way to get certain information from conferences, I bought a MacBook Air for travel, as well as some external USB drives to use as additional storage, and for macOS Time Machine backups. Because the internal drive of the MacBook Air was encrypted, Time Machine insisted the backups also be encrypted.
In the last few years the MacBook Air has not left the house, and I have not travelled much, so there is no longer a need for a "travel external drive" with additional files. (If I were to travel these days I would definitely be taking a newer laptop; and those all have larger internal drives.)
But in order to be able to retire that MacBook Air Time Machine backups off one of my network drives (to make more room for network backups of newer machines), I wanted to have more space for Time Machine backups of that older MacBook Air on those external drives, so there was more backup history available.
Since I was planning on deleting one of the older sets of Time Machine backups for that machine (on the network drive), I wanted to retain the existing backups on those external drives and expand the space available (so they would be retained for longer).
Unfortunately, due to an accident of history, the Time Machine backups and the "external file system with extra files" shared the same USB drive, with the "external file system" partition first on the USB drive. (With the benefit of hindsight: do not do this, put the Time Machine backup volume first, as it is easiest to expand that way, and hardest to move to another drive.)
So this started An Adventure, with lots of yak shaving.
Warning
If you have found this blog post while searching on how to move or
expand a time machine backup, please note that you should not attempt
this if you have not been using dd and diskutil for years. Moving
a time machine backup on a drive is complex, and fragile, and carries a
very high risk of accidental data loss.
The process described below (mostly for my own reference) is guaranteed to lose data, as it destroys at least one file system in the process.
If you are very experienced, and very careful, you might only lose the data that you intended. But if you typo something, or identify the wrong drive volume at the wrong point (they do change names part way through the steps), then it is entirely possible you will lose data you do not intend to lose. If you make a mistake you could overwrite the internal drive of your system and render it unbootable.
Make sure you have other backups of the data in question, offline, that you could fall back to if necessary. I would also highly recommend doing these steps from a spare system that does not have any critical data on it, not even on the internal drive, so that worst case you are reinstalling that spare system and restarting your time machine backups of the planned system.
In my case I had multiple external drives with time machine backup partitions on them, and I did these steps one drive at a time, making sure the expanded one was working again before moving on to the next one. Plus I also had that network time machine backup (which I did not delete until all the external drives had been successfully expanded).
If you choose to follow this guide on your own system you do so at your own risk, having been warned there is a very high risk of data loss (including accidental data loss).
I am writing this up mostly because I have off site drives which will also need these same updates as they are brought back on site. So I wanted a record of the steps.
Preparing to move the Time Machine Volume
As noted in the introduction, unfortunately the external drives with encrypted Time Machine backups had the "can be reclaimed" file system at the start of the drive.
ewen@macos:~$ diskutil list external /dev/disk9
/dev/disk9 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk9
1: EFI EFI 209.7 MB disk9s1
2: Apple_HFS travel1 600.0 GB disk9s2
3: Apple_CoreStorage mandir_tm1 399.7 GB disk9s3
4: Apple_Boot Boot OS X 134.2 MB disk9s4
ewen@macos:~$
The Apple_CoreStorage volume is the encrypted Time Machine backup volume.
Apple_CoreStorage is a volume manager built into older macOS (before
APFS was added), which supports similar Physical Volumes, Volume Groups,
and Logical Volumes as the Linux LVM (Logical Volume Manager). Importantly
it also supports layering in an encryption layer over the logical volume.
The first step was to make certain that everything on that data volume at the start of the drive was unnecessary to retain, and was held elsewhere.
In my case I did that by examining each folder on the data drive, to
determine what it was supposed to be copied from, and then confirmed
with md5sum values that I had copies of the data on other drives that
were going to be retained (including my NAS and other file servers and
laptops/desktops). Because I knew these drives were "copies of data
to travel with", once I confirmed I had identical copies of the data
I was fairly confident I had multiple additional copies of the data and
did not need the copy on the travel drives.
For the photos directory on the travel drives, I elected to make a
complete copy of the data onto my NAS, for certainty. Especially once
I determined that the travel drives had different subsets of photos
(indicating they had been updated at different times).
It is important to confirm that you do not need any of the data on the
first volume (the Apple_HFS volume) before proceeding, as the next
step will irrevocably delete that volume and the data will be unrecoverable.
Creating another Time Machine volume to the start of the disk
Having confirmed that the volume at the start of the disk (here travel1)
is not needed, and can be completely overwritten, it is now possible to
make a duplicate of the Time Machine volume.
Fortunately the volume at the start of the disk was larger than the existing Time Machine volume that I was attempting to move; so it was possible to store two copies on the disk at once. (If that was not true, you would temporarily need another larger drive to store the Time Machine Volume, and need to copy it twice -- off the drive, and then back onto the drive -- which would take even longer.)
Because I did not know if macOS would like a partition table with a gap
in the middle -- the Disk Utility utility application will not allow
creating this partition scheme -- I elected to split the first partition
into one slightly bigger than the Time Machine volume I was copying, and
another Apple_HFS file system that used up the remaining space.
To do this I used:
diskutil splitPartition /dev/disk9s2 '%Apple_CoreStorage%' mandir_tm1_new 399.8G '%Apple_HFS%' gap_1 200.2G
The % around the file system types appear to be required by
splitPartition, but not by other similar commands (see below later).
The arguments after those file system types are the "volume names",
which are required; but as best I can tell those are ignored (they
did not end up in the partition table).
I choose to make the copied Time Machine partition just slightly larger than the source partition, to make sure 100% of the source volume could be copied.
diskutil ends up creating some additional partitions of its own, including
another Apple_Boot partition, so the resulting partition table after
the splitPartition was:
ewen@macos:~$ diskutil list external /dev/disk9
/dev/disk9 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk9
1: EFI EFI 209.7 MB disk9s1
2: Apple_CoreStorage 399.8 GB disk9s2
3: Apple_Boot 134.2 MB disk9s3
4: Apple_HFS 200.1 GB disk9s4
5: Apple_CoreStorage mandir_tm1 399.7 GB disk9s5
6: Apple_Boot Boot OS X 134.2 MB disk9s6
ewen@macos:~$
Note the lack of volume names on the new partitions.
And particularly note that the partition identifiers have changed,
as this matters a great deal when we come to copy the Time Machine
volume in the next step. Before the splitPartition the Time
Machine volume at the end was disk9s3, and after the splitPartition
the Time Machine volume at the end is disk9s5.
Copying the Time Machine volume from the end of the disk to the start of the disk
To do this copy, the source volume must be unmounted, so it is not being modified during the copy of the underlying (encrypted) bits. I did this copy step on a system that did not have the password to the encrypted volume, and skipped mounting it at each step. If you are doing this on a system that has the encryption volume password, make sure you unmount it again at each step. Because each partition change will trigger a new automatic mount of the file system (as the partition table is re-read).
After the partition change (above) we have two Apple_CoreStorage
partitions, of the same size. These are Apple Core Storage physical
volumes, which contain the volume group and (encrypted) logical volume.
The one at the start of the disk is "empty" (contains stray data from the old file system), and the one at the end of the disk is the original partition we want to copy. Be very sure you know which is which, as the next step will overwrite one of those volumes -- and you want to be very sure that you are retaining the older backups, and overwriting the "junk" at the start of the disk.
Once you have identified the source disk partition and destination disk
partition, it is then possible to use dd to copy the data. For me,
this was:
sudo dd if=/dev/rdisk9s5 of=/dev/rdisk9s2 bs=33554432
Note particularly the r at the start of the disk volume names:
this tells dd to use the "raw" (unbuffered) disk volumes, which
is extremely important for copy performance. macOS, like BSD Unixes,
and traditional unixes, has distinction "raw" disk volumes (character
devices), and "buffered" disk volumes (block devices). The buffered
block devices break all input and output into page size (4kB on x86)
blocks, which is verrrrry slow, especially
when you are copying to and from the same volume.
If iostat shows large numbers of 4kB disk transactions to the
disk you have probably used the wrong version of the disk devices;
if you use the raw disk devices the bytes per transaction should
match your bs value above, assuming the disk is otherwise idle
-- which it should be as it needs to be unmounted for the copy to
have any chance of working. (On my external USB drive this makes
the difference between about 15MB/s I/O to the external drive, so
copying at about 7.5MB/s taking many hours, and about 90MB/s I/O to the
external drive, so copying at 45MB/s, taking fewer hours -- it runs
about 6x faster. So if you see it is wrong near the start of the
copy you probably want to stop the copy and start it again with the
better device filenames, as it will be faster than waiting.)
The macOS dd does not have a status=progress option of GNU dd,
but you can carefully ask dd to report its current progress by
sending it the INFO signal from another shell session:
kill -INFO PROCESS_ID
and then output will be in the dd shell window.
Wait for that copy to finish, successfully, before continuing.
Freeing up space at the end of the disk
Once you are certain you have a copy of the Time Machine backup volume at the start of the disk, the next step is to remove the partitions at the end of the disk.
In my case I turned all of the partitions at the end of the disk into free space:
diskutil eraseVolume "Free Space" "" /dev/disk9s4
diskutil eraseVolume "Free Space" "" /dev/disk9s5
but note that when you delete the old Apple_CoreStorage volume at
the end of the disk (/dev/disk9s5) the Apple_Boot volume at the
end of the disk will also be removed, which causes problems for the
next step.
For this reason you may wish to not remove the Apple_Boot
partition at the start of the drive -- /dev/disk9s3 here -- to
ensure there is at least one Apple_Boot volume left on the drive.
Assuming that works you will be able to skip the re-creating the
Apple_Boot partition step below.
Also if Apple CoreServices has mounted the physical volume at the end
(/dev/disk9s5 here), it will probably not be possible to remove it
("The target disk is in use by Core Storage as a Physical Volume; use
diskutil coreStorage verbs"). Since there is no coreStorage command
to release a physical volume, deleting the end Apple_CoreStorage volume
may have to be done on a system that does not have the password to mount
the volume (or on a Linux system, eg with parted). It appears not
to be sufficent to umount or eject the volume; Core Storage still
hangs onto the physical volume (and there does not appear to be any
Core Storage deactivate command). So removing the physical volume
at the end needs to be done on another system (eg, a macOS system
that is not able to mount it ever, or a Linux system).
Once the unnecessary volumes are removed, the partition table should look like:
ewen@macos:~$ diskutil list external /dev/disk9
/dev/disk9 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk9
1: EFI EFI 209.7 MB disk9s1
2: Apple_CoreStorage 399.8 GB disk9s2
(free space) 600.2 GB -
ewen@macos:~$
with the (copied) Time Machine encrypted volume at the start of the
drive, and free space at the end of the drive. (Possibly with
an Apple_Boot partition remaining if you did not delete the first
one -- /dev/disk9s3 here -- above, to save a step.)
Labelling the partition
So I could keep track at this point I unplugged the external drive from
the macOS system, and plugged it into a Linux system, and used parted
to explicitly name the replacement Time Machine partition the same as
the now deleted Time Machine partition.
The usage is something like sudo parted /dev/sdf, then print the
partition table (print), and then name 2 mandir_tm1. Verify that it
is correct by printing the partition table again (with print), and then
quit (the partition table is automatically written by parted, unlike
some other tools).
ewen@linux:~$ sudo parted /dev/sdf
[sudo] password for ewen:
GNU Parted 3.3
Using /dev/sdf
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: WD Elements 25A2 (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 20.5kB 210MB 210MB fat32 EFI System Partition boot, esp
2 210MB 400GB 400GB travel1
(parted) name 2 mandir_tm1
(parted) print
Model: WD Elements 25A2 (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 20.5kB 210MB 210MB fat32 EFI System Partition boot, esp
2 210MB 400GB 400GB mandir_tm1
(parted) quit
ewen@linux:~$
Disconnect the drive from the Linux system, and unplug it into the macOS system. With luck at this point the encrypted Time Machine volume will automatically mount (if you plug it into a system that has the volume password for the encryption saved).
Recreate the Apple_Boot partition at the end of the drive
The next step (expanding the Apple_CoreStorage volume) can only be
done (a) if there is an Apple_Boot partition after it, and (b) if
the volume is mounted. If you deleted it in the steps above (as I
did the first time through, by accident), it needs to be recreated.
If the Apple_Boot volume is missing you get an obscure error
message
Error: -69722: You can't perform this resize unless it has a booter (target partition is probably too small)
The problem is not that the volume is too small; the problem is that
the Apple_Boot volume is used as part of the Apple_CoreStorage expansion
task (!!) and if it is missing the size for the expansion cannot be
judged.
So the next step is to recreate the Apple_Boot partition, if you
deleted it above. (It might be possible just to skip deleting the
Apple_Boot partition after the first Apple_CoreStorage volume,
and avoid this problem.)
To recreate the Apple_Boot volume, it is easiest to use a third
party disk tool like gdisk, which can be installed from MacPorts
with:
sudo port install gptfdisk
And then you need to create a partition after the Apple_CoreStorage
partition, which is type Apple_Boot (AB00) and 1269536 in size:
ewen@macos:~$ sudo gdisk /dev/disk9
GPT fdisk (gdisk) version 1.0.9
Warning: Devices opened with shared lock will not have their
partition table automatically reloaded!
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): p
Disk /dev/disk9: 1953458176 sectors, 931.5 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 42717690-01FF-4AA2-9C73-FA985D60E05C
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953458142
Partitions will be aligned on 8-sector boundaries
Total free space is 1172189141 sectors (558.9 GiB)
Number Start (sector) End (sector) Size Code Name
1 40 409639 200.0 MiB EF00 EFI System Partition
2 409640 781269007 372.3 GiB AF05 mandir_tm1
Command (? for help): n
Partition number (3-128, default 3): 3
First sector (34-1953458142, default = 781269008) or {+-}size{KMGTP}:
Last sector (781269008-1953458142, default = 1953458135) or {+-}size{KMGTP}: +1269536
Current type is AF00 (Apple HFS/HFS+)
Hex code or GUID (L to show codes, Enter = AF00): AB00
Changed type of partition to 'Recovery HD'
Command (? for help): p
Disk /dev/disk9: 1953458176 sectors, 931.5 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 42717690-01FF-4AA2-9C73-FA985D60E05C
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953458142
Partitions will be aligned on 8-sector boundaries
Total free space is 1170919605 sectors (558.3 GiB)
Number Start (sector) End (sector) Size Code Name
1 40 409639 200.0 MiB EF00 EFI System Partition
2 409640 781269007 372.3 GiB AF05 mandir_tm1
3 781269008 782538543 619.9 MiB AB00 Recovery HD
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/disk9.
Warning: Devices opened with shared lock will not have their
partition table automatically reloaded!
Warning: The kernel may continue to use old or deleted partitions.
You should reboot or remove the drive.
The operation has completed successfully.
ewen@macos:~$
Once that was done, I ejected the drive, unplugged it, and plugged it
back in again, to force the macOS kernel to refresh its view of the drive.
(In theory because we just added a partition it should have been safe,
but since the Apple_Boot partition was necessary for the next step
it seemed best to have the kernel view of the drive up to date.)
Expanding the Time Machine volume (now at the start of the disk)
In order to expand the Time Machine volume it must be mounted, which means this step needs to be done on a macOS system. To trigger the expansion, you need to know the UUID of the inner logical volume, which can be found with:
ewen@macos:~$ diskutil cs list
CoreStorage logical volume groups (1 found)
|
+-- Logical Volume Group 60FE7879-D2AC-4B98-97E0-217FE8A45F9B
=========================================================
Name: mandir_tm1
Status: Online
Size: 399692394496 B (399.7 GB)
Free Space: 8773632 B (8.8 MB)
|
+-< Physical Volume 771FAE36-AA9D-42F7-81F3-612FFF8984EF
| ----------------------------------------------------
| Index: 0
| Disk: disk9s3
| Status: Online
| Size: 399692394496 B (399.7 GB)
|
+-> Logical Volume Family 55582F40-CF74-4623-BA38-079B93EBB70D
----------------------------------------------------------
Encryption Type: AES-XTS
Encryption Status: Locked
Conversion Status: Complete
High Level Queries: Fully Secure
| Passphrase Required
| Accepts New Users
| Has Visible Users
| Has Volume Key
|
+-> Logical Volume 928AAEE1-26FB-4995-85DC-D456188D2A92
---------------------------------------------------
Disk: -none-
Status: Locked
Size (Total): 399331295232 B (399.3 GB)
Revertible: No
LV Name: mandir_tm1
Content Hint: Apple_HFS
ewen@macos:~$
It is the last UUID, of the inner volume that you need.
Then the entire Apple_CoreStorage "stack" (physical volume, volume
group, encrypted logical volume, inner volume) can be expanded to
maximum size with:
diskutil cs resizeStack 928AAEE1-26FB-4995-85DC-D456188D2A92 0g
where 0g is magic value which means "maximum size possible", ie use
the remainder of the disk. Which is what I wanted.
That expansion will do a full fsck of the inner volume, which takes
quite a while (especially for a Time Machine volume that has hard
links to directories!). But once it is successfully checked it will
then expand the physical volume, volume group, logical volume, and
file system for you:
ewen@macos:~$ diskutil cs resizeStack 928AAEE1-26FB-4995-85DC-D456188D2A92 0g
The Core Storage Logical Volume UUID is 928AAEE1-26FB-4995-85DC-D456188D2A92
Started CoreStorage operation
Checking prerequisites for resizing Logical-Physical volume stack
Growing Logical-Physical volume stack
Verifying file system
Volume was successfully unmounted
Performing fsck_hfs -fn -x /dev/rdisk4
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking multi-linked directories
Checking volume bitmap
Checking volume information
The volume mandir_tm1 appears to be OK
File system check exit code is 0
Restoring the original state found as mounted
Growing Core Storage Physical Volume from 399,799,996,416 to 999,310,827,520 byt
es
Copying booter
Growing disk partition
Modifying partition map
Growing Core Storage data structures
Resizing Core Storage Physical Volume structures
Resized Core Storage Physical Volume to 999,310,827,520 bytes
Growing Logical Volume
Resizing Core Storage Logical Volume structures
Resized Core Storage Logical Volume to 998,842,126,336 bytes
Growing file system
Finished CoreStorage operation
ewen@macos:~$
Once that was successful, I ejected the Time Machine volume again, disconnected the external drive, then reconnected it again, to ensure the kernel view of the drive was current. Then confirmed that the drive had been expanded correctly:
ewen@macos:~$ diskutil list external /dev/disk9
/dev/disk9 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk9
1: EFI EFI 209.7 MB disk9s1
2: Apple_CoreStorage mandir_tm1 999.3 GB disk9s2
3: Apple_Boot Recovery HD 650.0 MB disk9s4
ewen@macos:~$
ewen@macos:~$ sudo gpt -r show /dev/disk9
Password:
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 6
40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B
409640 1951778960 2 GPT part - 53746F72-6167-11AA-AA11-00306543ECAC
1952188600 1269536 3 GPT part - 426F6F74-0000-11AA-AA11-00306543ECAC
1953458136 7
1953458143 32 Sec GPT table
1953458175 1 Sec GPT header
ewen@mandir:~$
ewen@mandir:~$ diskutil cs list
CoreStorage logical volume groups (1 found)
|
+-- Logical Volume Group 60FE7879-D2AC-4B98-97E0-217FE8A45F9B
=========================================================
Name: mandir_tm1
Status: Online
Size: 999310827520 B (999.3 GB)
Free Space: 116375552 B (116.4 MB)
|
+-< Physical Volume 771FAE36-AA9D-42F7-81F3-612FFF8984EF
| ----------------------------------------------------
| Index: 0
| Disk: disk9s2
| Status: Online
| Size: 999310827520 B (999.3 GB)
|
+-> Logical Volume Family 55582F40-CF74-4623-BA38-079B93EBB70D
----------------------------------------------------------
Encryption Type: AES-XTS
Encryption Status: Unlocked
Conversion Status: Complete
High Level Queries: Fully Secure
| Passphrase Required
| Accepts New Users
| Has Visible Users
| Has Volume Key
|
+-> Logical Volume 928AAEE1-26FB-4995-85DC-D456188D2A92
---------------------------------------------------
Disk: disk10
Status: Online
Size (Total): 998842126336 B (998.8 GB)
Revertible: No
LV Name: mandir_tm1
Volume Name: mandir_tm1
Content Hint: Apple_HFS
ewen@macos:~$
Confirming it had worked
After expanding the drive, I had Time Machine do another backup
to the drive, which worked without any problems. Between the fsck
of the file system before the expansion, the volume automatically
mounting, and the Time Machine backup working, I am fairly confident
that the disk volume was copied correctly.
But as noted in the warnings above, copying the wrong drive partition at the wrong time could end up with a drive not working. And copying onto the wrong drive partition could overwrite vital data, and cause additional data loss. So considerable care is required in this process.