Apple's Time Machine software, included with macOS for about the last 10 years is a service to automatically back up a computer to one or more external drives or machines. Once configured it pretty much looks after itself, usually keeping hourly/weekly/monthly snapshots for sensible periods of time. It can even rotate the snapshots amongst multiple targets to give multiple backups -- although it really wants to see every drive around once a week, otherwise it starts to regularly complain about no backups to a given drive, even when there are several other working backups. (Which makes it a poor choice for offline, offsite, backups which are not brought back onsite again frequently; full disk clones are better for that use case.)
More recent versions of Time Machine include local snapshots, which are copies saved to the internal drive in between Time Machine snapshots to an external target -- for instance when that external target is not available. This is quite useful functionality on, eg, a laptop that is not always on its home network or connected to the external Time Machine drive. These local snapshots do take up some space on the internal drive, but Time Machine will try to ensure there is at least 10% free space on the internal drive and aim for 20% free space (below that Time Machine local snapshots are usually cycled out fairly quickly, particularly if you do something that needs more disk space).
On my older MacBook Pro, the internal SSD (large, but not gigantic, for the time when it was bought, years ago) has been "nearly full" for a long time, so I have been regularly looking for things taking up space that that do not need to be on the internal hard drive. In one of these explorations I found that while Time Machine's main local snapshot directory was tiny:
ewen@ashram:~$ sudo du -sm /.MobileBackups
1 /.MobileBackups
ewen@ashram:~$
as expected with an almost full drive causing the snapshots to be expired rapidly, there was another parallel directory which was surprisingly big:
ewen@ashram:~$ sudo du -sm /.MobileBackups.trash/
21448 /.MobileBackups.trash/
ewen@ashram:~$
(21.5GB -- approximately 2-3 times the free space on the drive). When
I looked in /.MobileBackups.trash/ I found a bunch of old snapshots
from 2014 and 2016, some of which were many gigabytes each:
root@ashram:/.MobileBackups.trash# du -sm *
2468 Computer
412 MobileBackups_2016-10-22-214323
16824 MobileBackups_2016-10-24-163201
1746 MobileBackups_2016-10-26-084240
1 MobileBackups_2016-12-18-144553
1 MobileBackups_2017-02-05-125225
1 MobileBackups_2017-05-18-180448
root@ashram:/.MobileBackups.trash# du -sm Computer/*
1480 Computer/2014-06-08-213847
58 Computer/2014-06-15-122559
156 Computer/2014-06-15-162406
166 Computer/2014-06-29-183344
608 Computer/2014-07-06-151454
3 Computer/2016-10-22-174000
root@ashram:/.MobileBackups.trash#
Some searching online indicated that this was a fairly common problem (there are many other similar reports). As best I can tell what is supposed to happen is:
/.MobileBackupsis automatically managed by Time Machine to store local snapshots, and they are automatically expired as needed to try to keep the free disk space at least above 10%./MobileBackups.trashappears if for some reason Time Machine cannot remove a particular local snapshot or needs to start again (eg a local snapshot was not able to complete); in that case Time Machine will move the snapshot out of the main/.MobileBackupsdirectory into/MobileBackups.trashdirectory. The idea is that eventually whatever is locking the files in the snapshot to prevent them from being deleted will be cleared, eg, by a reboot, and then/.MobileBackups.trashwill get cleaned up. This is part of the reason for reboots being suggested as part of the resolution for Time Machine issues.
However there appears to be some scenarios where it is impossible
to remove /.MobileBackups.trash, which just leads to them gradually
accumulating over time. Some people report hundreds of gigabytes
used there. Because /.MobileBackups.trash is not the main Time
Machine Local Snapshots, it shows up as "Other" in the OS X Storage
Report -- rather than "Backups". And of course if it cannot be
deleted, it will not be automatically removed to make space when
you need more space on the drive :-(
Searching for /.MobileBackups.trash in /var/log/system.log turned up
the hint that Time Machine was trying to remove the directory, but
being rejected:
Jul 18 16:31:36 ashram com.apple.mtmd[852]: Failed to delete
/.MobileBackups.trash, error: Error Domain=NSCocoaErrorDomain
Code=513 "“.MobileBackups.trash” couldn’t be removed because you
don’t have permission to access it."
UserInfo={NSFilePath=/.MobileBackups.trash, NSUserStringVariant=(
Remove
), NSUnderlyingError=0x7feb82514860 {ErrorDomain=NSPOSIXErrorDomain
Code=1 "Operation not permitted"}}
(plus lots of "audit warning" messages about the drive being nearly full, which was the problem I first started with). There are some other references to that failure on OS X 10.11 (El Capitan), which I am running on the affected machine.
Based on various online hints I tried:
Forcing a full Time Machine backup to an external drive, which is supposed to cause it to clean up the drives (it did do a cleanup, but it was not able to remove
/.MobileBackups.trash).Disabling the Time Machine local snapshots:
sudo tmutil disablelocalwhich is supposed to remove the
/.MobileBackupsand/.MobileBackups.trashdirectories; it did remove/.MobileBackupsbut could not remove/.MobileBackups.trash.Emptying the Finder Trash (no difference to
/.MobileBackups.trash)Wait a while to see if it got automatically removed (nope!)
Forcing a full Time Machine backup to an external drive, now that the local Time Machine snapshots are turned off. That took ages to get through the prepare stage (the better part of an hour), suggesting it was rescanning everything... but it did not reduce the space usage in
/.MobileBackups.trashin the slightest.
Since I had not affected /.MobileBackups.trash at all, I then did
some more research into possible causes for why the directory might
not be removable. I found a reference suggesting file flags might be an
issue, but
searching for the schg and uchg flags did not turn up anything:
sudo find /.MobileBackups.trash/ -flags +schg
sudo find /.MobileBackups.trash/ -flags +uchg
(uchg is the "user immutable" flag; schg is the "system immutable"
flag). There are also xattr attributes (which I have used previously
to avoid accidental movement of directories in my home
directory),
which should be visible as "+" (attributes) or "@" (permissions) when
doing "ls -l" -- but in some quick hunting around I was not seeing
those either (eg sudo ls -leO@ CANDIDATE_DIR).
I did explictly try removing the immutable flags recursively:
sudo chflags -f -R nouchg /.MobileBackups.trash
sudo chflags -f -R noschg /.MobileBackups.trash
but that made no obvious difference.
Next, after finding a helpful guide to reclaiming space from Time Machine Local snapshots I ensured that the Local Snapshots were off, then rebooted the system:
sudo tmutil disablelocal
followed by Apple -> Restart... In theory that is supposed to free
up the /.MobileBackups.trash snapshots for deletion, and then delete
them. At least when you do another Time Machine backup -- so I forced
one of those after the system came back up again. No luck,
/.MobileBackups.trash was the same as before.
After seeing reports that /.MobileBackups.trash could be safely
removed
manually,
and having (a) two full recent Time Machine shapshots and (b) having
just rebooted with the Time Machine Local Snapshots turned off, I
decided it was worth trying to manaully remove /.MobileBackups.trash.
I did:
sudo rm -rf "/.MobileBackups.trash"
with the double quotes included to try to reduce the
footgun potential of typos
(rm -rf / is something you very rarely want to do, especially by
accident!).
That was able to remove most of the files, by continuing when it had errors, but still left hundreds of files and directories that it reported being unable to remove:
ewen@ashram:~$ sudo rm -rf "/.MobileBackups.trash"
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume/Library/Preferences/SystemConfiguration: Operation not permitted
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume/Library/Preferences: Operation not permitted
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume/Library: Operation not permitted
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume/private/var/db: Operation not permitted
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume/private/var: Operation not permitted
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume/private: Operation not permitted
rm: /.MobileBackups.trash/MobileBackups_2016-10-22-214323/Computer/2016-10-22-182406/Volume: Directory not empty
[....]
At least most of the disk space was reclaimed, with just 45MB left:
-=- cut here -=-
ewen@ashram:~$ sudo du -sm /.MobileBackups.trash/
45 /.MobileBackups.trash/
ewen@ashram:~$
-=- cut here -=-
In order to get back to a useful state I then moved that directory out of the way:
sudo mv /.MobileBackups.trash /var/tmp/mobilebackups-trash-undeleteable-2017-07-18
and rebooted my machine again to ensure everything was in a fresh start state.
When the system came back up again, I tried removing various parts of
/var/tmp/mobilebackups-trash-undeleteable-2017-07-18 with no more
success. Since the problem had followed the files rather than the
location I figured there had to be something about the files
which prevented them from being removed. So I did some more research.
The most obvious is the Time Machine Safety
Net,
which is special protections around the Time Machine snapshots to
deal with the fact that they create hard links to directories (to
conserve inodes, I assume) which can confuse rm. The recommended
approach is to use "tmutil
delete",
but while it will take a full path doing something like:
tmutil delete /var/tmp/mobilebackups-trash-undeleteable-2017-07-18/MobileBackups_2016-10-22-214323
will just fail with a report that it is an "Invalid deletion target":
ewen@ashram:/var/tmp$ sudo tmutil delete /var/tmp/mobilebackups-trash-undeleteable-2017-07-18/MobileBackups_2016-10-22-214323
/private/var/tmp/mobilebackups-trash-undeleteable-2017-07-18/MobileBackups_2016-10-22-214323: Invalid deletion target (error 22)
Total deleted: 0B
ewen@ashram:/var/tmp$
and nothing will be deleted. My guess is that it at least tries to ensure that it is inside a Time Machine backup directory.
Another approach suggested is to use Finder to delete the directory, as that has hooks to the extra cleanup magic required, so I did:
open /var/tmp
and then highlighted mobilebackups-trash-undeleteable-2017-07-18
and tried to do permanently delete it with Alt-Cmd-Delete. After
a confirmation prompt, and some file counting, that failed with:
The operation can't be completed because an unexpected error occurred (error code -8072).
deleting nothing. Explicitly changing the problem directories to be owned by me:
sudo chown -R ewen:staff /var/tmp/mobilebackups-trash-undeleteable-2017-07-18
also failed to change anything.
There is an even lower level technique to bypass the Time Machine Safety
Net,
using a helper bypass tool, which on OS X 10.11 (El Capitan) is in
"/System/Library/Extensions/TMSafetyNet.kext/Contents/Helpers/bypass".
However running the rm with the bypass tool did not get me any
further forward:
cd /var/tmp
sudo /System/Library/Extensions/TMSafetyNet.kext/Contents/Helpers/bypass rm -rf mobilebackups-trash-undeleteable-2017-07-18
failed with the same errors, leaving the whole 45MB still present.
(From what I can tell online using the bypass tool is fairly safe
if you are removing all the Time Machine snapshots, but can leave
very incomplete snapshots if you merely try to remove some snapshots --
due precisely to the directory hard links which is the reason that the
Time Machine Safety Net exists in the first place. Proceed with caution
if you are not trying to delete everything!)
More hunting for why root could not remove
files,
turned up the OS X 10.11+ (El Capitan onwards) System Integrity
Protection which adds
quite a few restrictions to what root can
do.
In particular the theory was that the file had a restricted flag on it
which means that only restricted processes, signed by Apple, would be
able to modify them.
That left me with the options of either trying to move the files
back somewhere that "tmutil delete" might be willing to deal with,
or trying to override System Integrity Protection for long enough
to remove the files. Since Time Machine had failed to delete the files,
apparently for months or years, I chose to go with the more brute
force approach of overriding System Integrity Protection for a while
so that I could clean up.
The only way to override System Integrity Protection is to boot into
System Recovery mode, and run "csrutil disable", then reboot again
to access the drive with Sytsem Integrity Protection disabled. To
do this:
Apple -> Restart...
Hold down
Cmd-Rwhen the system chimes for restarting, and/or the Apple Logo appears; you have started a Recovery Boot if the background stays black rather than showing a color backdrop prompting for your passwordWhen Recovery mode boots up, use Utilities -> Terminal to start a terminal.
In the Terminal window, run:
csrutil disableReboot the system again from the menus
When the normal boot completes and you log in, you are running without System Integrity Protection enabled -- the foot gun is now on automatic!
Having done that, OS X was happy to let me delete the left over trash:
ewen@ashram:/var/tmp$ sudo du -sm mobilebackups-trash-undeleteable-2017-07-18/
Password:
45 mobilebackups-trash-undeleteable-2017-07-18/
ewen@ashram:/var/tmp$ sudo rm -rf mobilebackups-trash-undeleteable-2017-07-18
ewen@ashram:/var/tmp$ ls mob*
ls: mob*: No such file or directory
ewen@ashram:/var/tmp$
so I had finally solved the problem I started with, leaving no
"undeleteable" files around for later. My guess is that those snapshots
happened to run at a time that captured files with restricted flags
on them, which then could not be removed (at least once Time Machine
had thrown them out of /.MobileBackups and into
/.MobileBackups.trash). But it seems unfortunate that the log
messages could not have provided more useful instructions.
All that was left was to put the system back to normal:
Boot into recovery mode again (Apple -> Restart...; hold down
Cmd-Rat the chime/Apple logo)Inside Recovery Mode, re-enable System Integrity Protection, with:
csrutil enableinside Utilities -> Termimal.
Reboot the system again from the menus.
At this point System Integrity Protection is operating normally,
which you can confirm with the "csrutil status"
command
that you can run at any time:
ewen@ashram:~$ csrutil status
System Integrity Protection status: enabled.
ewen@ashram:~$
(changes to the status can be made only in Recovery Mode).
Finally re-enable Time Machine local snaphots because on a mobile device it is a useful feature:
sudo tmutil enablelocal
and then force the first local snapshot to be made now to get the process off to an immediate start:
sudo tmutil snapshot
At which point you should have /.MobileBackups with a snapshot
or two inside it:
root@ashram:~# ls -l /.MobileBackups/Computer/
total 8
-rw-r--r-- 1 root wheel 263 18 Jul 17:37 .mtm.private.plist
drwxr-xr-x@ 3 root wheel 102 18 Jul 17:37 2017-07-18-173719
drwxr-xr-x@ 3 root wheel 102 18 Jul 17:37 2017-07-18-173758
root@ashram:~#
and if you look in the Time Machine Preferences Window you should see the line that it will create "Local snapshots as space permits".
Quite the adventure! But my system now has about three times as much free disk space as it did previously, which was definitely worth the effort.