My own computing environment is Linux and OS X, as is probably obvious from the blog posts. I do not use Microsoft Windows myself, but sometimes I need to transfer files to people who do use Microsoft Windows.
For small files email or putting them onto my webserver is fine; for medium sized files copying onto a FAT16 or FAT32 formatted thumbdrive or external drive works pretty well (give or take the pain of sneakernet) since FAT16/FAT32 are pretty widely supported, including by Linux, OS X and Microsoft Windows. But for large files none of those are practical. Where "large" these days means "over 4GiB", particularly "significantly over 4GiB". The problem is that FAT16/FAT32 files are limited to 4GiB even with Large File Support.
To transfer files larger than 4GiB from OS X to Microsoft Windows requires either:
Splitting the files into 4GiB chunks; or
Using ExFat, which is supported for reading and writing on OS X and Microsoft Windows (since OS X 10.6.5, and Microsoft Windows XP SP2) and allows files of at least 512 TiB in size. But it is patented by Microsoft and has been the subject of multiple law suits, which makes it less useful as an exchange format since it is less widely supported. (However it is used by a number of modern still/video cameras due to lack of viable alternative formats, so that is probably one reason why Apple supports it in OS X.)
Using NTFS which is the native format on Microsoft Windows. It supports files up to at least 16 TiB in size (larger in NTFS5). OS X includes read only NTFS support in the base OS, but not NTFS write support. Which makes transfer from Microsoft Windows to OS X trivial, but transfer in the other direction impossible without third party drivers.
Copying the file over a local LAN connection using some sort of file sharing, or temporary web server. (Since OS X and Windows support SMB/CIFS, the native Windows File Sharing, I did try a network share copy some months back, but at least trying to do it as a guest user seemed to keep hanging at the start of the transfer with no obvious errors -- so I gave up. Plus this requires both computers to be in the same location, and remain connected for the duration of the transfer, so it is not that convenient for ad-hoc copies of large amounts of data.)
Since I needed to transfer about 200 GiB including several large files (in the 10 GiB to 100 GiB range; mostly raw video camera footage), I decided that the best option to transfer them would be buy another external drive to use for the transfer (and then use again later for general backups). I purchased a Western Digital Elements 2TB (2.5") external drive, since it was relatively inexpensive -- and large enough to be useful for backups later.
When I plugged the out-of-the-box external drive in, OS X auto-mounted
it as /Volumes/Elements (the default volume label on all WD Elements
drives). Using mount in a Terminal, I could find the device
associated with it:
/dev/disk4s1 on /Volumes/Elements (ntfs, local, nodev, nosuid, read-only, noowners)
which is /dev/disk4s1 on my system at present, due to the other
drives that have been plugged in.
I could also see that the drive was factory formatted with NTFS, as well as that it was mounted read-only by the default OS X drivers. Years ago all these external drives came formatted with FAT16 or FAT32 depending on their size, but presumably the manufacturers now recognise that people buy larger external drives to put large files on them so automatically format them to work with larger files on Microsoft Windows systes. (Leaving Linux and OS X users to reformat them, which I've done routinely for years anyway. Although it appears Seagate provide a NTFS read-write driver for OS X to work with their drives.)
Because it was a brand new drive, and I had been meaning to try out writing to NTFS from OS X for ages to see if it was more stable now, I decided to leave the drive as NTFS for now and install a third party driver to enable writing to NTFS, rather than reformatting it as ExFat to do the data transfer.
Tuxera develop NTFS-3G, an Open Source read-write NTFS driver implementation, which is available for a large number of platforms, including most Linux distributions (see the NTFS-3G release history). For ease of use on OS X, they offer a commercial bundle of Tuxera NTFS for Mac with what looks like fairly complete integration -- although it looks like the System Integrity Protection feature built into the up-coming OS X El Capitan (10.11) release makes that bundled approach not work. (Of note, the commerical bundle includes a caching layer which makes disk access much faster.)
However since I only had a one-off need to write to NTFS, and do not mind getting my hands dirty at the command line, I chose to work with the Open Source version, using MacPorts to install the necessary bits. The slower disk access was not a major concern as I was going to be starting the data copy and leaving it to do its own thing anyway -- simply due to the volume of data to copy.
The first step is to use MacPorts to install the Open Source ntfs-3g
version:
sudo port install ntfs-3g
which will also install the osxfuse
port. osxfuse is an OSX port of the Linux File System In
Userspace
interface, used as an adapter between the operating system file
system access interface and the ntfs-3g file system support
application. (See also the osxfuse
Wiki, and osxfuse
FAQ.)
After that, the next step is to unmount (but not eject) the external
drive, so that it is available to be used with the ntfs-3g support:
sudo umount /Volumes/Elements/
To test that the ntfs-3g support was working, my first thought was to
change the volumne label from Elements to something more recognisable:
sudo ntfslabel /dev/disk4s1 ewen_2tb
but that failed immediately with an error that the drive had not been cleanly unmounted:
ewen@ashram:~$ sudo ntfslabel /dev/disk4s1 ewen_2tb
The disk contains an unclean file system (0, 1).
Failed to mount '/dev/disk4s1': Operation not supported on socket
Access is denied because the NTFS journal file is unclean. Choices are:
A) Shutdown Windows properly.
B) Click the 'Safely Remove Hardware' icon in the Windows taskbar
notification area before disconnecting the device.
C) Use 'Eject' from Windows Explorer to safely remove the device.
D) If you ran chkdsk previously then boot Windows again which will
automatically initialize the journal.
E) Submit 'force' option (WARNING: This solution it not recommended).
F) ntfsmount: Mount the volume read-only by using the 'ro' mount option.
ewen@ashram:~$
and since it was literally fresh out of the retail packaging, and had only ever been mounted (read-only) on by OS X (once), I assume that it shipped from the factory like that -- uncleanly unmounted. Presumably that's faster for the factory, and Microsoft Windows systems auto-check the drive on first connection and clear up that "unclean mounting" option.
Since I do not have any Microsoft Windows systems, none of the first
four options are of any use to me, and the last option of mounting it
read-only defeats the purpose of using ntfs-3g :-) I also did not
want to force the mount, because I wanted a higher chance that the
files would be readable on the Microsoft Windows side. Fortunately
there is a solution to fix simple issues on NTFS systems included
within ntfs-3g:
ewen@ashram:~$ sudo ntfsfix -d /dev/disk4s1
Mounting volume... The disk contains an unclean file system (0, 1).
FAILED
Attempting to correct errors...
Processing $MFT and $MFTMirr...
Reading $MFT... OK
Reading $MFTMirr... OK
Comparing $MFTMirr to $MFT... OK
Processing of $MFT and $MFTMirr completed successfully.
Setting required flags on partition... OK
Going to empty the journal ($LogFile)... OK
Checking the alternate boot sector... FIXED
NTFS volume version is 3.1.
NTFS partition /dev/disk4s1 was processed successfully.
ewen@ashram:~$
and after doing that it was possible to change the file system label:
sudo ntfslabel /dev/disk4s1 ewen_2tb
without any problems. (I've since found that the ntfs-3g driver
defaults to attempting to recover the NTFS volume automatically on
mount if it has not been unmounted cleanly and has only these minor
issues; presumably using the same approach as ntfsfix. But ntfslabel
does not do this, hence the need for a manual fix -- it's unfortunate
they did not mention ntfsfix in the error help text!)
The next challenge was to mount the NTFS file system read-write.
Most of the guides assume that you want to automount NTFS
read-write, and
have you make various changes to your system
directories
(eg /sbin) which I would prefer not to do for a one-off usage
(and as NTFS-3G for Mac OS
X notes, you also
have to disable the System Integrity Protection feature -- with
rootless=0 -- on OS X El Capitan (10.11) to be able to make those
changes.)
It took some hunting, but I eventually found a hint for how to
mount the file NTFS file system with the MacPorts
ntfs-3g,
by running the ntfs-3g command directly (which is mentioned in
the ntfs-3g manpage, but you need to know to look at the ntfs-3g
manpage!):
mkdir /Volumes/ewen_2tb
sudo ntfs-3g /dev/disk4s1 /Volumes/ewen_2tb
after which I can see that it is mounted:
/dev/disk4s1 on /Volumes/ewen_2tb (osxfusefs, synchronous)
using the osxfuse file system adapter, and all the usual WD Elements
factory supplied junk is there:
ewen@ashram:~$ ls /Volumes/ewen_2tb/
ReadMe.pdf autorun
System Volume Information autorun.inf
WD Smartware Pro Free Trial
ewen@ashram:~$
so it appears to be working.
The synchronous mount attribute is important: as the osxfuse
NTFS-3G page
points out, there is no block cache for the direct access to
/dev/disk4s1 that the ntfs-3g program is doing -- which means
that all reads and writes go directly to the disk. (By contrast
the Tuxera NTFS for
Mac has a caching
layer which is apparently much faster, and may well justify the cost
for regular usage.)
To be somewhat more useful, it is actually more convenient to mount the
drive with your own uid (user ID) and gid (group ID):
sudo umount /Volumes/ewen_2tb
mkdir /Volumes/ewen_2tb
sudo ntfs-3g /dev/disk4s1 /Volumes/ewen_2tb -o uid=$(id -u),gid=$(id -g)
after which you can see that the files are owned by your own user:
ewen@ashram:~$ ls -l /Volumes/ewen_2tb/
total 2064
-rwxrwxrwx 1 ewen staff 1048617 8 Apr 2013 ReadMe.pdf
drwxrwxrwx 1 ewen staff 0 3 Sep 2011 System Volume Information
drwxrwxrwx 1 ewen staff 0 28 Mar 19:16 WD Smartware Pro Free Trial
drwxrwxrwx 1 ewen staff 0 28 Mar 19:16 autorun
-rwxrwxrwx 1 ewen staff 59 3 Nov 2012 autorun.inf
ewen@ashram:~$
Note that it is necessary to recreate the /Volumes/ewen_2tb mount
point after unmounting the drive, as the OS X automounter is fairly
quick to tidy it up when it is unmounted.
From that point, at least from the command line, the drive is as usable as any other drive. It does not show up as a device in OS X Finder without doing the Finder system integration steps (due to being mounted via an application, rather than as a device). You can do, eg:
open /Volumes/ewen_2tb
and a Finder window to navigate the directories, but for some reason it
appears that the files do not show in that view (I am not sure why,
possibly just related to not having rebooted); the files are visible
from the Terminal command line, and, eg, doing an open ReadMe.pdf
does show the content of the file.
Copying files from the command line works just as normal, eg, with
rsync. And while the write performance certainly is not stunning,
copying from a HFS+ formatted USB2 external drive onto a NTFS formatted
USB3 external drive happens respectably quickly -- around 20MB/s --
for large one-off transfers that you set copying and go away to do
something else. (20MB/s is around 40-50% of the theoritical maximum
speed of USB2 exteral drives anyway; and means around 2-3 hours to copy
200 GiB versus a minimum of about 1.25-1.5 hours at USB2 maximum speed.)