Two Pass Import in Apple Aperture

I have taken a lot of photos over the years. Since I started with digital photography fairly early (being an IT geek encourages that) almost all of my photos are digital. Back when I started there were relatively few programs for efficiently dealing with large numbers of digital photos -- especially on Linux (which I was using for Desktops and servers back in 2000). So I built my own scripts (in Perl) using Image Magick to convert the photos for the web, and would sort through the photos with an image preview program cataloguing them into folders based on date and location/photo project.

Fast forwarding to now, I want to be able to use Apple Aperture (and/or Lightroom, but I'm starting with Aperture since I "bought" Apple Aperture, while Lightroom is "only rented"). Both to catalogue the images, and also to help with image editing and refinement. But still fit in with some of my existing workflow, as well as leave me the option to choose something else later on. This post documents my attempt to refit Aperture to the workflow I want. (It will probably get updated a few times as I streamline the process.)

My requirements

My aims are:

Keep the images accessible in the file system, in directories organised by year, month, and project, not hidden in some application's database.
Have the image filenames be "shell scripting" friendly (so, eg, all in lower case, without spaces in the filenames)
Give each image a unique filename, even through multiple cameras, but also be able to track them back to the original on-camera image (eg, to avoid duplicates)
Process through the images efficiently without having to "touch" each one individually by hand
Track older images and new images in a similar manner
Be able to deal with both in-camera JPEGs and RAW photos, but keep them separate (most of my earlier cameras and some of my current cameras will only do JPEG; some others will do RAW but I usually shoot in RAW+JPEG mode).
Avoid storing multiple copies of the same image (either on disk, or in the Aperture database) -- particularly since I had to upgrade my computer in part due to running out of disk space due to all the photos...
But store a backup of each image on a separate drive ASAP when copying images off the card (I eventually purge the older "on import" backups once I've sorted through all the photos and know I have multiple backups of the good ones and do not want the rest)
Be able to deal with importing multiple days/months of cards in one go without getting confused.

After watching a few videos on using Apple Aperture and looking at a few other photographer's workflows, and a bit of experimentation, I think I've found a way to make Aperture do something close to what I want, with a bit of external scripting help.

The basic idea is to do a "first import" with Aperture that dumps all the photos into one unsorted directory and makes backups, cull out the unwanted ones (eg, random "notes" snapshots, or "I guess I did accidentally bump the shutter then" unwanted ones) and remove them competely. Then drop the remaining new photos from Aperture, and reimport them again a permanent set of filenames and folder structures.

A key feature is that all photos are kept as referenced masters (ie outside the Aperture library), to avoid unnecessary copies and keep them in a directory structure that can be moved around.

When considering this process it is worth remembering that mostly I only take photos in one location or for one project on each day -- so the date pretty directly maps to a location of project. Only rarely do I have to manually split photos amongst multiple locations projects on the same day.

Two-pass import with Aperture

I keep my photos in /photos. For the new photos I have a folder structure under /photos/org (org for "original"). Below that is:

unsorted -- freshly imported
A folder per year, with a folder per month in it, each with:
- A folder per location (eg, "wellington") or project (eg "photowalk")
- An unsorted folder, for a new import for that month

The /photos/org/unsorted folder is the destination for the first pass of the import, and the source for the second pass of the import. Because Aperture does not want to know about that location directly in its import dialogue, so I've placed the "unsorted" folder in ~/Pictures and added a symlnk to ~/Pictures/unsorted into /photos/org/unsorted for ease of reference:

mkdir ~/Pictures/unsorted
ln -s ~/Pictures/unsorted /photos/org

so /photos/org/unsorted and ~/Pictures/unsorted are the same below.

First pass

Open Aperture, and make sure the Aperture Trash is empty. Check that the /photos/org/unsorted directory is also empty. If not, tidy up in Aperture and/or the file system before starting a new import.
Insert SD card into laptop, and select "Import". Choose a "new project" and (arbitrarily) call it "unsorted" (the images won't actually stay there, see below).
Choose to copy the images to /photos/org and put them in a folder with a custom name: unsorted. Let them keep their original names. (This creates, eg, /photos/org/unsorted/IMG_ABCD.JPG.)
Run the AppleScript ImportProjectDate.scpt on import (where ImportProjectDate.scpt is the result of pasting this AppleScript text into the AppleScript editor and saving it to ImportProjectDate.scpt; I keep it in ~/Pictures/Scripts/ImportProjectDate.scpt; thanks to F. Caggiano for the original AppleScript -- mine is just a slight tweak). That AppleScript creates a folder structure for YYYY/MM, and puts projects for each date (YYYY-MM-DD) inside that structure (which is why the project name at 1. above does not matter -- once it's empty again it gets removed).
Make a backup of the images on import, to an external drive, under backup/photos/all, with a YYYY/MM/DD structure, so eg, the final file would be backup/photos/all/YYYY/MM/DD/IMG_ABCD.JPG. This is my "last resort" backup copy if I accidentally delete an image I later decide I wanted to use.
Start the import. Once the import is complete the AppleScript will automatically move the images into day-by-day projects in a YYYY/MM directory tree.
Make a "first pass" through those day by day projects, deleting anything that you really do not want to keep (eg, accidental shutter releases, visual notes of signs, terribly out of focus/blurry shots, etc). Usually I find that some days on some my "carry always" cameras (like my phone) are entirely those so the whole day can be deleted.
Empty the Aperture trash, and tell it to move those "really do not want" photos to the System Trash (ie, out of /photos/org/unsorted so they won't be reimported).

Second Pass

Make sure the Aperture Trash is empty (see last item of First pass!).
Delete from Aperture all the images imported in the First Pass.
Empty the Aperture trash, this time telling Aperture not to move the photos to the System Trash (ie, leave them in /photos/org/unsorted).
Rename all the files to lower case, and fix their file permissions, with a shell script:
```
cd /photos/org/unsorted && tidyphoto
```
(where tidyphoto is the shell script linked to just before). This avoids all the files ending up in uppercase forever, and thus being difficult to manage from the shell. (Linux has mount options to squash the FAT case down to lower case, which used to be the automount default, but are not any longer; I have not found anything equivalent for OS X.)
Return to Aperture and select Import again, this time from ~/Pictures/unsorted (which is a symlink to /photos/org/unsorted, but easier to find). Import them into an "unsorted" project (which will be discarded by the AppleScript). Choose to move the photos to "Image Year/Month/Custom Name", under /photo/org and enter "unsorted" for the custom name. (This results in them being stored in /photo/org/YYYY/MM/unsorted.)
Select to rename the files at this stage, to "Image Date then Original Name" (a custom present that is works out to be YYYY-MM-DD_img_ABCD.jpg). Choose to rename the original files. Disable the backups on this import.
Make sure the AppleScript ImportProjectDate.scpt is selected to run at the end of the import (as above in the first pass).
Start the import and let it create the YYYY-MM-DD projects.
Rename the projects in Aperture to YYYY-MM_TOPIC, where the YYYY-MM is the year and month of the date, and the TOPIC is the location the photos were taken or the event they were taken for. If a given days photos need to be split up create a second Project in Aperture, and move the photos that need to be separate into that Project.
Ensure the project is selected in the Aperture Library, then go to File -> Relocate Originals for Project... to change the storage location from YYYY/MM/unsorted to a location reflecting the actual topic. Choose /photo/org in the browser, and "Image Year/Month/Custom Name" for the subfolder name. Enter the lowercase version of the TOPIC as the custom name (replacing spaces with underscores, etc, to make a useful directory name). Choose "Original File Name" (ie, the filename that was created on this second pass import, with YYYY-MM-DD_img_ABCD.jpg). And let it Relocate Originals.
Move the now renamed and relocated project out of ZZ_Lastimport into a more sensible folder (eg, All_New/2nd_Pass) for further editing.

When you are done with this process, the ZZ_Lastimport folder should be empty of projects, just containing unneeded folders (which can be trashed from Aperture), and /photos/org/unsorted should have no files left in it. Which means you are ready to move on to another memory card.

(The directory can be checked from the shell or Finder; unfortunately there doesn't seem to be any way built into Aperture to check if it still references files in that directory, hence the suggestion to make sure it is empty before starting this process and empty after finishing.)

Handling RAW/JPEG pairs

Aperture has the ability to handle RAW and JPEG pairs in a variety of ways. For my work flow I wanted to keep the JPEG files in /photos/org and the RAW files in /photos/raw, so that I can optionally just have the JPEG files online (eg, on my older laptop). Many of my RAW/JPEG pairs are also HDR sets of three images, which I want to keep separately again (in /photos/hdr) since on any space constrained system I wouldn't want to keep any of the RAW HDR parts online -- and I only want the JPEG of the "correct" exposure online. (Other Raw+JPEG options.)

This requires some additional steps.

To make this process easier, create a symlink for the HDR unsorted directory:

mkdir ~/Pictures/hdrunsorted
ln -s ~/Pictures/hdrunsorted /photos/hdr/unsorted

(so /photos/hdr/unsorted and ~/Pictures/hdrunsorted are the same directory below).

First pass

Import as above, importing both JPEG and RAW, to get a backup made and everything in one directory on the computer (/photo/org/unsorted). Use the JPEG as the original.

Select the photos which are HDR sets, and then use File -> Relocate Originals to move them to a separate directory (/photos/hdr/unsorted) so that they are not imported on the second pass (they need special handling of the RAW/JPEG files). Even though the relocate progress bar only lists JPEG files, it moves both files.

Second pass

Import as above, but only import the JPEG files. Organise those projects appropriately.

Third pass

At this point /photos/org/unsorted should contain only the matching RAW files for the JPEG files you import on the previous step. In order to get RAW files into Aperture and matched up, we need to:

Rename the Raw files to match the new names of the JPEG files with a shell script:
```
cd /photos/org/unsorted && renamecr2withdate go
```
(where renamecr2withdate is the script linked just above; the script alone without the go parameter can be used to check what it will do)
In Aperture, select the Aperture project that has the JPEGs you want to match up (if there are multiple projects, repeat these steps for each project).
Choose Import in Aperture, making sure that Project to be matched up is the destination. Choose to store the files in /photo/raw with "Image Year/Month/Custom" as the name and make sure the folder name matches what you chose for the JPEG files (so you have /photo/org/TOPIC and /photo/raw/TOPIC). Make sure you're not renaming the files (as you did that in the shell already).
Grab "RAW+JPEG Pairs" out of the Import Settings, and choose to Import "Matching RAW files", including "All matching files".
Be sure to clear out the AppleScript and Backup Location, as this is a later import, and it is one project at a time anyway. Start the import.

This should move the RAW files to /photo/raw/YYYY/MM/TOPIC and leave the JPEG files in /photo/org/YYYY/MM/TOPIC.

The "Show in Finder" option will show the JPEG, as the JPEG (imported first) is the Original. You might want to select all the photos in the project and choose "Select RAW as Original" from the context menu, in which case the RAW file is what will be located with "Show in Finder".

That solves everything except the HDR files.

Managing HDR files

For HDR files I only want to import the RAW files (there is not much point trying to do high dynamic range with JPEG files that have already had their dynamic range clipped!). The RAW files were already backed up in the first pass above, so we can start the process from the second import.

Apple Aperture experts recommend stacking HDR sets to keep them tidy, and it seems best to keep the "average" exposure as the stack pick until a better one is created.

So to handle the HDR files:

Tidy up the files in /photos/hdr/unsorted:
```
cd /photos/hdr/unsorted && tidyphoto
```
Choose Import in Aperture, into a new project, importing from ~/Pictures/hdrunsorted. Choose to move the files to /photos/hdr into "Image Year/Month/Custom Name" subdirectories, and enter "unsorted" for the custom name.
Rename the files with "Image Date then Original", and rename the originals.
From "Raw+JPEG Pairs" choose "Raw files only" (no need for the JPEG files with HDR sets!)
Choose "~/Pictures/Scripts/ImportProjectDate.scpt" as the AppleScript.
Make sure no Backups will be made, and choose to import the images.
Rename the projects and relocate the originals as with the normal images (ie, into a folder like /photos/hdr/YYYY/MM/TOPIC)
In the resulting projects, go through the images selecting sets of three (all my HDR sets are sets of three), and then use Command-K (Stack -> Stack) to stack them, and if necessary select the "normal" exposure as the Stack Pick with Command-\ (Backslash; see Apple Aperture Stack keyboard shortcuts; for Magic Lantern HDR, the first shot should be the "normal" one, and automatically be the stack pick, saving a step, the next should be darker ("-") and the third lighter ("+"); see also notes on bracketing on Canon DSLRs without Magic Lantern).
When done, using Option-; (Semicolon) to close all the stacks.
The project can then be moved to the 2nd_pass category, or the photo stacks moved to an appropriate folder.

If in doubt about the HDR sets, double check with the date/time the images were taken (this can sometimes be an issue when making multiple attempts at composing a HDR shot in basically the same location).

(This does not include any HDR processing -- that can be done directly from the stack, by opening the stack and processing the photos, and then the resulting HDR can be placed back into the stack and made the stack pick.)

When done the unneeded JPEG files can be removed with:

cd /photos/hdr/unsorted
rm *.jpg

which should leave that directory empty again. (Alternatively it might be worth bringing in the matching JPEG files for the "normal" exposure, keeping those in /photos/org/ -- but there doesn't seem any point in doing so for the +/- exposures.)

Tidying up

The last remaining catch with this process is that it leaves some "unsorted" empty directories lying around in the file system, under various dates. These can be found with:

cd /photos
find . -type d -name "unsorted"

(and won't match the two helper top level "unsorted" folders, as they are actually under ~/Pictures and thus are symlinks not directories). The ones found can be removed by hand if they are empty and no longer required.

Other notes

ETA, 2014-06-15:

Sometimes Aperture will display a corrupted thumbnail. The fix is to force it to rebuild the thumbnail (example with Finder pictures). It seems to be some sort of race between importing/thumbnail building and other actions. At least some of the time simply deleting the file from Aperture, quitting Aperture, and reimporting the file is sufficient to get a new Thumbnail built. (But regenerating the previews is not sufficient.)

ETA, 2014-06-28:

Now that Apple have apparently confirmed that Aperture development has stopped (some time ago in practice; Metafilter's "Aperture is Closing" article has more links to news about the announcement). It is to be replaced by the recently announced Photos.app. In theory there will be an Aperture to "Photos for OS X" migration path, but it may well require an OS upgrade (again :-( I just upgraded my portable devices to OS X 7.1.1 so I could choose to delay upgrading to iOS 8 until that was stable/usable). Anyway it appears Aperture is not going to be a particularly good long term home for photos. (Photo.app seems likely to be very iCloud everything, which is not my aim, and less likely to focus on the local-computer Digital Asset Management part which is mostly what I wanted.)

So with the Adobe Photoshop Lightroom program becoming permanent I'm seriously considering making Lightroom my final destination (and location for managing edits). For now I'll probably still continue using Aperture to do the initial import/sort into directories described above, as a kind of "advanced Finder/Preview" prior to my final work. If nothing else, Aperture being optimised for the Retina display makes the photos look really nice at first glance -- which is a welcome reward for the chore of digging through lots of photos to find the best ones :-)