Background

Last month I set up a macOS Server on an old MacBook Pro. Since then I have added the second external ("data") drive, and set up some file shares -- using the macOS Server Application to set up the shares. Other than deciding what to name the directories and shares it was pretty simple to set up.

Since I have been using git-annex for managing media files for a couple of years, I also wanted to set up a git-annex server as a central point for storing one of the copies of those files, which hopefully would always be online (versus the many copies on offline external drives, which are more fiddly to access).

For this system I was aiming to avoid installing the (large) Apple Developer Tools, and MacPorts (which require installing a substantial fraction of the Apple Developer Tools, to be able to build programs), so I wanted to find a "stand alone" way to install git-annex and have it work as a remote target via ssh.

The pre-built git-annex bundle for OS X includes a bunch of git and related tools within the bundle, it seemed like it should be possible -- but actually doing so, for "server" usage (eg, git annex sync and git annex copy ... initiated from another system) proved more subtle than I expected. Since there is not really a git-annex installer to look after these details on OS X, and they do not seem to be documented anywhere easily found, I am recording the steps needed for future reference.

Installing git-annex on OS X

The git-annex install page links to pre-built binaries for Mac OS X. To start, I downloaded the most recent release for OS X 10.11/El Capitan (and its signature file to use with verifying git-annex downloads).

After verifying the download (the same way as last time), I then installed git-annex by:

  • mkdir /Applications/OpenSource (if it does not already exist)

  • open git-annex.dmg

  • Dragged the git-annex.app into /Applications/OpenSource/

  • mkdir /usr/local/bin (if it does not already exist)

  • Created symlinks to key git and git-annex related programs in /usr/local/bin so they can be (easily) on the $PATH:

    cd /usr/local/bin
for FILE in git-annex git git-shell git-receive-pack git-upload-pack; do
  if [ -f "${FILE}" ]; then
    :
  else
    sudo ln -s /Applications/OpenSource/git-annex.app/Contents/MacOS/$FILE .
  fi
done

That list of programs (git-annex, git, git-shell, git-receive-pack and git-upload-pack) was determined partially experimentally (without some of those, a remotely initiated git annex sync failed with weird errors), and partly by comparing the Apple bundled "proxy" programs (which just prompt to install the Apple Developer Tools):

ewen@bethel:~$ ls /usr/bin/git* | cat
/usr/bin/git
/usr/bin/git-cvsserver
/usr/bin/git-receive-pack
/usr/bin/git-shell
/usr/bin/git-upload-archive
/usr/bin/git-upload-pack
ewen@bethel:~$

with the git-annex bundled wrapper programs:

ewen@bethel:~$ ls /Applications/OpenSource/git-annex.app/Contents/MacOS/git* | cat
/Applications/OpenSource/git-annex.app/Contents/MacOS/git
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-annex
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-annex-shell
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-annex-webapp
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-annex.MANIFEST
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-receive-pack
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-shell
/Applications/OpenSource/git-annex.app/Contents/MacOS/git-upload-pack
ewen@bethel:~$

The aim being to ensure that for each Apple-provided proxy program the git-annex bundled wrapper programs should be on the $PATH first (something we arrange in the next setup step). I assumed that because git-annex did not provide wrappers for git-cvsserver and git-upload-archive that weren't necessary to the functionality of git-annex.

Setting your $PATH

To ensure that /usr/local/bin is on your $PATH either create ~/.bashrc with this contents, or add it at a suitable place in your ~/.bashrc:

# Ensure that /usr/local/bin is in the path
if echo "$PATH" | grep "/usr/local/bin" >/dev/null 2>&1; then
  : # Already there, great!
else
  # Not present, prepend to the start of the path
  PATH="/usr/local/bin:${PATH}"
fi

Note that it is important that /usr/local/bin ends up in your path before /usr/bin, in order that (the symlinks to) git-annex wrapper programs effectively hide the Apple-provided Developer Tools proxy programs; if you do not want /usr/local/bin at the very beginning of your $PATH then at least ensure it gets inserted before /usr/bin.

Before carrying on, ensure that the basic git and git-annex binaries do actually run now:

source ~/.bashrc
hash git
hash git-annex
git --version
git-annex

You should get a git version report, and a git-annex help message, rather than Apple's prompt for you to install the Developer tools.

Now that git is working for basic things you might also want to do some common git setup:

git config --global user.email ...    # Insert your email address
git config --global user.name ...     # Insert your name

which should run without any complaints.

Creating the git-annex wrappers

The final setup step is to persuade git-annex to create the remaining wrappers it needs (which surprisingly are not files that can be copied or linked to, but actually created on "first run" if they are not present), and then ensure that these are also in /usr/local/bin so that they are now on your $PATH.

To do this:

  • Run /Applications/OpenSource/git-annex.app/Contents/MacOS/runshell to enter a shell with git-annex on the $PATH and create two wrapper programs:

    ~/.ssh/git-annex-shell
~/.ssh/git-annex-wrapper

  • Copy these wrapper programs into /usr/local/bin:

    cd /usr/local/bin
sudo cp -p ~/.ssh/git-annex-shell ~/.ssh/git-annex-wrapper .

Testing, and git-annex server usage

To test that this is working, from a client machine do:

ssh SERVER 'echo $PATH'

and make sure that /usr/local/bin appears before /usr/bin in the result echo'd out.

Then to "clone" a git-annex onto the server, with all interaction going from client to server (ie, not relying on the server being able to ssh into the client), assuming NAME is the name of your git-annex:

On the client:

cd /EXISTING/GIT/ANNEX
git bundle create /tmp/NAME.bundle --all
scp -p /tmp/NAME.bundle SERVER:/tmp

On the server:

cd /PARENT/DIRECTORY
git clone /tmp/NAME.bundle
cd NAME
git annex init '....'        # Enable git-annex, name it, eg SERVER
git remote remove origin     # Remove dependency on NAME.bundle
rm /tmp/NAME.bundle

Back to the client, test that it is working an copy content into it:

cd /EXISTING/GIT/ANNEX
git remote add SERVER SERVER:/PARENT/DIRECTORY/NAME
git anenx sync
git annex copy --to=SERVER .

Hopefully the git annex sync runs cleanly as normal without any problems; if you get any strange errors be sure to check the install steps above and make sure that all the git-related and git-annex-related programs are in /usr/local/bin -- the errors when some tools that git-annex needs are its own bundled versions and some are the Apple-provided proxy programs are very weird.

(Note that because git-annex does not use a bare repository, it is not possible to use the normal git trick of just doing a git push into a blank git init'd repository -- it is not possible to git push into a non-bare repository.)

In theory updating this setup to a later git-annex version should just be a matter of moving the git-annex.app aside, and dragging in a new version -- the symlinks should keep pointing at the current (ie, new) version, and the two wrapper scripts that are created are so simple that they should not need changing.