git-annex on the Synology DS216+

After running into problems trying to get git-annex to run on an SMB share on my Synology DS216+, prompted by the git-annex author, and an example with an earlier Synology NAS I decided to install the standalone version of git-annex directly on my Synology DS216+.

My approach was similar to the earlier "Synology NAS and git annex" tip, but the DS216+ uses an x86_64 CPU:

ewen@nas01:/$ grep "model name" /proc/cpuinfo | uniq
model name  : Intel(R) Celeron(R) CPU  N3060  @ 1.60GHz
ewen@nas01:/$

and I chose a slightly different approach to getting everything working, in part based on my experience setting up the standalone git-annex on a Mac OS X server. I am using Synology DSM "DSM 6.1.1-15101 Update 4", which is the latest release as I write (released 2017-05-25).

To install git-annex:

In the Synology web interface (DSM) enable the "SSH Service", in Control Panel -> Terminal, by ticking "Enable SSH Service", and verify that you can ssh to your Synology NAS. Only accounts in the administrators group can use the ssh service, so you will need to create an administrator account to use if you do not already have one. (If your Synology NAS is exposed to the Internet directly now would be a very good time to ensure you have a strong password on the account; mine is behind a separate firewall.)
In the Synology web interface (DSM) go to the Package Center and search for "Git Server" (git) from Synology and install that package. It should install in a few seconds, and currently appears to install git 2.8.0:
```
  ewen@nas01:/$ git --version
  git version 2.8.0
  ewen@nas01:/$
```
which while not current (eg my laptop has git 2.13.0), is only about a year old. It is a symlink (in /usr/bin/git) into the Git package in /var/packages/Git/target/bin/git.

Verify that you can now reach the necessary parts of the git package:

for FILE in git git-shell git-receive-pack git-upload-pack; do
    which "${FILE}"
done

should produce something like:

/bin/git
/bin/git-shell
/bin/git-receive-pack
/bin/git-upload-pack

Download the latest git-annex standalone x86-64 tarball, and gpg signature
Verify the git-annex gpg signature (as with previous installs):
```
gpg --verify git-annex-standalone-amd64.tar.gz.sig
```
which should report a "Good signature" from the "git-annex distribution signing key" (DSA key ID 89C809CB, Primary key fingerprint: 4005 5C6A FD2D 526B 2961 E78F 5EE1 DBA7 89C8 09CB).

If you have not already verified that key is the right signature key it can be verified against, eg, keys in the Debian keyring as Joey Hess is a former Debian Developer.

Once you are happy with git-annex tarball you downloaded, copy it onto the NAS somewhere suitable, eg on the NAS:

sudo mkdir /volume1/thirdparty
sudo mkdir /volume1/thirdparty/archives
sudo chown "$(id -nu):$(id -ng)" /volume1/thirdparty/archives

then from wherever you downloaded the git-annex archive:

scp -p git-annex-standalone-amd64.tar.gz* nas01.em.naos.co.nz:/volume1/thirdparty/archives/

Extract the archive on the NAS:

cd /volume1/thidparty
sudo tar -xzf archives/git-annex-standalone-amd64.tar.gz

The extracted archive is about 160MB, because of bundling all the required tools:

ewen@nas01:/volume1/thirdparty$ du -sm git-annex.linux/
161 git-annex.linux/
ewen@nas01:/volume1/thirdparty$

to make it a stand alone version (as well as static linking everything).

Symlink git-annex into /usr/local/bin so we have a common place

to reference these binaries:
```
cd /usr/local/bin
sudo ln -s /volume1/thirdparty/git-annex.linux/git-annex .
```
In a normal login shell /usr/local/bin will be on the PATH, and:
```
which git-annex
```
should print:
```
/usr/local/bin/git-annex
```
and you should be able to run git-annex by itself and have it print out the basic help text.

Unfortunately this does not work for non-interactive shells, because the Synology NAS uses the "/bin/sh" symlink to bash, which means that non-interactive shells do not process ~/.bashrc, and non-interactive shells also do not read /etc/profile (which is where /usr/local/bin/ is added to the PATH). So we have to add some more work arounds, with symlinks into /usr/bin/ later (see below).

For reference, this is my /etc/passwd entry created by the Synology NAS web interface (DSM):
```
ewen@nas01:~$ grep "$(id -un):" /etc/passwd
ewen:x:1026:100:Ewen McNeill:/var/services/homes/ewen:/bin/sh
ewen@nas01:~$
```
To fix the warning:
```
warning: /bin/sh: setlocale: LC_ALL: cannot change locale (en_US.utf8)
```
we have to pre-create the locales directory that the git-annex runshell script tries to write the locales into, with permissions that a regular user can write into, and then run git-annex once.
```
sudo mkdir /volume1/thirdparty/git-annex.linux/locales
sudo chown "$(id -un):$(id -gn)" /volume1/thirdparty/git-annex.linux/locales
```
On the Synology NAS, with the default locale:
```
ewen@nas01:~$ set | egrep "LANG|LC_ALL"
LANG=en_US.utf8
LC_ALL=en_US.utf8
ewen@nas01:~$
```
this should create:
```
ewen@nas01:~$ ls /volume1/thirdparty/git-annex.linux/locales/en_US.utf8/
LC_ADDRESS  LC_IDENTIFICATION  LC_MONETARY  LC_PAPER
LC_COLLATE  LC_MEASUREMENT     LC_NAME      LC_TELEPHONE
LC_CTYPE    LC_MESSAGES        LC_NUMERIC   LC_TIME
ewen@nas01:~$
```
And then we can revert the file permissions to root owned:
```
sudo chown -R root:root /volume1/thirdparty/git-annex.linux/locales
```
Note that it is possible to change the interactive locale by setting LANG and LC_ALL in, eg, ~/.bash_profile (but this will not work for non-interactive shells). git-annex only supports utf8 locales, but that is probably the most useful modern choice anyway. (I chose not to bother as en_US.utf8 is close enough to my usual locale -- en_NZ.utf8 -- that it did not really matter at present; the main difference would be the date format, and I do not expect to use git-annex interactively on the Synology NAS often enough for that to be an issue. I just wanted the warning message gone, as it turns up repeatedly in interactive use.)
To usefully use git-annex you probably also want to enable the "User Home" feature, so that the home directory for your user is created and you can store things like ssh keys; this also enables a per-user share via (CIFS, etc). To do this, in the Synology web interface (DSM) go to Control Panel -> User -> User Home and tick "Enable user home service", and hit Apply. This will create a /volume1/homes directory, a directory for each user, and a /var/services/homes symlink pointing at /volume1/homes so that the shell directories are reachable.

Once that is done, when you ssh into the NAS, the message about your home directory being missing:
```
Could not chdir to home directory /var/services/homes/ewen: No such
```
file or directory

should be gone, and you should arrive in your home directory at login:
```
  ewen@nas01:~$ pwd
  /var/services/homes/ewen
  ewen@nas01:~$
```
If you do have a home directory, you might also want to do some common git setup:
```
git config --global user.email ...    # Insert your email address
git config --global user.name ...     # Insert your name
```
which should run without any complaints, creating a ~/.gitconfig file with the values you supply.
Assuming you do have a user home directory you can usefully run the next step to have git-annex auto-generate a couple of necessary helper scripts in ${HOME}/.ssh/ -- which cannot be automatically created otherwise (but see the contents below if you want to try to create them by hand).

To create the helper scripts automatically run:
```
/volume1/thirdparty/git-annex.linux/runshell
```
which will start a new shell, with /volume1/thirdparty/git-annex.linux/bin in the "${PATH}" so you can interactively use the git-annex versions of tools (eg, for testing).

It also creates the two helper scripts that we need:
```
$ ls -l ${HOME}/.ssh
total 8
-rwxrwxrwx 1 ewen users 241 May 28 11:20 git-annex-shell
-rwxrwxrwx 1 ewen users  74 May 28 11:20 git-annex-wrapper
$
```

Since (a) these scripts are not user specific and (b) "${HOME}/.ssh" is not on the PATH by default, it is much more useful to move these scripts into, eg, /usr/local/bin/, so they are in a central location.

To do this:

cd /usr/local/bin
sudo mv "${HOME}/.ssh/git-annex-shell" .
sudo mv "${HOME}/.ssh/git-annex-wrapper" .
sudo chown root:root git-annex-shell git-annex-wrapper
sudo chmod 755 git-annex-shell git-annex-wrapper

This should give you two trivial shell scripts, which hard code the path to where you unpacked git-annex:

ewen@nas01:/usr/local/bin$ ls -l git-annex-*
-rwxr-xr-x 1 root root 241 May 28 11:20 git-annex-shell
-rwxr-xr-x 1 root root  74 May 28 11:20 git-annex-wrapper
ewen@nas01:/usr/local/bin$ cat git-annex-shell
#!/bin/sh
set -e
if [ "x$SSH_ORIGINAL_COMMAND" != "x" ]; then
exec '/volume1/thirdparty/git-annex.linux/runshell' git-annex-shell -c
"$SSH_ORIGINAL_COMMAND"
else
exec '/volume1/thirdparty/git-annex.linux/runshell' git-annex-shell -c "$@"
fi
ewen@nas01:/usr/local/bin$ cat git-annex-wrapper
#!/bin/sh
set -e
exec '/volume1/thirdparty/git-annex.linux/runshell' "$@"
ewen@nas01:/usr/local/bin$

(which gives you enough to create them by hand if you need to, substituting the path where you unpacked the git-annex standalone archive for /volume1/thirdparty/).

To be able to run these helper scripts, and git-annex itself, from a non-interactive shell -- such as when git-annex itself is trying to run the remote git-annex, we need to ensure that git-annex, git-annex-shell and git-annex-wrapper are reachable via a directory that is in the default PATH. That default PATH is very minimal, containing:
```
ewen@ashram:~$ ssh nas01.em.naos.co.nz 'set' | grep PATH
PATH=/usr/bin:/bin:/usr/sbin:/sbin
ewen@ashram:~$
```
Since /bin and /sbin are both symlinks anyway:
```
ewen@nas01:~$ ls -l /bin
lrwxrwxrwx 1 root root 7 May 21 18:57 /bin -> usr/bin
ewen@nas01:~$ ls -l /sbin
lrwxrwxrwx 1 root root 8 May 21 18:57 /sbin -> usr/sbin
ewen@nas01:~$
```
that gives us only two choices -- /usr/bin and /usr/sbin -- which are on the default PATH. Given that git-annex is not a system administration tool, only /usr/bin makes sense.

To do symlink them into /usr/bin:
```
cd /usr/bin
sudo ln -s /usr/local/bin/git-annex* .
```
I am expecting that this step may need to be redone periodically, as various Synology updates update /usr/bin, which is why I have a "master" copy in /usr/local/bin and just symlink it into /usr/bin. For git-annex this is a chain of two symlinks:
```
ewen@nas01:~$ ls -l /usr/bin/git-annex
lrwxrwxrwx 1 root root 24 May 28 11:57 /usr/bin/git-annex -> /usr/local/bin/git-annex
ewen@nas01:~$ ls -l /usr/local/bin/git-annex
lrwxrwxrwx 1 root root 45 May 28 11:01 /usr/local/bin/git-annex -> /volume1/thirdparty/git-annex.linux/git-annex
ewen@nas01:~$
```
which is slightly inefficient, but still convenient for restoring later.
Now is a convenient time to set up ssh key access to the Synology NAS, by creating ${HOME}/.ssh/authorized_keys as usual. Since we do not need a special key to trigger a special hard coded path to git-annex-shell (because it is on the PATH) you can use your regular key if you want rather than a dedicated "git-annex on Synology NAS" key.

Ensure that the permissions on the ${HOME}/.ssh directory and the authorized_keys file are appropriately locked down so that sshd will trust them, eg:
```
cd
chmod go-w .
chmod 2700 .ssh
chmod 400 .ssh/authorized_keys
```
and then you should be able to ssh to the NAS with key authentication; if it does not work use "ssh -v ..." to figure out the error, which is most likely a permissions problem like:
```
debug1: Remote: Ignored authorized keys: bad ownership or modes for directory /volume1/homes/ewen
```
because the permissions on the default created directories are very permissive (and would allow anyone to create a ssh authorized key entry), so sshd will not trust the files until the permissions are corrected.

All going well at this point you should be able to verify that you can reach all the necessary programs from a non-interactive ssh session with something like:

for FILE in git-annex git-annex-shell git-annex-wrapper git git-shell git-receive-pack git-upload-pack; do
    ssh NAS "which ${FILE}"
done

and get back answers like:

/usr/bin/git-annex
/usr/bin/git-annex-shell
/usr/bin/git-annex-wrapper
/usr/bin/git
/usr/bin/git-shell
/usr/bin/git-receive-pack
/usr/bin/git-upload-pack

if one or more of those is missing from the output you will want to figure out why before continuing.

To centralise my git-annex storage, I created an "annex" share through the Synology NAS web interface (DSM) in Control Panel -> Shared Folder. This created a /volume1/annex directory.

To make that easily accessible, I created a top level symlink to it:

sudo ln -s /volume1/annex /

giving:

ewen@nas01:~$ ls -l /annex
lrwxrwxrwx+ 1 root root 14 May 28 12:10 /annex -> /volume1/annex
ewen@nas01:~$

This matches the pattern I use on some other machines.

Once all these setup is done, git-annex can be used effectively like any other Linux/Unix machine. For instance you can "push a clone" onto the NAS using "git bundle" and "git clone" from the bundle, and then add that as a "git remote" and use "git annex sync" and "git annex copy ...`" to copy into it.

The "standalone git-annex" will probably need updating periodically (for bug/security fixes, new features, etc), but it should be possible to do that simply by replacing the unpacked tarfile contents as required; everything else points back to that directory. (Possibly the locale generation step might need to be done by hand again.)

Finally for future reference, it is also possible to run a Debian chroot on the Synology NAS, which would open up even more possibilities for using the NAS as a more general purpose machine.

ETA 2017-06-25: Beware that (certain?) Synology updates will rebuild the root file system, and/or clean out unexpected symlinks. So after an update, or reboot, it is necessary to redo:

sudo ln -s /volume1/annex /
cd /usr/bin
sudo ln -s /usr/local/bin/git-annex* .

before git-annex-shell will be automatically found, and the nas01:/annex/... paths will work again

I have worked around this by creating /usr/local/bin/activate-git-annex containing:

#! /bin/sh
# Relink git-annex paths onto the Synology root file system
#
if [ -e /annex ]; then
  :
else
  sudo ln -s /volume1/annex /
fi
cd /usr/bin && for FILE in /usr/local/bin/git-annex*; do
                  if [ -e "$(basename ${FILE})" ]; then
                    :
                  else
                    sudo ln -s "${FILE}" .;
                  fi
                done

So that, when git-annex breaks after an upgrade, I can just run:

activate-git-annex

from an interactive shell and it will all work again. (/usr/local/bin seems to survive upgrades, but unfortunately, as noted above, is not in the default path for a non-interactive shell, so it is not a complete solution. However it is in the default path for an interactive shell hence the simple command above.)

ETA 2017-07-16: Fixed up logic of "check if already done" test in /usr/local/bin/activate-git-annex. It appears that will need to be run every time that the Synology is updated, at least in a way that will cause it to reboot.

ETA 2018-06-08: Due to a git-annex security advisory, I updated the git-annex standalone version on my Synology DS216+ NAS by moving /volume1/thirdparty/git-annex.linux to another name, and then unpacking a fresh download of the git-annex Linux standalone version, following the instructions above.

Unfortunately I ran into a problem with the locale creation/parsing which resulted in the amd64 (x86_64) standalone build of git-annex not running after the locales had been built with the error message:

sh: loadlocale.c:129: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_TIME) / sizeof (_nl_value_type_LC_TIME[0]))' failed.
error: git-annex died of signal 6

but based on a comment from Joey a couple of months ago on that issue, I instead tried the "ancient kernels" i386 build, and that seemd to work. Per my comment on the issue I expect the issue is with libc and locale tool versions, rather than the kernel -- but it appears at present the i386-archaic version is still built with an old enough version of a Linux environment to work.

ETA 2018-09-08: Unfortunately the work around of using the "Ancient i386 build" stopped working on the Synology DS216+ NAS after upgrading the firmware to DSM 6.2-23739 Update 2. Every attempt to run git annex after upgrading then failed with:

git-annex: timer_create: Bad address

as I reported in a git-annex bug. timer_create is a Linux kernel function, used by Haskell programs for threading. I initially suspected that maybe Synology had removed that from their kernel (historically Microsoft WSL did not implement timer_create which caused similar issues).

However after filing Synology ticket 2082132, they gave me the hint that it was probably a linking issue, so I guessed possibly the kernel/libraries on the Synology were now too new/changed from what the "Ancient i386 standalone build" expected.

So I moved back to trying the 64-bit version again -- having seen others run into the LC_TIME issue (noted above) elsewhere and now having a better idea how to work around it. I downloaded the latest git-annex standalone build for x86-64, and installed that (as described above/earlier), but I did find that the locales were not being built by the git-annex scipts... which possibly explains the LC_TIME issues, if it is trying to access the system's locale information with a different libc linked in (it looks like the LC_TIME structure changed in size).

After some experimentation the easiest sufficient fix seemed to be:

LC_ALL=C git-annex version

which did work:

ewen@nas01:~$ set | egrep 'LANG|LC_'
LANG=en_US.utf8
LC_ALL=en_US.utf8
ewen@nas01:~$ LC_ALL=C git-annex version
git-annex version: 6.20180807-g48d11a5df
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.2.2 http-client-0.5.13 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: linux x86_64
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
ewen@nas01:~$

so to make that work automatically, I edited the shell script wrappers of:

git-annex
git-annex-shell
git-annex-wrapper

to add some lines saying:

 # 2018-09-08 - work around for LC_TIME mismatch in locales
 LC_ALL=C
 export LC_ALL

near the top of each shell script, to override the Synology DS216+ default value (LC_ALL=en_US.utf8).

Particularly for running git-annex I am quite happy with the historic "C" formatting of things, and because those are compiled in rather than loaded from a file, I do not run into the LC_TIME issues on trying to load the locales from a file.

Thankfully this means git-annex works again for me, and I can close the Synology ticket. I've also posted a comment on the git-annex bug to hopefully give others a hint on how to get it working again.

ETA 2023-07-19: The Synology DS216+ NAS still seems to need the:

 # 2018-09-08 - work around for LC_TIME mismatch in locales
 LC_ALL=C
 export LC_ALL

work around for the git-annex x86_64 stand alone build, but otherwise git annex 10.20230408 seems to run on the Synology DS216+ (at least with the older Synology OS; I still have not upgraded to the later 2021/2022 OS major release on this old NAS). The 10.20230408 seems to be the "current" stand alone release, even though there was a git annex 10.20230626 release, and the "current" release files are from 2023-06-26. So my Synology NAS install is not completely current, but is at least much closer to current now.

ewen@nas01:~$ git annex version
git-annex version: 10.20230408-g5b1e8ba77
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
ewen@nas01:~$