Backing up a NAS to CrashPlan using a Raspberry Pi

[Update on July 28th 2006] I have updated the post to reflect that CrashPlan 4.7.0 still works the same way as well as adding instructions for running the UI on MacOS/OSX

Motivation

I currently have a Synology DS212 (discontinued, but similar to more current DS213), which works fine, but I wanted to have another backup of the data on it off-site, meaning not in my house.

A few years ago I got myself a CrashPlan subscription and have since used to to back up my personal and my families data from multiple machines. Ideally I would like to run the CrashPlan software on my NAS, but the DS212 is not powerful enough to handle the Java-based CrashPlan client.

Since I anyway already have a Raspberry Pi 2 Model B set up as a home server (I am running a custom home automation server and some other servers on it), I thought it would be great to add the CrashPlan client to it.

Getting the CrashPlan headless client running on a Raspberry Pi is luckily possible, but there are a few issues that prevent it from working right out of the box. Thanks to other online tutorials though I've been able to get it working and I want to share here what I did. See the end of this article for a full list of sources I consulted, and without which this would not have been possible.

Aside from this I also want to explain my setup in general: I have around 2TB worth of data on the NAS right now, at least 1TB of which are photos that I want to back-up first. Even with a fast Internet connection this will take a while and I don't want to keep the precious hard disks in the NAS spinning for weeks or even months. They would not be able to go to standby and spin with no pause, decreasing their live expectancy.

So I decided to instead connect a portable USB drive to my Raspberry Pi to which I clone the data I want to back-up from the NAS. Cloning once is relatively fast, since both the Pi and the NAS are in the same network, and incremental syncs will not take long either. I simply mount the NAS drives and then use rsync to sync new files once a day. This is how my setup looks like:

Below you will find details about:

• My hardware set-up and use of rsync
• How to install CrashPlan and get it to run headless on the Pi
• How to connect from a Windows box using the CrashPlan UI
• How to fix the 'Waiting for backup' bug caused by InotifyManager that prevents the backup from starting.

Getting data to the Raspberry Pi

So lets start with the setup first. As stated above I use a simple USB disk as an intermediate place to store my backup before sending it to CrashPlan. I got myself a Western Digital WD Elements 2TB drive for this, but any somewhat reliable USB drive will do. Just make sure it has enough capacity to hold the data you want to back up. If the drive dies, no problem, your original data is still on the NAS.

Format and mount USB drive

I formatted the USB drive with ext4, which is the standard nowadays for Linux systems and reliable. I created just one large partition and mounted it at /mnt/usb-drive - but of course you can mount it anywhere you like.

The next step is to access the NAS data, for this I simply mounted my "Photos" folder from the NAS by using the following /etc/fstab entry:

//storage/Photos /mnt/storage-photos cifs credentials=/etc/cifspwd,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

The first part 'storage' is the NAS server name, but you can also use an IP address. 'Photos' is the name of the drive on the NAS I want to mount. Next comes the directory I want to mount it at, followed by the file system (cifs if the standard windows file sharing protocol). I added my username/password credentials to the /etc/cifspwd file and the specified character mode and permissions. Done. Now the drive will be mounted every time the Pi restarts.

Copy data using rsync

With both the NAS and the USB drive mounted, I use rsync to copy and periodically update the files. Now rsync has a ton of options, but I found that for my setup the following works great:

$ rsync -vr --progress --stats

• -r: Most important, it will sync recursively through all subdirectories
• -v: Verbose logging, in case something goes wrong. Useful when manually executing the command.
• --progress --stats: Also useful when manually executing the command to see what is going on.

I then put the above into /etc/crontab and let it run once during a day, during the night. I decided to go 3 a.m. since this is a time I am unlikely to be online, changing the files on the NAS. If you schedule CrashPlan to only run at a given time frame, it's good to coorindate the rsync time so that it runs before CrashPlan kicks in. E.g. you could have rsync run ar 2 a.m. and then have CrashPlan run from 3 to 7 a.m., for example.

Even without uploading the files to the cloud this is already pretty useful. We created one more copy of the data we care about, so if the NAS for some reason malfunctions and deletes all data, we have one more copy on the USB drive.

Getting CrashPlan to run on the Raspberry Pi

Now having the data in two places locally is great, but the main goal of this exercise is to get the data to a different physical location, in my case using CrashPlan.

Luckily the CrashPlan software is well designed and consists of a headless server and a GUI that can be run separately. This means that you can run the client on a Raspberry Pi without an X-Server and run the GUI on a different Windows, Mac or Linux box, then connect to the Pi through the network.

First, let's get CrashPlan installed on the Raspberry Pi. Luckily they have a Linux version of their software, however it is optimized for x86 CPUs and not the ARM architecture. This means we have to make a few modifications to the package before it successfully runs.

Download and installation

CrashPlan comes packaged with a Java runtime that is compiled for x86 and not ARM. So lets make sure we have Java installed before we get going. We apparently also need the native access library, so use the following to get an up-to-date copy of both:

sudo apt-get install libjna-java oracle-java8-jdk

The latest CrashPlan Linux version can be downloaded from their download site. At the time I am was writing this article it was version 4.3.0, but I have since upgraded to 4.7.0 (on 7/28/2016) and it works the exact same way. Extract the file, then go into the CrashPlan-install directory that was extracted and execute the install script as root:

$ sudo ./install.sh

I leave everything as is, expect for the location for the backups. I use /home/pi/crashplan-backups instead.

Once you accepted all the configurations the script will download all the files required for the installation, which might take a few minutes. Once it's done it will tell you:

Installation is complete. Thank you for installing CrashPlan for Linux.

So far, so good. It also tells you that you can start the engine using sudo /usr/local/crashplan/bin/CrashPlanEngine start, but unfortunately this will fail on the Pi. Although the output will be "OK", ps aux | egrep -i crash will show you it's not actually running.

The problem is that this version of CrashPlan for Linux is not made for the ARM architecture. Although CrashPlan is running on Linux, it still uses a bunch of native libraries to do its job. And the ones that come with the package are not made the ARM platform. You can find a detailed error in the engine error logs:

$ cat /usr/local/crashplan/log/engine_error.log

/usr/local/crashplan/jre/bin/java: 1: /usr/local/crashplan/jre/bin/java: ⌂ELF☺☺☺☻♥☺►84490: not found

/usr/local/crashplan/jre/bin/java: 2: /usr/local/crashplan/jre/bin/java: Syntax error: "(" unexpected

We need to tell CrashPlan to use our installed Java instead of the one it came with. Some sites say to remove it and add a symlink. An easier way is to simply edit /usr/local/crashplan/install.vars and set JAVACOMMON=/usr/bin/java. Trying to start the engine now we get a bit further, but still no success:

$ cat /usr/local/crashplan/log/engine_error.log

java.lang.UnsatisfiedLinkError: /usr/local/crashplan/libjtux.so: /usr/local/crashplan/libjtux.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform)

        at java.lang.ClassLoader$NativeLibrary.load(Native Method)

        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1929)

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1847)

        at java.lang.Runtime.loadLibrary0(Runtime.java:870)

        at java.lang.System.loadLibrary(System.java:1119)

        at jtux.UUtil.init(UUtil.java:14)

        at jtux.UProcess.(UProcess.java:30)

        at com.code42.os.posix.PosixProcessCommands.getuid(PosixProcessCommands.java:56)

        at com.backup42.service.CPService.logIds(CPService.java:1536)

        at com.backup42.service.CPService.start(CPService.java:436)

        at com.backup42.service.CPService.main(CPService.java:1922)

As can be seen, the libjtux.so file CrashPlan comes with is also not ARM compatible, so we need to replace it. The same is true for libmd5.so. Thanks to Jon Rogers, there are patched versions available for both. So remove or rename the old one and get the ones that work:

$ sudo mv libjtux.so libjtux.so.x86

$ sudo wget http://www.jonrogers.co.uk/wp-content/uploads/2012/05/libjtux.so
$ sudo mv libmd5.so libmd5.so.x86
$ sudo wget http://www.jonrogers.co.uk/wp-content/uploads/2012/05/libmd5.so

Starting up the service now should be successful, and engine_error.log should stay empty. The following command confirms that CrashPlan is indeed running:

$ sudo service crashplan status

CrashPlan Engine (pid 11241) is running.

To wrap it up, make sure to start CrashPlan when the Pi boots up:

$ sudo update-rc.d crashplan defaults

Access CrashPlan from a different machine

Now that CrashPlan is running on the Raspberry Pi we need to be able to connect our account and configure it. While this can be done through configuration files, the most convenient way is to use CrashPlan's own UI. Luckily you can run it on a different machine. This can be another Linux machine, Windows or MacOS/OSX which runs the UI and then connects to the engine running on the Raspberry Pi.

CrashPlan has a good tutorial for this, but emphasizes that this is not an officially supported feature. Here are the highlights for getting this to work on Windows:

Access without SSH tunnel or port forwarding

CrashPlan recommends using an SSH tunnel to forward the port on the server that accepts the UI connection to the computer you are running the UI on. Instead, you can change one parameter to have the engine accept connection from any machine. Edit /usr/local/crashplan/conf/my.service.xml and change the serviceHost value to the following, to allow connections from any computer.

 0.0.0.0

If this is too insecure for you you can put it in the IP of the computer you are going to connect from.

Copy authentication token

Next copy the contents of /var/lib/crashplan/.ui_info, which we will need to bring over to the machine running the UI. On Window, you can find the file at C:\ProgramData\CrashPlan\.ui_info., on OSX it is /Library/Application Support/CrashPlan/.ui_info. If you have trouble editing the file, make sure to run the editor you use as Administrator, otherwise you might not be able to save the changes (Windows will complain that another program has the file opened, which is not the case, instead a normal Windows user simply has no write access for this file).
Copy the contents of the file from the Raspberry Pi in there. Despite the CrashPlan tutorial, do not change the port to 4200 since we will connect directly without port forwarding. But do change the IP at the end to point to the server (do not leave it as 0.0.0.0, since the UI will not be able to find the server otherwise).

Start CrashPlan UI and configure service

Now you should be able to start he CrashPlan UI form the start menu. If it doesn't work, make sure to kill the CrashPlanDesktop.exe service from the Task Manager and try again.

CrashPlan should now let you log into your CrashPlan account. Configure everything and select the folders on the usb-disk that you want to back up. IMPORTANT: Don't select the mounted NAS directly, the whole point of the USB disk was to let CrashPlan use that location for it's backup.

'Waiting for backup' problem

Now the world would be perfect, however, you will see that the backup is not starting. Back on the Pi we can see the reason:

$ tail -f /usr/local/crashplan/log/engine_error.log
Exception in thread "W15324091_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
        at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.(JNAInotifyFileWatcherDriver.java:21)
        at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:393)
        at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:331)
        at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:66)
        at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1073)
        at com.code42.utils.AWorker.run(AWorker.java:158)
        at java.lang.Thread.run(Thread.java:744)

I did two things to prevent this from happening:

Firstly, I create a new directory for temporary files:

$ sudo mkdir /var/crashplan

and added this path to first position of the SRV_JAVA_OPTS variable in /usr/local/crashplan/bin/run.conf. So my line now starts like this:

SRV_JAVA_OPTS="-Djava.io.tmpdir=/var/crashplan -Dfile.encoding=UTF-8 ...

Secondly, I needed to tell CrashPlan to use the JNA library we installed at the very beginning. To do this, edit /usr/local/crashplan/bin/CrashPlanEngine and search for the line that defines the FULL_CP variable, and put the path to /usr/share/java/jna.jar in the first position. My complete line in the end looks like this:

FULL_CP="/usr/share/java/jna.jar:$TARGETDIR/lib/com.backup42.desktop.jar:$TARGETDIR/lang"

Now restarting the service will not show any errors and once I connect with the CrashPlan UI from my Windows box, I can successfully start the backup.

Conclusion + Thanks

This rather lengthy article is more of "notes to self" piece than a classic tutorial, but I think it might be useful for some, which is why I didn't just leave these notes in my folder and instead published them here. There are a lot of useful resources out there that I had to pull together to make this work, so maybe this tutorial is helpful for some that try to do the same.

The following sources were extremely helpful for most of the steps and workarounds above, I couldn't have gotten this to work without them:

• Bion helped me find the FULL_CP workaround for make it all work:
  http://www.bionoren.com/blog/2013/02/raspberry-pi-crashplan/
• Jon Rogers provided general installation steps and the all-important ARM native libraries:
  http://www.jonrogers.co.uk/2012/05/crashplan-on-the-raspberry-pi/
• François Marier provided a tip regarding the temporary directory change:
  https://feeding.cloud.geek.nz/posts/crashplan-and-non-executable-tmp-directories/
• Mel Grubb has a full rundown that also pointed out the FULL_CP workaround, that helped me eventually get things going. If you use CrashPlan UI on a Linux box, his article will help you out:
  https://melgrubb.wordpress.com/2014/08/01/raspberry-pi-home-server-part-10-crashplan/