Time Machine vs Crashplan and other thoughts on backup

Those of you who follow me on social media know that I complain often about the work required to keep my data securely and reliably backed up. Since 1997, I've been a hardcore [Crashplan](http://crashplan.com) user. I love that it features continuous, versioned backups to multiple destinations, and that it supports multiple platforms. You can backup any Mac, Windows or Linux machine to virtually any destination (folder, second computer, or cloud). I continue to recommend Crashplan over Time Machine for normal users; for a very-small monthy subscription fee, you can back up every machine in your family to an external drive, and to the cloud—a fast, local backup plus a secure offsite backup, all in one solution.

Unfortunately, I am not a typical user. I have about 10 terabytes of data that I need to keep easily accessible and securely backed up. About half of that are static image files, which are stored on a redundant NAS volume and backed up offsite, which leaves the other half—about 5 TB—needing to be actively backed up. To seed 5 TB to a Crashplan destination on my gigabit LAN, it takes between 7-10 days, which yields an average backup speed of 6-8 MB/s. That is really, really slow.[^1] Still, Crashplan mostly just worked, and I have used it in the past to retrieve old versions of files that had somehow become corrupted. The problem was that when Crashplan didn't work, it was often catastrophic. After months of continuous backups of huge datasets, Crashplan inevitably started to do bad things. Typical problems included:

[^1]: Note that seeding 5 TB to the cloud on my ~180 KB/s ADSL2 connection would take nearly a year.

1. Neverending synchronization. Crashplan was left in "Synchronizing..." state for hours or days at a time. 2. Connection problems. Crashplan would stop being able to connect to a destination, even though a machine right next to the problem machine would connect and backup without problems.

Each time, I submitted support tickets that were answered within about 48 hours, and each time, I was left without a working backup. In order to recover from these bad states, I had to start a new backup, leaving me in a vulnerable state during the time necessary to seed a new backup (7-10 days).

After 3-4 instances of this, I have come to the connclusion Crashplan is simply not reliable for large datasets. I think that if I had less than 1 TB of data, I would still happily be using Crashplan, but it stops working way too often when datasets are too large.And so, I recently gave Apple's Time Machine a try again, pointing it at a Synology NAS.

I was surprised by how fast Time Machine seeded its backup, and was pleased that Time Machine didn't simply crash when I launched it, which is what happened the first time I tried it (when the feature was new). The ~5 TB backup took less than 1.5 days, which means that it averaged nearly 40 MB/s—totally respectable for moving almost 5 million files across the network. However, after about a week, Time Machine told me that it wanted to create a new backup "to improve reliability," which is dumbed-down speak for having detected corrupted sparse bundle. This would have left me in a state without a backup again—precisely what I want to avoid. A [Google search](https://www.google.com/search?q=time+machine+synology+start+new+backup) yields many reports of problems using Time Machine to backup to NAS boxes that have built-in Time Machine support.

As a final attempt to make this work, I am now using Time Machine to backup to an iSCSI volume on a Synology NAS. [Synology's wiki](http://forum.synology.com/wiki/index.php/Backup_Mac_data_using_Time_Machine) suggests using iSCSI "if the network connection will be regularly interrupted," and pretty much every modern machine falls into this category, since notebooks are closed and moved, and desktop machines regularly go to sleep or are turned off. So we'll see what happens with Time Machine over iSCSI.

If my most recent effort doesn't work, I'll likely try to decrease the size of the data that needs to be actively backed up by switching photo and video backups to cron'ed, incremental rsync to a NAS volume. I assume that both Crashplan and Time Machine do better when dealing with smaller amounts of data.

I know there is a better way. NAS boxes seem to backup to other NAS boxes very well (e.g., Synology to Synology via rsync), and working with RAW image files off of gigabit is totally fine (I can barely tell that they aren't local). I envision a setup in which my entire working image library is on NAS, backed up to a second local NAS and to a third offsite NAS. All of my video files would be on a Thunderbolt-attached RAID, which would be backed up to NAS using rsync. My boot disk, Applications, general data and project files would be backed up using Time Machine. I think it could work, but I'd have to be strict about compartmentalizing data into source files that never change (RAW image and video files), files that are constantly in flux that need to be continuously backed up (libraries, project files and other working files).