File storage and backup for photographers

Eric Cheng Pictures
I spend way too much time thinking about data storage and backup. I've been a [professional photographer](/photo) for nearly 10 years, and have accumulated over 10 terabytes of pictures, video, and project data. I have finally implemented a storage and backup scheme that I'm happy with. It took a long time to set up, but I have direct access to all of my media now, and have comfort in knowing that it is securely backed up.

### Can I back up to the cloud?

A lot of normal people (non-photographers) are starting to store their pictures exclusively on the cloud, and while there are some great cloud storage services out there that cater to photographers, none of them are really suitable for storing or backing up multiple terabytes of data. Also, uploading to the cloud is slow. A fairly-fast DSL or Cable connection will probably allow you to sustain upload speeds of 200KB/s (I'm being generous). At that speed, uploading 1TB takes about 2 months. Uploading 10TB would take nearly 2 years, and a mainstream ISP will likely throttle you before allowing you to use that much data. So cloud backup is out.

### Simplicity

Over the years, I've had crazy backup schemes involving multiple computers, multiple software products and services, and custom scripts, all requiring coordinated (but automated) execution to keep my data safe. All of these schemes required that I create flow charts to track how data moved during backups; without that documentation, I might have forgotten how things work, over time.

I'm sick of these crazy schemes, and have finally settled on something much more simple.

### Kinds of data

As someone who collects pictures and video files, I think of my data as living in two different categories. I've had to simplify my thinking a lot in order categorize data this way, but that's OK. It's got to be rough to be simple.

1. **Working data:** all system files, applications, documents, email, and project data (including temporary or intermediate files and final output files) 2. **Raw data:** pictures, video, audio and other media generated by cameras or capture devices

The main difference between the two is that Raw data doesn't really change after it is captured. After a single photography trip, I might have 300GB of pictures and video, which I consider to be Raw data. I may then create 50GB of additional data in project data over time (e.g., slideshows, produced videos, edited pictures saved as TIFs). I consider all of this to be Working data. Even if it doesn't change in a long time, I may decide—at any time—to re-open and tweak a project, which will result in a need to back up again. It also means that I might accidentally screw up a project, so saving multiple versions of Working data is desirable.

### Backup requirements

- **Working data** should be continuously, incrementally backed up in a versioned manner so I can roll back to a prior state for any given file

- **Raw data** should be backed up in a versioned manner as well, but doesn't need continuous backup. I can kick this off manually, but need to have the discipline to do so regularly.

All data also needs to be stored offsite, so I don't lose everything is there is a fire or flood.

### So here's how I've implemented backup:

My main machine is a mid-2010 Mac Pro. Inside, I have:

- 1 x 500GB Samsung 840 Series SSD (boot, applications, fast data) - 4 x [Seagate Barracuda 4TB 7200RPM 3.5" drives](http://refer.ly/r/ajf1/show) in a RAID 0 stripe (16GB volume)

Mac Pro drive configuration

I am currently using 10TB of the 16TB available space, which gives me 6TB of growing room. 6TB should last a long time... unless I suddenly get a RED camera and start shooting RAW video. :) About 9TB of this data is picture / video data (Raw data), and 1TB is Working data.

I connect a [Sans Digital TowerRAID TR5UT+B](http://refer.ly/r/aVJ3/show), which is a 5-bay, USB 3.0/eSATA box that features hardware RAID. The box has 5 x 3TB Seagate Barracuda 3TB drives in it configured as a concatenated array[^1] (15TB volume). Accessed over a single eSATA port (port-multipled), this setup sustains around 90 MB/s, but when using something like rsync, I see transfer speeds between 30-70 MB/s. You can also configure the box to use RAID 0[^2] or RAID 5[^3], if you so desire.

[^1]: In theory, a concatenated array, which the box supports via switches, results in the loss of only a single drive's worth of data if a drive fails. In practice, I've never had to deal with a failure in this kind of array, so I'm just guessing.

[^2]: A RAID 0 stripe is an option as well; I see 130 MB/s from a RAID 0 stripe over a single eSATA port, and real-world rsync speeds of 80 MB/s. This is much faster than using a concatenated array, but you lose the entire set if a drive fails instead of losing only a single drive's worth of data.

[^3]: You should think of RAID 5 as a way to not lose your data if 1 drive fails, but I [wouldn't assume that you can rebuild the RAID successfully](http://hardware.slashdot.org/story/08/10/21/2126252/why-raid-5-stops-working-in-2009) if you have a lot of data. If a drive fails, copy the data off as soon as possible and start over. Considering that RAID 5 performance degrades a lot once a drive fails (by up to 80%, according to stuff I've read on the internet), this may take a long, long time. In my opinion, it's to assume the entire volume is toast when a single drive fails, so multiple backups are necessary. I much prefer newer, proprietary RAID implementations like [Synology Hybrid RAID](http://www.synology.com/support/tutorials_show.php?q_id=492), which are dynamically expandable and allows for 2-drive redundancy.

**For the 1TB of Working data, I use [Crashplan](http://www.crashplan.com)** for incremental backups to two locations:

1. a Mac Mini, which has a 3TB drive attached to it via USB 3.0 (backup set includes entire boot drive, as well as Working data) 2. Crashplan Cloud (Working data only; no system or applications, nor Raw data). The initial backup seed is still in progress: the Crashplan app tells me it will take 5 months to upload 1TB, so I will likely mail in a drive to seed the backup (a service they offer).

**I backup the 9TB of Raw data to the TowerRAID using a custom rsync script** that supports incremental snapshots (a modified copy of [Mike Rubel's script](http://www.mikerubel.org/computers/rsync_snapshots)). It took about 40 hours to do the initial backup (9.3TB over 40 hours is an average of 64.5MB/s), but successive backups take less than an hour. I keep 4 daily snapshots, 3 weekly snapshots, and 3 monthly snapshots. I may add a semi-yearly or yearly snapshot as well. For those of you who are more technical, the script uses hard links for files that have not changed, which means that I can effectively copy those files to a snapshot without using any additional drive space. Only files that have changed are actually copied to the backup during each incremental backup process.

I actually backup my entire computer, including both Working and Raw data, to the TowerRAID. Why not? I have the space to do so, and it doesn't take that much more time.

### Why snapshots?

Why use a crazy snapshot script to version files instead of just cloning a drive using SuperDuper!? Recently, two of my photographer friends discovered that they had some corrupted pictures. Both their master and backups were corrupted because once the master copy was corrupted, future backups were also corrupted. Luckily, both of them had very old backups that they used to restore good versions of the files. With versioned backups, the backup will notice that the file is different (potentially, corrupted) and make a new version. It keeps the old version so you can always go back.

### Other notes:

1. For much of my active data, I work out of [Dropbox](http://db.tt/dORRNOc), which is a fantastic cloud sync service. All data in Dropbox is instantly backed up, versioned, and accessible to any device. It works very well, and nearly everyone I know uses the service. 2. I use [SuperDuper!](http://www.shirt-pocket.com/SuperDuper/SuperDuperDescription.html) to maintain a bootable clone of my machine's boot disk. If the drive fails, I want to be able to boot up and be productive immediately. I do this every once in awhile, but am not too rigorous about doing it frequently. If you're a Windows person, try [Acronis True Image](http://www.acronis.com/homecomputing/products/trueimage/), instead. 3. I actually have two of the TowerRAID boxes, each with 5 x 3TB drives installed. One is configured as a concatenated array (as described above), and the other, as a RAID 0 stripe. One is stored offsite, and the other lives at home. I backup regularly to the box at home, and periodically swap it out with the one that is stored offsite.

There is a full list of all of the hardware referred to in this article [over at my refer.ly page](http://refer.ly/storage_and_backup_for_photographers/c/d8e324e0579911e2b5ab22000a1db8fa). Full disclosure: I get referral fees for many of the items on that page. Feel free to click through from there if you'd like to, but don't feel obligated to do so.

Backups in the field are another topic, which I'll write about at a later date.

What do you use to backup your data? I'm very interested in how other photographers—or people with large data sets—keep their data secure.

Iceberg arch collapse in Antarctica

When I was on the Sea Shepherd boat in Antarctica during 2009/2010, I captured an unusual sequence of images: a sequence of an iceberg arch collapsing. I literally raised my camera to my eye (Canon 1D Mark II w/70-200/2.8L lens), and the arch collapsed. I mashed the shutter button down and captured 20 frames—in bursts. I shot in bursts because I was afraid that the buffer wouldn't hold. Later, I pulled the frames into Motion and created this video.

Joshua Tree National Park at night

This was my first motion dolly timelapse attempt. We learned a lot that night; somehow, multiple failures still resulted in a video! Thanks to Jody Elliott for loaning me her 1Ds Mark III for the shoot and to Jason Bradley for shooting with me and hauling the gear down from the Bay Area.

Diving the Eastern Fields of Papua New Guinea, 2012

Hello, everyone. I'm back from the remote [Eastern Fields of Papua New Guinea](http://wetpixel.com/i.php/pngeasternfields). We only had 6 days out there because two strong tropical storms moved into the area, but the diving was good (until we moved back to the coast). My 4-week cough turned into a 6-week cough, so I only did 3-4 dives in the Eastern Fields. It was extremely frustrating, but I enjoyed being out on the ocean with a great group of people. Here's the video slideshow I made for the group:

Music selection by [kozyndan](http://kozyndan.com). End credits by me and Dan.

Tony Wu (my co-trip leader) has posted a [fantastic write-up of the trip](http://www.tonywublog.com/20120205/diving-the-eastern-fields-of-papua-new-guinea.html), and a [Wetpixel thread](http://wetpixel.com/forums/index.php?showtopic=44993) is tracking more pictures and experiences.

As always, you can see the expeditions we run over at Wetpixel:

Underwater Indonesia: Alor and Komodo, November 2011

Underwater footage from a 26-day Wetpixel underwater photography expedition to Alor and Komodo, Indonesia. Footage taken by me, with Canon EOS 7D, Tokina 10-17mm fisheye zoom lens and Canon 100mm macro lens. Macro footage was lit with dual Light & Motion SOLA 1200 video lights. Additional footage captured with GoPro HERO cameras in modified Eye of Mine underwater 3D housings.

Final Cut Pro X missing media problem

I used Final Cut Pro X, Apple's controversial new "professional" video editing program to cut the trip video during my latest [Wetpixel expedition to Indonesia](http://wetpixel.com/forums/index.php?showtopic=43966). These slideshows have become a tradition for guests and combine video, stills, and music into something people can take home to show their friends and family. Using Final Cut Pro X was, for the most part, fantastic. It is really more like an "iMovie Pro" and includes one-click video stabilization, color balancing, and look filters, which were all features I used extensively. Aside from dealing with small bugs, I was able to produce a 30-minute video fairly easily, without being impeded by any critical issues. However, I am back home, and am now in the process of trying to migrate my slideshow project and event files to my main Mac Pro from my MacBook Pro. This is proving to be impossible. Two [excellent](https://discussions.apple.com/thread/3133085?start=75&tstart=0) [threads](http://www.danmccomb.com/posts/1175/tips-for-reconnecting-offline-media-in-final-cut-pro-x/) provide work-arounds for reconnecting missing media, but upon following the instructions outlined within those threads, this is what happened:

1. After copying events and projects to the Mac Pro, they showed up and opened in FCPx, but all of the media showed up as missing (red thumbnails). 2. Per suggestions, I re-imported all of the media (annoying, since the files are spread out over many folders). The red missing media icons for videos and audio were all restored to color (no longer missing). This seemed like a good sign. 3. Many JPGs in the media list were not successfully re-connected; instead, they were duplicated. I have hundreds of JPGs in the slideshow, which are now all "missing" and therefore, would need to be re-cut into the timeline. 4. Soon after re-import, I noticed an import background task running. Upon closer inspection, I realized that FCPx was in the process of *copying* over all the newly imported media instead of honoring my request to leave the files in their original location (by leaving "Copy files to Final Cut Events folder" unchecked). The source media is hundreds of gigabytes. Duplicating them is not an option. 5. Closing FCPx and re-opening it immediately causes the import process to re-spawn. If I cancel the process, the media simply does not show up in my events.

Put quite simply, there is no way to move a complicated project and its associated event media from one machine to another. The fragility of the media storage system in FCPx is shameful—it's like the product was never tested in the real world.

I will, in this case, essentially lose this project and never be able to edit it again. I suppose I'll archive the events and projects in case FCPx improves its media handling capabilities. One significant side effect: if I upgrade my machine or need to reinstall, I'll effectively lose all of my FCPx projects. Until there is serious improvement in media management, I can't see myself using Final Cut Pro X again.

Shark Diving in French Polynesia (podcast)


Support us: [download the podcast on iTunes](http://itunes.apple.com/us/podcast/divefilm-hd-video-hd/id214353624) ( find "Shark Diving French Polynesia!")
In July, I went to French Polynesia with Fabrice Charleux of [Plongeur.com](http://plongeur.com) to capture stills and video of the incredible schools of gray reef sharks that call its remote atolls home. At that time, I [posted only a few still frames](/journal/tag/frenchpolynesia2011/) from the trip—not because I didn't want to share, but because I had something larger planned. When I returned home, I called up Mary Lynn Price of [DiveFilm](http://divefilmhd.com) and asked her if she would be interested in collaborating in a podcast episode. Luckily, Mary Lynn was excited about the project, and together, we have finally finished and published a new DiveFilm HD episode on the sharks of French Polynesia. To see the video, please download it from iTunes. DiveFilm is currently the only ocean-related podcast featured by Apple, and the only way to make sure it stays there is to have folks download and view it through iTunes!

Links to podcast: [DiveFilm HD on iTunes](http://itunes.apple.com/us/podcast/divefilm-hd-video-hd/id214353624). The episode is called "Shark Diving French Polynesia!"

Special thanks to [Fabrice Charleux](http://plongeur.com) (gracious host and organizer; translator; subtitling), [Mary Lynn Price](http://mlptravelvideo.com/) (editor; podcast goddess), Don Kehoe (grumpy photo assistant), [Dave Patchen](http://davidpatchen.com) (supporting friend), [Adam Tow](http://tow.com) (interview camera assistance) and Rae Chang (interview assistance). This would not have been possible without all of you!

Footage taken with Canon 7D, Canon S95, and [GoPro](http://gopro.com) Hero camera in [Eye of Mine](http://eyeofmine.com/) flat-port housing.

Eric Cheng's Top 10 Underwater Photography Tips

Here are the slides from the 90-minute seminar I gave at [Monterey Shootout 2011](http://montereyshootout.com). I always create slides without many words, preferring to talk through the points spontaneously, but it may be useful to some folks out there. When I have more time, I'll come back and add captions with notes about each of the points (and why I included specific pictures and videos).

On a side note, I have discovered that the photo and video player over at Google+ is excellent as a slideshow player on the web (as long as you do not require intra-slide interactivity).[^1] I can mix still images / video and re-order slides, and the blacked-out "theater" (to steal the Facebook word for it) is pleasing and promotes real-time interaction.

[^1]: Note that I do not really use Google+, yet, so if you interact with me in that social environment, I may not respond. I'm in evaluation mode. I have tried to import my connections via the various methods out there, and Google+ seems to fail constantly when I do anything in batch (e.g., add more than 10 people to a circle at once).

Link:

Monterey Underwater Film Festival 2011

The Monterey Underwater Film Festival audience at the Golden State Theater. I took this picture as an opener to my presentation.
The Monterey Underwater Film Festival audience at the Golden State Theater. I took this picture as an opener to my presentation.
Tonight, I spoke at the Monterey Underwater Film Festival, along with Berkley White, [Chuck Davis](http://www.tidalflatsphoto.com/), [Stephen Frink](http://stephenfrink.com/) and [Rick Rosenthal](http://www.imdb.com/name/nm1427827/). The film festival, which was attended by approximately 725 enthusiastic audience members, took place at the [Golden State Theater](http://www.goldenstatetheatre.com/) and was put on by [NCUPS](http://ncups.org) and [Backscatter](http://backscatter.com) as part of the [Monterey Shootout](http://www.montereyshootout.com/) 2011.

The audience was incredible—very "live" and reactive! There was great energy in the air, which always makes an event a lot of fun to be a part of. I was honored to present alongside underwater-imaging pioneers who have decades of experience.

Speakers: Rick Rosenthal, Chuck Davis, Berkley White, Eric Cheng and Stephen Frink. Monterey Underwater Film Festival, September 10, 2011.
Speakers: Rick Rosenthal, Chuck Davis, Berkley White, Eric Cheng and Stephen Frink. Monterey Underwater Film Festival, September 10, 2011.

Whale shark feeding in 3D

Here is a 3D video of a whale shark feeding at the surface during a huge whale shark aggregation in Isla Mujeres, Mexico. I shot it on August 15, 2011, using a [GoPro 3D HERO System](http://gopro.com/3D) and an [Eye of Mine 3D flat lens housing](http://www.eyeofmine.com/gopro/order-gopro-hero.html#euwl3d) (a flat-lens solution is required for a GoPro to focus properly underwater). The video is best viewed at 720p in some sort of 3D mode.

If you own a 3D display at home, you can [download a higher-quality side-by-side version](https://www.yousendit.com/download/ZUd2K0d1ZDVBNkUwTVE9PQ) for local display (~99MB; link is good for 500 downloads; if it fails, please [let me know](/contact)). The downloadable video is still highly-compressed and doesn't quite convey the same 3D coolness that original version does, but it is still effective!

Wetpixel Isla Mujeres Expedition 2011, Days 1-3

Day 3 of the 3rd Wetpixel whale shark expedition in Isla Mujeres, Mexico: We've had 3 days of whale shark action so far, and each day has given us something different (but spectacular). The first day, a couple hundred whale sharks were spread out in a rather long stretch of the glassy-calm ocean. The water was relatively clear, considering that it was completely full of transparent tunny eggs from the mass-spawning event three nights earlier. Whale sharks gulped down eggs around us from 8am until our boat left (at 1:30pm). On the second day, we discovered a small patch of ocean with hundreds of tightly-packed whale sharks. They were so dense that they were forced to feed in layers, and we saw as many sharks ascending and descending as we did on the surface of the ocean (very rare). Our guides were totally excited, saying that the ocean was infestado with whale sharks. After thirty minutes of total whale-shark insanity, the sharks vanished in a coordinated descent into the depths—it was totally bizarre. One minute, we were surrounded by literally hundreds of sharks, and the next, there were only a few left on the surface. All of us, including the local guides, were totally dumbfounded by the strange behavior.

Today (day 3), we found the sharks 4 miles east and 2 miles south of where they were yesterday. It took a coordinated search effort by multiple boats to find them (which took 3.5 hours on the water), and we weren't in the water until 9:45am. The action was fantastic, with botellas almost literally everywhere we looked (a botella is a stationary whale shark that is vertical in the water, "gulping" water constantly to feed.

I've been shooting with both a Nauticam-housed Canon 7D with Tokina 10-17 fisheye zoom lens, and with a 3D GoPro HERO setup (with Eye of Mine 3D underwater GoPro housing). The 3D GoPro setup has been yielding some very interesting footage because I can get the camera in places where a big housing can never go (e.g. right in front of a whale shark that is cruising at speed). I have some interesting 3D footage that I'd love to present, but two failed upload attempts to YouTube are enough; I'll upload when I return to the States.

In the meantime, here's a 3D screen-grab from the video (red/cyan 3D glasses required):

3D whale shark gulp with GoPro 3D HERO camera / Eye of Mine 3D underwater housing

I also have cute / precious footage of Kieran Liu (the 5-year-old son of my friends Kenny and Lori) swimming madly after a whale shark (and managing to get really, really close). He is fearless!

**Update:** here are links to the videos:

- [Kieran Liu swims with a whale shark](/journal/2011/08/20/kieran-liu-age-5-swims-with-a-whale-shark/) - [3D whale shark feeding video](/journal/2011/08/20/whale-shark-feeding-in-3d/)

Space shuttle Endeavour STS-134 launch video

Here's a video I made of STS-134 using the [50-image launch sequence](http://echeng.com/journal/2011/05/16/photo-of-space-shuttle-endeavor-launch-sts-134-hires-download/) shoved into Adobe After Effects. Time warp FTW!
You can [download this video for your iPhone or iPad here](/video/ECHENG-STS-134-horiz-H264-iPad.mp4) (7.6 MB). It has been rotated so it shows up correctly on mobile devices.

I'm glad Pam took audio. The sound, even from 6 miles away, was incredible.

GoPro 3D HERO Review

GoPro 3D Hero housing with two GoPro HD HERO cameras installed

A few days ago, GoPro announced the [GoPro 3D HERO System](http://gopro.com/hd-hero-accessories/3d-hero-system/), the world’s smallest 1080p 3D camera. The 3D Hero System consists of a housing that accepts and aligns two GoPro HD HERO cameras for 3D capture, and GoPro CineForm Studio software, which is available as a free download on GoPro’s website.

GoPro sent me a 3D HERO to test, and it arrived today. [Full review over at Noodletron](http://noodletron.com/2011/460/).