Adobe Lightroom 4 DNG performance increase test

Earlier this month, Adobe announced that there would be [three upcoming improvements](http://news.cnet.com/8301-30685_3-57371809-264/adobe-offering-new-reasons-to-get-dng-religion/) to the DNG camera raw "standard" that would improve performance. I've not historically converted my RAW files to DNG, but performance improvements would certainly be a big factor in a decision to move to that workflow. I don't know whether any of the announced improvements have actually made it into Adobe Lightroom 4, but I the upgrade from version 3 to 4 last night ($79), converted some files to DNG, and ran tests. Navigation and editing was as sluggish as ever on my 12-core Mac Pro (with plenty of RAM)—I couldn't tell if there were improvements when working with DNGs. Then, I did some exports to 1920-pixel JPG files:

**Test 1: 28 ARW raw files from Sony NEX-5N** ARW->JPG, 2010 Process: **52 seconds** [ARW/DNG]->JPG, 2010 Process: **40 seconds** DNG export **23% faster**

**Test 2: 28 CR2 raw files from Canon 7D** CR2->JPG, 2012 Process: **62 seconds** [CR2/DNG]->JPG, 2012 Process: **43 seconds** DNG export **30.6% faster**

Here's the CPU graph for the export process. You can see the clear improvement in CPU utilization in the DNG export.


CR2->JPG export (green), DNG->JPG export (orange)
*24 virtual cores in 12-core Mac Pro*

It does seem that there are real reasons to convert raw files to Adobe's DNG raw format. If Adobe continues to improve DNG performance, I'll likely move my workflow over to DNG conversion upon import.

I love Lightroom—it is my photo organization and processing app of choice—but in general, Adobe Lightroom 4 feels sluggish, and it took as many as 6 seconds to move between Library and Develop modes (inconsistent, presumably due to caching). No interface move should take 6 seconds on this computer. I hope there is a performance update in the near future that deals with the poor performance.[^1]

[^1]: To be clear, Adobe Lightroom 3 also feels sluggish.

My email workflow (evolved)

I've been struggling with email workflow since selling my MacBook Pro. My notebook computer used to be communication central -- the (somewhat) mobile device that held my master email archives and the most current version of projects in process. My more powerful Mac Pro was used for heavier projects (media-related), and occasionally received a sync of working files from the MacBook Pro. I'm more "local" now, but strangely, it has also made me more mobile. At the same time, I've lost access to my main email archive when I'm away from my Mac Pro (which has become the new central repository).

In 2005, I started redirecting a copy of all of my email to Gmail. I didn't really use Gmail at that point, but I liked that I could go online and search my email archives (from 2005 onward) from anywhere. Shortly after, I started using Gmail as an email aggregator for mobile devices: all of my email accounts redirected mail to Gmail, which then redirected a copy to whatever mobile device I happened to be using at the time. It was fantastic back then because Gmail's collaborative anti-spam filters were so good compared to what others had to offer.

During all this time, I didn't actually use my gmail account actively. If someone emailed my gmail address, it would get redirected to my mobile device, but my computer mail apps would never see it. This was clearly not ideal because I missed a lot of mail and had to expend effort to try to prevent people from emailing me at Gmail. I've since added my gmail address to my list of active email addresses and use it as my main email service.

At work, we use Google Apps for Domains. I use Gmail's web client, Mail.app on Mac OS X (via IMAP), and Mail on iOS, and religiously maintain a zero-message inbox. This is easy because I am able to subscribe to the "All Mail" label from IMAP clients; because Google favors archive over delete, I can just delete messages from any client Inbox and know that they will remain archived and searchable in "All Mail" both in Gmail and on local clients.

I'd love to implement a similar workflow at home, but there are a couple problems with enabling IMAP access to "All Mail" on my normal account:

1. My "All Mail" label currently has over 100,000 messages in it and takes up 9GB. Most mail clients do not play well with these numbers. 2. I also have an organized local mail archive containing over 100,000 messages (pre-2005 email), and searches would bring up duplicate messages.

Also, I hate the idea that my mail archives would be split between Gmail and a local message store. I want the local message store to be complete, but I also require that my local Mail applications stay stable (which they don't when you have so much email).

So my email workflow goals are:

1. Have a full archive of email on both Gmail and in Mail.app (local) 2. Be able to archive messages properly from any e-mail client 3. Retain good performance in local Mail apps

I think I've come up with a solution:

1. **Sync full archives by copying unarchived email into Gmail.** The dumb way to do this is to enable access to "All Mail" and copy all of the messages that aren't already on Gmail into it. IMAP will synchronize the folder and push all the messages to Gmail. However, this is clearly not a good solution since enabling IMAP access to "All Mail" in the first place breaks my mail app because it immediately tries to download 100,000 messages. Instead, I created a new Gmail label called "old" and enabled IMAP access to it. The new label appears as an IMAP folder in Mail, and when I copy messages to it, they are uploaded to Gmail. After I copy all 100,000+ old messages into Gmail, I will remove the "old" label from the system. This will give me full archives in both Gmail and Mail.app. 2. **Automatically label new messages coming into Gmail.** I created a new IMAP-enabled label (I called it "~") and a filter that automatically tags all incoming Gmail. This means that a new message will appear in an IMAP mail client twice: once in the inbox, and once in the "~" IMAP folder. When I'm done with the message, I just delete it from the inbox, and it remains archived in "~". However, my mail client only retains the message in an IMAP cache folder, so I will have to periodically flush the cache out to a local message store by moving the messages in "~" to a local folder (which automatically removes the "~" label from those messages in Gmail).

Step 1 will take hours, if not days. I have a lot of email, and it has been taking an hour to upload 2000 messages or so. Once all of this is done, I will be able to search my entire email archive from anywhere and delete messages from the inbox without having to worry that they won't appear on my local Mail.app instance.

I hope it works!

Transcode war: MPEG Streamclip vs Compressor

A rather unscientific comparison of MPEG Streamclip and Compressor speed during transcoding shows that MPG Streamclip is about 2x the speed of Compressor during transcode of H.264 to ProRes. I've not tested against Compressor with a virtual cluster on my quad-core Mac Pro because I have had problems with Compressor jobs hanging every once in awhile when I target clusters.

Source clips are 1080p clips from a GoPro camera. Video is sped up 20x.

Compressor is rather disappointing these days. I feel like it hasn't changed for years.