Author Archives: gregr

Lightroom and Mac Pro performance – part 2

Back in February, I wrote about the performance I was seeing with my Mac Pro, and compared it to several other machines. My intent was to compare performance between various different machines and configurations…but it had the unexpected side effect of giving me a baseline with which to measure new versions of Adobe Photoshop Lightroom.

This data proved to be quite useful when Lightroom 2.0 was released, and I could test the 32-bit build against the 64-bit build. What I found was surprising.

Just like last time, the basic test was generating 1:1 previews for 211 raw images (compressed NEF format from a Nikon D200), about 1.75GB of files.

First, Lightroom 2.0 32-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization between 350-500% throughout
Total time 4:57, average 1.41 sec/image

And Lightroom 2.0 64-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization between 500-550% throughout
Total time 7:06, average 2.02 sec/image

So the 32-bit version on OS X 10.5.5 was actually a bit faster than my previous tests of Lightroom 1.3.1 on 10.5.2, which could be due to either the Lightroom 2.0 upgrade, or 10.5.5 changes.

But the 64-bit version of Lightroom 2.0 totally blew chunks, as you can see. Pretty disappointing, and totally reproducible for me, so I opened a ticket with Adobe. They got back to me saying they could reproduce the problem, and also had a bunch of mumbo-jumbo about how there is more overhead in accessing memory in 64-bit mode (which I’m not sure I believe, but I’m not up to speed enough on the Leopard 64-bit implementation to know for sure).

Well, Lightroom 2.1 was recently released, and here is the data:

Lightroom 2.1 32-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization between 350-500% throughout
Total time 4:55, average 1.40 sec/image

Lightroom 2.1 64-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization about 400% throughout
Total time 4:05, average 1.16 sec/image

Wow – obviously a big change; the 64-bit version now totally rocks. There was something in the 2.1 release notes about enabling SSE extensions in 64-bit mode – sounds like that might have been the bug!

I didn’t record the memory usage numbers, unfortunately…but I remember watching during the runs, and the 64-bit version seemed to be using about twice the memory that the 32-bit version did.

NetNewsWire for iPhone usage

Josh Larson on NewsGator’s Media and Consumer team has a blog post up with some usage stats for NetNewsWire for iPhone in its first 9 weeks or so since release. Go take a look…over 200,000 users, and over 115,000 new-to-NewsGator users have signed up in the last 30 days.

We have some other data as well that’s more difficult to explain, so we didn’t publish it…but generally, what we’re finding is users of the iPhone app are sticking with it and using it regularly. Said another way, the abandonment rate of iPhone users is quite a bit lower than our average rate (which is already fairly low, but the difference is significant). It seems the combination of NetNewsWire and the iPhone is a winner!

NewsGator’s sync platform details

For the vast majority of NewsGator users (including folks using NetNewsWire, FeedDemon, or any of our other applications), NewsGator’s sync system works totally transparently. But there are some nuances of our implementation that are sometimes visible to users. So, in the hopes of giving people a definitive place to go to understand this, I offer the following in-depth explanation of NewsGator’s sync platform.

The Mechanics

All content is stored on NewsGator’s servers. When an application like NetNewsWire does a sync, it sends up some bookkeeping information up to the NewsGator system (a “sync token”), and the system returns a list of feeds that NNW needs to update. NNW then requests updates for each of those feeds (which is generally a subset of the list of subscribed feeds), again using a sync token, and the system returns the new (not yet seen) or updated articles for that particular feed.

This system is extremely efficient. For feeds that haven’t updated, NNW will not even have to request them. For feeds that only have a single new item, that’s the only data that will be returned to the client. For scenarios like mobile applications (like NewsGator Go! or NetNewsWire for iPhone), this is pretty close to the minimum theoretical bandwidth required to deliver the content.

It’s actually possible to reduce the number of calls even further, but at the cost of a potentially large (and expensive to process) response. Our APIs are instead optimized around the case I describe above.

[note: this is somewhat simplified; for example, metadata also travels both ways during a sync, but I’m leaving discussion of that out for purposes of this article.]


NewsGator’s online platform processes about 3.5 million feeds, and stores about 9 million new articles per day, as of this writing. There are a total of about 3 billion articles in the system.

Suppose you subscribe to a feed from, and further assume that that feed publishes 100 new articles per day (I have no idea how many it actually publishes – just go with me here). Now imagine you go on vacation for a month, and you come back, fire up FeedDemon, and sync. There are now 3000 articles you haven’t seen. Should we deliver them all to you?

Probably not.

Do we?

No, for several reasons. First and foremost, the user experience therein would totally suck; no one wants to wade through 3000 articles from a single feed. And second, it’s pretty inefficient to retrieve all of this content from the API – we could deliver it, but it’s going to take a lot of bandwidth to retrieve it, and a lot of work to process it. Using a mobile phone? This might well lock it up.

And the other reason is, our system allows you to mark individual articles as read, and that data is synchronized throughout the system and all of the clients you use. But we don’t store your read states for all time – we only store it for fairly recent data. Do you really care if you marked an article read 2 years ago? Probably not.

So what do we do? We have a limit of how many articles will have their metadata state synced through the system. Here’s the rule we currently use:

The number of articles in the current feed,


14 days, up to 200 articles.

Whichever of the two conditions above yields more articles is applied.

Here are three examples of applying this rule:

Imagine a blog that publishes 5 times per month, and its feed has the most recent 10 items in it. This feed would sync 10 articles.

Now imagine the hypothetical CNN feed above, which had 100 items per day, and imagine the feed held the 20 most recent items. In this case, we would sync 200 items.

And finally, if a feed published 10 times per day, and held the most recent 20 items in the feed, we would sync 140 items.

If you really want to go back and browse through all 3000 articles you missed, you still can – they’re all stored in NewsGator Online, and you can view them from the web site. In fact, you can go back all the way to when we first discovered the feed – over 4 years ago, in many cases.

The Gotcha

For most feeds, the algorithm described above makes things completely transparent, and articles and unread counts across all NewsGator-integrated applications will match up perfectly.

The gotcha is with feeds that have a lot of articles. For example, I have a smart feed for the term “NewsGator”, and I see probably 400 new articles there per day, 200 at a time. So in this case, only 200 articles have state synchronized.

What can happen is the following:

1. NetNewsWire downloads the feed, and shows 200 items, all unread.

2. 3 hours later, you sync from FeedDemon, and you see 200 items, all unread. You read them all, and mark them read.

3. An hour after that, you sync again from NetNewsWire…it syncs state from the old articles, and downloads say 25 new ones. You see 47 unread articles. You immediately sync again with FeedDemon, and it shows 25 unread. What gives?

The problem is the 200-article limit, and the fact that some of the articles fell off that ledge while still showing as unread in one application…and thus don’t have all of their state synchronized.

This problem used to be much more acute – it’s more rare now that we’ve raised the article-sync limit up to 200. But it’s still possible to run into this, specifically with very prolific feeds.

We’ve experimented with various different limits – the current 200 articles seems to be a good compromise, perfectly syncing the vast majority of feeds while maintaining the efficiency that our client applications demand. We’re also working on some things that will make this invisible to API clients, while still working within the constraints.

Exporting video for iPhone

I was editing a video yesterday using Final Cut Express 4, and wanted to create a version that would play on the iPhone. The timeline I was using was 1080i60, and the aspect ratio was 16:9.

First I tried File / Export / Using QuickTime Conversion, and selected “iPhone” as the format:


This created a video that would indeed sync through iTunes to my iPhone, and play successfully, but it messed up the aspect ratio – it exports a video that’s 480×320, which is a 4:3 ratio, so everything in my video was tall and skinny. Pretty useless.

Then I tried File / Export / Using QuickTime Conversion, selected “QuickTime Movie” as the format, and set the compression type to H.264. Dial the bit rate down to say 900 kbps, set the audio to AAC 128kbps, and export. But – no matter what I did (and I tried just about every combination of settings I could think of), this video would not sync to the iPhone through iTunes – it had to go through a conversion (in iTunes, Convert Selection for iPhone/iPod) first. Definitely not what I wanted.

Then finally, I tried File / Export / Using QuickTime Conversion, and selected “MPEG-4” as the format:


Click “Options”, then make the following settings:

File format: MP4
Video format: H.264
Bit rate: 900 kbps
Image size: Custom (set resolution you want here, e.g. 480×270)
Frame rate: current


But then, here’s the key to making the whole thing work, which unfortunately took a while to figure out. Click “Video Options”, and set the profile to “Baseline”:


Finally, set the audio to AAC-LC, and you should be good to go.

Hope this is helpful for someone…but at very least, the next time I need to do this, I will have it written down. :-)

MobileMe growing pains

By now we all know about the couple of days it took for the .Mac to MobileMe transition to happen. I didn’t really complain about it – during the whole time, email was working fine (from a mail client), and sync was working most of the time. It was just the new web apps that took a while to come up.

More recently, on Friday, there was a complete mail outage for 1% of MobileMe users, and said outage has still not been fixed (as of Wednesday mid-morning). I’m in that 1%.

I’m not really too upset for myself; I’m watching this more from an academic perspective. I use the MobileMe sync a lot between two Macs and an iPhone, and I do use the email, but not often. The fact that mail is down doesn’t dramatically affect my life at the moment. And in general, I’ll be the first to say I really like MobileMe and what they’re doing with it.

But after the email being down for 5 days so far, the techie in me can’t help but wonder what is wrong. Apple put up a KB article about it, saying:

On Friday, July 18, 2008 (2008-07-18) we experienced a serious issue with one of our MobileMe mail servers. This issue is currently affecting approximately 1% of MobileMe members. Affected members are unable to send or receive email at or access email using any email client software such as Mail on a Mac or Microsoft Outlook on a PC.

Let’s look at what could perhaps cause something like this:

1. Complete server failure. Well, in 5 days, you could for sure have another server in there. Actually in more like a couple of hours, assuming their service contract with Sun (they appear to be using Sun’s mail servers) is up to date.

2. Disk failure. Perhaps the entire disk array that this 1% of mailboxes is stored on melted down. Even if this was true, you could almost certainly restore from backups in hours, or worst case days.

3. Centralized disk failure. If they’re using large storage arrays, it’s possible there was a systemic failure and they can’t get it back online. However, they’d probably have a much bigger problem than a 1% outage if this was the case.

4. Multiple storage failures. If multiple drives all failed at one time, causing an array to come down, and there was no usable backup, then they might send the failed drives out for data recovery – eek. But still – I would think this could be completed in a day. I’ve never done it, though, so perhaps this takes a while.

5. Data corruption. If something went terribly wrong, and the server was writing corrupted data, it could conceivably destroy a lot of data before your monitoring knew something was wrong. Hmm. Restoring from backup is the obvious thing to do, although you might have incremental data loss from the window since the last backup (assuming transaction logs corrupt also).

Like I said, I like MobileMe, although it’s certainly taking a PR beating right now. But I’m definitely curious what’s going on, and what could possibly take 5+ days to recover from. Any ideas?

iPhone and Exchange – push and DNS

It seems lot of folks are having problems getting Exchange push email working reliably with the new iPhone 2.0 software. For me, it worked flawlessly when I was outside of the office, but when I was in the office and connected to our corporate LAN via wi-fi, it was unreliable at best. If I instead connected to another wi-fi network (like the guest network from the folks two floors below us), everything worked fine.

There is an Apple KB article talking about this:

When roaming between home and office networks with Wi-Fi enabled, “push” may stop working if your company’s Exchange ActiveSync server has a different IP address for intranet and Internet clients. Make sure the DNS for your network returns a single, externally-routable address to the Exchange ActiveSync server for both intranet and Internet clients. This is required so the device can use the same IP address for communicating with the server when both types of connections are active. A workaround to avoid this issue is to disable Wi-Fi on the iPhone.

Yep, that sounded like the problem. Our internal and external DNS for our corporate mail server is different, just as the article surmised. But it turns out in our case, it was non-trivial to change them to be the same thing.

But it turns out there is a workaround that works for me. If you go into Settings / Wi-Fi, find your wi-fi network, and click the blue button next to it, you’ll see something like the following:


And here’s the tricky part. Tap on the “DNS” setting, and edit it. In my case, rather than using the internal DHCP-assigned DNS servers, I typed in two external DNS servers. These new servers will override whatever is returned from DHCP, and when asked for the IP of our mail server, they will return the externally-facing IP, since that’s all they know about.

And that was it! The push email is now working 100% reliably. A little too reliably, actually. :-)

Note – I obviously no longer have internal DNS resolution within my corporate LAN, but that’s not a problem for me. Your mileage may vary.

Note 2 – this also assumes your external mail server IP is accessible from your LAN. This may or may not be the case, depending on how your firewalls and the rest of your network are configured.

NGES in 60 seconds – group clippings

Last time, we looked at how clippings work in NewsGator Enterprise Server.  At the end of the video, I mentioned that there are other ways to share clippings with other folks…I was referring to a feature called group clippings.

Group clippings are basically clippings that are shared among all the members of a group.  That group might be an Active Directory (or LDAP) group, or it might be an ad-hoc group defined in NGES.  The clippings for that group are, by default, private to that group – so you can have a confidential conversation among the members.

View QuickTime video

YouTube version:

And as always, you can try all this out for yourself by downloading the free 20-user version of NewsGator Enterprise Server.

iPhone template for Delicious Library 2

One of the first Mac apps I bought was Delicious Library. Not that I desperately needed it, but it was just too darn pretty not to buy, especially for a new Mac owner to use to show his friends. And when version 2 came out, I was super excited about being able to publish my library on the web.

Here’s my use case. I travel a lot, and I like to read books on the plane, rather than do my usual work…somehow I feel more relaxed when I arrive to where I’m going. Anyway, so I’ll be in the book store in the airport, and I’ll see an interesting book. But if it’s not a new release, then I often can’t remember whether I’ve read it before; I know I like the author, and I know I’ve read a lot of his work, but not positive about this particular book.

So DL2 and the iPhone to the rescue, right? Pull up my library, do a quick search, and I’d be all set. But herein lies the problem.

The out-of-the-box templates can be seen on Adam Betts’ blog. While pretty, here are the issues with the iPhone templates, at least for my particular use case:

1. There are only 12 books per page. Yikes – that means I have 20-some pages, and I’m not sure which page I need to go to. Or with the other template, all the books are on one page – which is nice, but leads to problem #2.

2. See all those pictures of the book covers? Those are coming to about 90KB each. Each! That means a page of 12 books is over half a meg. That’s a lot on a phone.

3. I don’t need the book descriptions and links to Amazon in there either. Those are pretty big also, although down in the noise compared to the image sizes.

What I really wanted was one page with all my books, or at least 100 or so of them, and have the page be small enough that I could load it over EDGE without having to sit down. I’d like search features too, and I’d like to be able to sort by author’s last name (so Tom Clancy would be before Brad Meltzer), but hey, there are bigger problems to worry about.

While I was browsing around, trying to figure out a way around this problem, I saw Mark Burgess’ site, and he had some sample templates. While his template wasn’t what I wanted, it showed me how to make one. So I worked on it for a bit, basically just modifying the one that ships with DL2, and this is what I ended up with:


Over 200 books on one page, and it’s coming in at just over 200KB including the images on the page. Now that I can live with!

As to how to install it…it’s not super easy. Here’s what you need to do:

1. Install Mark Burgess’s HTML template, and install it per his instructions (in the readme file).

2. Download my changes, and replace the files in the “iphone” directory in Mark’s template with the contents of this zip file.

No warranty, of course…this is at the “it seems to work for me” stage. :-)

If there is some documentation somewhere about how to make DL2 templates and get them installed without resorting to such hackery, I’d love to know about it, and then I can make this into a real template.

Anyway – hope this is helpful for someone!

NetNewsWire for iPhone now available!

The development of NetNewsWire for iPhone just might be the worst-kept secret in NewsGator’s history. But hey – that’s ok, especially since today is launch day!

NetNewsWire for iPhone uses the NewsGator Online sync platform, so you can use NetNewsWire on your Mac, FeedDemon on Windows, NewsGator Online on the web, or any of our other applications, and they will all sync together. I’ve talked about this a lot in the past…but this is really the only reasonable way to build a mobile reader, IMHO. We use mobile devices as companion devices, in addition to our desktop and/or laptop computers – they’re not the only devices we use.

The first thing to do is head on into the iTunes App Store (click the image below):


Assuming you have iTunes 7.7 installed, that link will take you to the App store, where you can download NetNewsWire:


And then you’re all set. Here are a few screenshots while you’re waiting for your iPhone 2.0 firmware to download!

Main screen:


News items:


Reading an article:


This new app also supports clippings – so you can find articles on your phone that you’d like to read later, and save them in your clippings folders. Those folders are also synchronized to your desktop apps and the online system as well.

If I sound excited, it’s because I am. :-) So stop reading now, and go to it!

And did I mention it’s free?

NGES in 60 seconds – clippings

Onward, I say. Time for the next installment in the series, this one covering clippings.

There’s a lot of buzz about clippings in NewsGator Online (and shared items in Google Reader), especially with apps like ReadBurner making them easy to discover. There are similar use cases for clippings in the enterprise.

NewsGator Enterprise Server supports two kinds of clippings – personal clippings, and group clippings (which are shared by a group). This video covers personal clippings; group clippings will be coming next time.

View QuickTime video

YouTube version:

And don’t forget you can download a free 20-user version of NewsGator Enterprise Server. :-)