smugmug: FUBAR

What was supposed to be a quick, feature-rich update has turned into hours and hours of downtime. Professional photographers are losing money; the smugmug people are remaining cheerful, somehow, but they've all been up for over 24 hours trying to get it working.

Here's the latest update:

For the technically curious, one of the new features required restructuring and reindexing our main image database.

Since that's over 32,000,000 photos, we conservatively thought it would take many hours to complete.

It turns out we weren't conservative enough, and it's taking many many hours to complete. I suspect some sort of hardware problem we haven't seen before, but we won't know until it's finished.

We think the feature will be a great one - you'll now be able to add geography information to your photos, allowing you to view your photos overlaid on Google Maps and Google Earth. So you can quickly and easily view your latest trip to Hawaii, for example, complete with a timeline animation that moves from photo to photo in the order you shot them.

I think it's worth mentioning that our track record for our schedule maintenance window , up to this point, has been pretty stellar. That doesn't excuse this morning - nothing does, and we'll take steps in the future to make sure this never happens.

Thanks again for your patience, and we'll continue to post more as we get it.

Don MacAskill
CEO, smugmug

I'm excited about the upgrade. Really excited. The new default theme looks pretty slick and feature-rich, and I'm even starting to want to create my own CSS for my galleries. (I'm sure that once this upgrade is in place, users will be able to create all kinds of styles, and hopefully submit them to a central database where everyone can use them.)

smugmug's getting some negative feedback due to their lack of foresight, and I hope that doesn't hurt them too badly. I hope it does, however, inspire them to establish systems for upgrades that don't require the site to be offline all day, and encourage them to give users--all users, not just people who've altered their stylesheets--a fair amount of warning before massive upgrades.

After all, shit happens.