About 16 months ago, I wrote NewsGator platform roadmap – where are we going?  It’s a little hard to believe it’s been that long…it seems like yesterday.  In any case, though, it’s about time to talk about this again, as we’ve come a long way in these last 16 months, and we’ve got a lot of exciting things planned. And it’s worth pointing out that everything I said we were going to do last time, we’ve done…so the roadmap track record is 100%. :-)

I’ve been working on this post for a while, and finally hit on the problem that’s kept me from finishing it sooner.  There has been so much going on these past 16 months, in so many different business lines, that most folks aren’t aware of all of it – and this makes talking about the roadmap difficult. So in this first part of the roadmap, I’m going to take a look back – and try to give you a baseline with which I can talk in part II about where we’re going.  Hopefully this won’t sound like a commercial – that’s not the intent; I just want to give you the background against which the forward-looking post will make more sense.

So let’s get started!

NewsGator Inbox, FeedDemon, NetNewsWire

What can I say about these products in the last year?  NewsGator Inbox got its shiny new name, and a completely revamped synchronization system where just about everything is synchronized.  FeedDemon and NetNewsWire joined the NewsGator family, and both have since had new releases (2.0 and 2.1 respectively) that add synchronization and other online-supported features.  It’s been a busy time for all of these apps…and frankly, I’ve been floored with the excitement level surrounding these products – it hasn’t let up even a little.

The synchronization is now tight and fast.  We’re on our third-generation sync platform, and it behaves exactly the way you think it would.  Read a post in FeedDemon, and it shows up read in NetNewsWire.  And mobile.  And the web.  And…well, you get it.  We’ve had a few hiccups with this system (including a fairly significant outage), but we think we’re past the worst of it.  And even with the occasional problem, the net gain for the user from this system is worth it, at least according to most of the feedback we get.

I’ve said many times that part of our product strategy is to have an awesome user experience on every platform.  People have asked why we didn’t just build a cross-platform client application in java or something, that would run on every platform.  Well, we’ve taken a different strategy.  Rather than take a lowest-common-denominator approach with a single application, we’ve instead chosen to have best-of-breed on every platform.  FeedDemon (for Windows), NetNewsWire (for Mac OS X), and Inbox (for Outlook) all provide this kick-ass experience in their respective environments, and each is the market leader in its segment.

NewsGator Mobile

Mobile is a big part of our product strategy – both for the consumer and enterprise markets.  I think RSS and mobile is a match made in heaven.  BUT – and this is a big but – I think a mobile RSS reader is useless unless it is synchronized with the other devices you use.  No one really wants to read all of their feeds on a mobile device; they want to read some of them there, and they want to be sure they don’t have to download something onto their phone that they just read in the office.

We’ve had our mobile HTML reader for some time, and it’s hugely popular.  It’s such a simple application, and yet so useful.  But HTML applications alone don’t really cover all of the use cases we want to hit.  What about rich functionality, like clipping, emailing, forwarding, and other per-article functions?  What about when you’re on a plane, with no wireless service, and you’d like to read your feeds?

Enter our new mobile applications.  The first is NewsGator Mobile for Windows Mobile devices – this is in beta right now, and delivers a rich and synchronized experience to Windows-based mobile devices.  It’s a great user experience – if you haven’t tried it, I’d encourage you to download the beta and give it a shot.

NewsGator core platform

At the heart of our online system is what we affectionately call the NewsGator core platform.  It needs a cool code name, I suppose…but even with a boring name, it’s evolved into an incredibly powerful beast. :-)  And growing very fast.

This platform provides content retrieval and archival, index/search, subscription management, synchronization (including incremental), analytics, and other base-level services.  These services are exposed for our own web applications (such as NewsGator Online), but are also exposed via web services API’s (SOAP and REST both).  These API’s are the same ones that our own client products use, including Inbox, FeedDemon, NetNewsWire, and our mobile products (more on those below).  Our private label platform also uses these API’s.

I talked about the synchronization system above…but one more thing to point out here.  It’s fast.  If you’re running FeedDemon, with several hundred feeds, it used to take a while to refresh all of those feeds, especially on a slow connection.  But with the latest version, which uses the new sync system, only the incremental updates have to be sent to the client…which means you can refresh hundreds of feeds in just a few seconds.

NewsGator Online

This is our online application, the one you see when you sign in at http://www.newsgator.com.  It’s growing at a huge rate – as we’ve added features, and added sync with client products into the core platform, users have been signing up en masse…it’s exciting to watch.

Most of what I have to say here is where we’re going – so I’ll save that for part II.

NewsGator Enterprise Server

My roadmap post 16 months ago was the first time we uttered its name publicly…but now NGES is shipping (actually in its fourth release, 1.3), and it’s getting some serious adoption in medium and large companies alike.  Enterprise RSS usage is skyrocketing – the use cases are clear, and and value proposition is compelling.

NGES is usually sold as a shrink-wrap product, to be installed on the customer’s servers.  It’s also available as an appliance – although interestingly, we don’t get a lot of demand for that.  A few smaller customers find the appliance idea appealing…but in the larger installations, and these are six-figure deals we’re talking about, we’ve found nearly all of our customers want to install on their own equipment.

NewsGator Private Label Platform

This was a sleeper – it came out of nowhere.  And now it’s a significant part of our business.

Publishers today, and other site operators, are very committed to doing what they need to do to drive additional traffic to their sites, and to increase customer loyalty.  The industry has been telling them “build RSS feeds, that’s the answer.”  This is great – but it’s a double edged sword…what happens is many folks consume their content in RSS-enabled consumer portal sites like My Yahoo! and others.

Additionally, there is a conversation happening, in the blogosphere or elsewhere, around the content produced by that publisher – but it’s not easy for them to find and surface this information in a way that’s useful for their readers.

These problems and others are what the NewsGator Private Label platform was built for.  Publishers (or anyone else, for that matter) can add anything from a complete RSS reading experience for their users, to exposing conversational information (or “hot topics” and the like).  They can either do this in a very custom manner, using the NewsGator platform API, or they can choose a turnkey solution and get up and running in literally days.  Examples of real customers of this platform include My USA Today, and My Newsweek – both are powered by the NewsGator private label platform, and they deliver completely different user experiences, designed by the companies themselves.

So now we should all be on the same page…in my next post, later this week, I’ll lay out the future for all of these product lines.

The last 12 hours will not go down in the books as NewsGator’s happiest hours…as posted here, there was an outage and NewsGator Online was down for several hours.  On a normal day, I’d be thrilled that “NewsGator” was the #1 search on Technorati; but not today. :-(

But while this was painful for us, it was painful for our users too.  So I’m going to walk through what actually happened, in quite some detail, and why it took so long to come back online.  And there are still a lot of questions here – our platform/operations folks are on the phone with EMC and Microsoft both as I type this post, trying to get to the root cause of all this.  But I’m getting ahead of myself.

Warning – lots of detail ahead.  Feel free to skip below if you don’t want to read all the details.

Monday 5:00pm

A query starts running against the database that contains user information, kicked off by an automated agent job. This query is run periodically, and is used for some reporting data.  It normally takes a minute or so to run…but this time, for some reason (I don’t have the details), it was going to block on something and take 300+ minutes.

Monday 6:00pm

SQL transaction log files are growing much more quickly than normal, due to the errant query.  It’s not yet noticed, however, as it’s still below the relatively high log monitoring thresholds we have set up.

Monday 7:00pm 

I’m sitting in the office with Darryl, who runs our platform team, and owns all of the non-GUI portions of the online platform.  We’re noticing large I/O bottlenecks on one file group in one of our databases, and some quick calculations indicate we are at basically 100% of the hardware capacity.  This has been consistent for the past week or so.

We use an EMC fiber-channel SAN for database storage.  The database file group that was running into problems is striped across 20 disk spindles.  We had a plan in place to increase the number of spindles to increase I/O capacity, which was going to happen later this week…but given the data at hand, we decided to schedule it for Monday night.  The immediate plan was to add 10 more disks to the stripe for that file group, bringing the total up to 30, and increasing I/O capacity for the group by 50%.  (For techies out there, we’re measuring I/O operations per second; physical disks have a maximum number of these operations they can handle per second, and the SAN caching increases the effective rate in some cases…but generally, the only way to really increase I/O capacity is to add drives).

So…we allocate the extra drives, and begin the re-striping process.  This happens transparently to the applications (i.e. SQL Server), as the SAN has the ability to re-stripe on the fly.  We’ve expanded storage a number of times, exactly like this, with no problems.  Total I/O capacity is impacted a bit during the process, but everything stays up and running.

Or at least it’s supposed to.

Monday 10:00pm

System starts throwing intermittent errors when applications try to connect to the database in question.  Monitor alerts are sent.  It’s pretty critical, as this database contains all user-related data (such as authentication info) for all system users.

Monday 10:30pm

Darryl discovers the errant query (the one that started at 5pm), and terminates it.  At this point the SQL transaction logs are well over 300GB.  SQL starts to recover, but intermittent errors still occur.

Tuesday 2:00am

SQL Server shuts down – cause is as of yet unknown, although there are some suspicious log entries prior to the failure.  It automatically restarts, and begins the recovery process to bring the database online, including replaying this huge 300GB log.  Other cluster members don’t take over, because the database requires recovery – as opposed to having a server problem.

However, while it’s trying to recover, a huge amount of traffic is still coming in from the web sites (mostly API and non-US sites at this time), which causes a large number of SQL transactions.  Some of these transactions deadlock with the recovery process, which causes SQL to terminate again and restart recovery.

At this point, every request coming into the system that hits that particular database fails.

Tuesday 2:30am

It’s clear by now that the only way to bring the database back online is to eliminate all database traffic.  “Maintenance” pages go up, all sites come down, and the database restarts its recovery process.

Tuesday 10:15am

SQL Server completes the recovery process and brings this database online.  Back-end services are turned back on in priority order, and sites are brought back online by 10:30am.

So there you have it. I’m not a database expert myself, so I may have some of these things not quite right, but this is more or less what happened.  The root cause, and which details along the way are actually relevant, is as of yet unclear.

The big question is why did SQL Server fail completely?  One possible cause would be if the SAN expansion caused some disk errors, and left the database in an inconsistent state.  So as you can imagine, we have tickets open with both EMC and Microsoft, and we will get to the bottom of this.  There are also other issues – such as the intermittent failures – that are cause for concern.

Frankly, this was a pretty frustrating experience.  We have a lot of redundant systems – pretty much any piece of hardware in our data center could fail, and we can absorb it without a significant outage.  For example, if an entire SQL box would have lost power, fallen on the floor, and broken into pieces, no problem, we’d have an approximately 10 second outage.  But this case, where the database gets into an inconsistent state, wasn’t helped by the redundant systems.

Moving forward, we’re doing a number of things. Specifically, we still need to get to the root cause of this particular problem, but in general, we’re working a lot on our back-end systems to build additional redundancy, scalability, performance, and resiliency.  We’re going to partition data in different ways, so a failure like this will only affect a portion of the user base, a portion of the content, or a portion of the functionality.  Our back-end cache servers will expand to be able to serve data while underlying data store is offline.  And lots more, which I won’t go into details about just now, since this post is getting a little long.

But for now – yep, we screwed up.  To our customers who were affected – you have my personal apology, and my assurance that we’re doing everything we can to ensure this doesn’t happen again.

I got an email today, which was sent to several aliases at newsgator.com.  I was typing up a response, when I noticed the same text of the note on a comment to the recent TalkCrunch podcast (which featured executives from four different RSS-related companies, including myself for NewsGator)…and given that the comment was made in public, I thought I would respond publicly as well.  Here’s the comment in its entirety:

John Pywtorak
April 5th, 2006 at 11:24 am

“Greg Reinacker, Founder and CTO, NewsGator” sounded like a Microsoft fanboy.

To suggest hat IE7 with RSS will be revolutionary and change the web is marketing speak and completely wrong. What about Firefox and other RSS readers? What about iTunes? I think it will have a longer and greater mpact than IE7.

From his discussion I can only come to the conclusion that your service is IE specific and that is the browser you will provide biased support for. I thought of using your service, but now I will not.

This is the first time I’m aware I’ve been called a fanboy. :-)  Here is my response:

John,

Sorry you feel that way.  Let me try to clarify what I was trying to say, and if you disagree, then we can agree to disagree.

I believe we’re in the middle (er, beginning) of a paradigm shift on the internet.  And that shift isn’t about RSS; it’s not about Atom.  It’s about subscribing to content.  Users will come to expect that when they visit a web page, there is a consistent way to subscribe to the content on that page.  If they can’t subscribe to some particular site’s content, they’ll consider it a bug.  They will likely visit their favorite search engines less often – and they will “cruise” less – because the content they care about is being delivered to them.

Microsoft is going to ship IE7 soon, and Windows Vista (including IE7) sometime after that.  Now here’s the thing.  I don’t think IE7 is the best experience for reading this content the user has subscribed to.  But – and this is a big but – it’s going to be the first exposure to RSS and subscriptions that millions of people see.

Tidbit – Microsoft said at one point they plan to ship 475 million units of Vista in the first 24 months.  That’s a lot.  If I add up the total number of people in the world that have ever heard of RSS, I bet it’s a lot less than 475 million.

Anyway, back to my point.  People aren’t going to look for subscribe buttons on a site any more; they’re going to look for “subscribe” buttons in the browser or client they’re using.  That browser might be Firefox, it might be Safari, it might be IE7.  But out of those three, IE is probably going to have the most market share.  That’s not fanboy talk – that’s real life.

So back to NewsGator.  We’ve announced that we will have synchronization support for the RSS platform that’s contained in IE7 and Vista – I demonstrated it personally on stage at MIX06.  This means people can use the cool auto-discovery and subscribe tools in the browser, and they can use as much or as little of the IE7 reading experience as they wish.  We’ll synchronize all of that information to either NewsGator Online, or their internal NewsGator Enterprise Server installation.  Once there, they can use a _different_ Vista/IE7 machine to read it, or use their mobile device, or use a rich desktop client like FeedDemon or NetNewsWire, or use a web client from a shared computer, or…you get the point.  We take the RSS experience and extend it off of the user’s single machine, while at the same time leveraging the strengths of IE7.

Does this mean we only support IE?  Does this mean we don’t love Firefox, Safari, and other browsers?  No!  Our system supports whatever browser you want to use.  Where possible, we integrate with the subscription and auto-discovery mechanisms in all of these browsers to varying degrees.  We’re not “biased” in any way – other than we devote resources to the platforms that our customers ask for.

So anyway, John, and everyone else reading this – I’m happy to continue this discussion with you if you’d like.  Just leave a comment below, write on your own blog, or you can even call me – 303-552-3900 – and we can chat.

Wow, what a crazy month it has been.  As I mentioned last week, we launched new versions of FeedDemon and Inbox last week, and the NetNewsWire beta is getting more and more widespread…and the new look for NewsGator Online has been extremely well received.

On a less visible front, our private label business is really ramping up.  Some clients have gone live, and others are running private betas.

All of this has combined to create a 30% growth in certain key load-related metrics of our online platform in the last 30 days.  You read that right – 30% growth.  Wow.  Here’s another tidbit – our daily new-subscriber numbers are more than triple what they were 30 days ago.

We of course plan for growth, many months out.  We know what our average growth rates are, and we add capacity accordingly.  The challenge here has been that we didn’t expect this dramatic growth in March – and we were basically caught a little behind on our capacity planning.  The good news, however, is that everything has stayed up and running well.  And if you’re using a client like FeedDemon or NewsGator Inbox, and synchronizing with NewsGator Online, you’re probably not noticing anything at all.

No worries, though. At certain times during the day, the system is a little bit slower than normal for certain users…but we’re adding significant capacity as quickly as we can get it shipped and installed.  Within the next week or so, everything should be back to normal!

NewsGator has had a web API using SOAP for some time.  It’s been very popular – it’s used in our own clients (NewsGator Inbox, FeedDemon, and NetNewsWire), third party clients such as RSS Bandit, and private label customers (online, but unlaunched as of yet). We currently process millions of API calls per day in our online system, and the API is also implemented in NewsGator Enterprise Server.

There has been quite a bit of feedback from developers about the API; most of this feedback centered around the lack of a REST version of the API.  We’ve found that some developers have never used SOAP, and there is a learning curve on certain platforms to learn a SOAP toolkit; other developers just don’t want the extra complication in the messages, and prefer a REST solution.

Whatever your reasoning, though, you can now choose for yourself. We now have both SOAP and REST versions of the NewsGator Online API available.  The SOAP version has been around for a while, and is used by our own products.  The REST version is brand new, is documented here, and enters a public beta today.  So try it out, and let us know what you think – we’re looking for feedback!

Well, the teams are exhausted :-), but this is a HUGE multi-product release day!

First, FeedDemon 2.0.  It’s been a long time in coming, but now it’s done.  It of course includes FeedStation 2.0…but FeedStation is also available as a separate download if you want to grab it separately to use just with NewsGator Online or NewsGator Inbox.  A huge thanks and congrats to Nick Bradbury, who I think has been on a caffiene high for quite a while getting this release ready.

Next, NewsGator Inbox 2.6.  As I mentioned earlier, we changed the name of NewsGator Outlook edition to NewsGator Inbox…and added a ton of new features that folks have been asking for.  More thanks and congrats here go to Nick Harris (who still doesn’t have a blog, but now that I’ve mentioned it twice maybe he will!), who I think worked himself right into the ground preparing this release with his team.

NewsGator Online now has a whole new look.  It adds a number of small features, like favicons and such…but the real news here is it has set us up to quickly add a ton of new features and capabilities that are on the backlog.  And the purchase process for new products is finally as easy as can be, integrating multiple products and subscriptions in a way that makes sense, and old subscription plans have been consolidated into the new catalog of options.  A huge thanks and congrats here to the online team – Tim, Demi, Noel, Dede, Brian, and James…and to the platform guys – Darryl, Glenn, Jeff, Gordon, Chris, and Rob.

And finally, as of these releases we’ve made good on our promise to abolish the subscription model as you used to know it.  Now, you can buy FeedDemon or NewsGator Inbox (and soon NetNewsWire) for $29.95, and you’ll get a free year subscription which includes value-added services such as synchronization (and lots more to come).  At the end of that year, you can choose to renew your subscription if you wish for $19.95.  But if you choose not to renew, your software will continue to work forever – just without the online-specific features.

So all in all, a busy day…let us know what you think!

Quite a day today…the product that started NewsGator, NewsGator Outlook edition, has a new name!  It’s now called NewsGator Inbox, and v2.6 release candidate 1 is available now.  This product has sure come a long way since the first prototype, and the 1.0 release, over 3 years ago.  Congrats to Nick Harris, the lead developer on NewsGator Inbox, who doesn’t have a blog I can link to (but maybe this will shame him into it! :-)

There’s a LOT of new features, including clippings sync (the first NewsGator product to do this), greatly enhanced podcasting support, improved synchronization with the NewsGator platform, and too much more to mention.  And of course, the synchronization in NewsGator Inbox works similarly to our other products – not only feed sync, but sync of individual items and read-states, and dramatic performance gains over “traditional” aggregators.

So take a look – details are in the forum post here!

You know it’s a busy day when we have a mobile announcement as cool as this and I’m the last to blog about it. :-)

So here’s the deal. NewsGator is into mobile in a big way, and we’ve had a mobile HTML reader for quite some time (hint – if you’re an existing subscriber, go here to find the mobile configuration area). But it’s time for much more than this.

Today, we announced a couple of things. First, we acquired SmartFeed and SmartRead for Windows Mobile, and hired Kevin Cawley (their creator) in the process. You can download the free previews of the applications right now; we’re working on a number of changes right now, including making them dramatically faster, and we expect to have a public beta available in the next month or so.

At the same time, we also announced our J2ME reader, for java-enabled devices.  This should be available in beta right around the same time, and will provide a similar experience to the windows mobile reader.

But why?

Well, here’s the thing.  We’ve always been about “any time, any place, any device.”  We’ve had a great story about it up to now, with rich applications for the web, Outlook, Windows, the Mac, Windows Media Center, and a lighter experience for HTML-capable mobile devices.  All of these applications sync together (FeedDemon and NetNewsWire in their current betas), and it’s really a great experience.

But on mobile devices, we found the experience wanting.  Using the HTML mobile app ourselves, and watching others use it, it’s very handy – but not as streamlined as it could be.  And there are certain missing functions – saving items to clippings, etc.  Some of these features are being added, but some are difficult to implement effectively in the confines of a mobile browser.

Enter the rich mobile clients.  These clients provide a great experience, tailored for the device you’re using.  Download items, save them, even browse for new feeds; they really take the mobile content experience to the next level.  And they can provide better responsiveness, even on low bandwidth connections like GPRS.

But there are already mobile clients out there; why NewsGator?

Well here’s the thing.  And I’m sure I’ll get some raised eyebrows for this, and at least one hate mail…but in my opinion, a mobile RSS aggregator is useless if it’s not synchronized with the rest of your life.

Useless?  Yep.  Other than the first week after we buy a new phone or PDA, when we play with absolutely everything, most of us use mobile devices as an extension of our everyday lives.  It’s not the only device we use – it’s a companion device that we use when we’re not sitting at our desk, or with our laptop on.  And it’s certainly not the only place we want to consume content.

When these mobile applications are synchronized with NewsGator Online and the rest of the suite of products, though, they become much more powerful.  Now it’s synchronized with your primary content application (be it NG/Online, FeedDemon, NetNewsWire, Outlook, or whatever other application you use), and you can use it as a companion device.  It’s amazingly efficient.

If you’re going to MIX, be sure to come to the RSS talk, where you’ll see current (and private) builds of both the NewsGator windows mobile application, and the new version of NetNewsWire!  And _much_ more – I’ll write some more about this next week.

And in the meantime…make sure your mobile data plan is active, and watch for NewsGator’s mobile beta launches.  And a big “welcome” to Kevin Cawley!

Cool!

SharePointGator

I’d been working on this webpart ages ago to enter into the NewsGator API competition but didn’t end up getting it finished in time to submit it. Left it sitting in the projects folder for a couple of months until I decided to finish it this weekend as I was feeling a little ill and didn’t want to leave the house!

So what’s it all about then? Well I called it SharePointGator (no doubt infringing on some copywrite), as it’s a NewsGator webpart for SharePoint. Using the NewsGator webservice and your newsgator account details you can now view the rss feeds and unread blog posts from your newsgator account in SharePoint.

What do you get when you put a client guy (Nick Bradbury) and an API guy (Gordon Weakliem) in a room together for an hour?

Insanely fast content retrieval/sync.

Last night I was playing with the new public FeedDemon 2.0 beta 2 release.  I downloaded my content, read through it, and a while later did one last full retrieve before leaving.  I have about 150 feeds.  How long do you suppose that last full retrieve took?

About 4 seconds.

How could it be this fast?  Well, in the normal case when you click “refresh” in an aggregator with 150 feeds, it will go hit those feeds and see if there is new content.  Many of them will return 304’s indicating they’re not modified, but even that takes time.

With the NewsGator API, you make one call that asks which of your feeds have updated content that you don’t already know about.  In my situation above, it said there were six such feeds.  It then retrieved content for those six feeds…and all this took a total of about 4 seconds.  150 feeds completely up to date – with very little bandwidth used, in very little time.  Very cool.

And a tease for the Mac folks out there waiting for a new build of NetNewsWire – Brent Simmons was in the same room with Nick and Gordon. :-)  It won’t be too long now!