If you take a step back and think about popular Web services, be they RSS feed readers, communications tools, social networks, link aggregators or lifestreams, their core deliverable is data. The way they are differentiating is through enabling that data to be accessed by more people, and manipulated in new ways. But as their popularity grows, with more users, more features, and richer media, be it photographs, music, or video, the data can spiral out of control, demanding larger databases, more servers, faster networks, better caching... you name it. And if they don't deal with it? Death. The services can slow down or crash, and users will move on.Not every Web service has had the structure, the planning or the money of Google, so not every site can keep going with all features, saving all data, forever and ever.
Twitter's recent struggles keeping up with user growth and resulting issues have been extremely visible. To cope with the load, the microblogging service started whacking features one by one. At various points, Twitter turned off the @Replies tab, removed pagination, reduced the number of times that external services could access its API, and eliminated the use of instant messaging services. Limping along, Twitter has slowly returned to speed, as one by one, most of the pieces have been turned back on, but not everybody is happy about it.
But don't think Twitter's alone. Just yesterday, I logged into SocialThing, a lifestreaming service thought by many to have a better GUI for FriendFeed, and found that it had been forced to scale back its offering as users grew.
Rather than supporting many different services, SocialThing only lets me include Twitter, Flickr and Facebook, with LiveJournal, Pownce and Vimeo also being options. But the many other sites I use? Blocked. They write, "We have had to temporarily disable a few of our services, due to user growth. We hope to have them back up soon!"
At a time when FriendFeed is gaining notoriety for supporting rapid access to the streaming of 35 different services without slowdown, that kind of scaling down can't be good.
For Shyftr, a new RSS reader I've covered quite a bit here, they too are seeing the challenge of user growth. As we saw with Mergelab's closure, the act of tracking thousands of feeds and storing their data for an infinite amount of time can be very challenging. I've already seen Toluu upgrade their server farm, and Shyftr has done the same recently. But more changes are coming to handle the rising user base.
As founder Dave Stanley wrote in a blog post, Notice: System Changes and New Implementations, Shyftr will be removing "a substantial portion of backdated data" which should reduce recent issues with database performance, and let them turn on services that had been disabled thanks to site slow-downs, including search. As he said, "It's time we trim some fat while making the system leaner, more efficient, and better prepared for future growth."
If you have a Shyftr login, see the full post:
Notice: System Changes and New Implementations
I expect that the rules for small companies are different than the rules for big companies. Twitter has gotten such a bad response for its issues in part due to its popularity, and in part due to its criticality for some communications. For smaller sites like Shyftr and SocialThing, users are more likely to be accepting of temporary scale-backs or data removal. But if the big guys like Yahoo!, Google, Apple or Microsoft, took the route of deleting data and reducing features, it'd be likely that we'd see an uproar.
So how should they handle it? Do you, as a user, have an absolute right to all your archives, ever? Do you have the right to demand that once features are added, they never be removed? And do services have an obligation to scale up their hardware and infrastructure to keep you happy? Would you help fund their upgrades? Handling growth is critical, in my opinion, and those best prepared for scaling will leapfrog those who are forced to "lay off" their data. We're wishing Twitter luck, but there are a lot of smaller services struggling with similar issues.