Twitter has had a bad few days. After a few weeks of wonderful stability, we started to see some Fail Whales. Then people started losing followers and subscriptions. Because of this, even more people are moving to alternative micro-blogging services like Identi.ca. Of course there have been several requests for a Twitter friend importer. I think people are getting caught up in the problem and not thinking about the best solution. Previously, I mentioned that Twitter was at the crossroads. In that post I started thinking about a possible solution:It is now late May and they are just admitting that they have infrastructure problems. Obviously, there are the redundancy and replication issues that can be solved with known techniques. The realtime API needs to be using a replicated database and not affect the main database. This is the part that is concerning me the most as it should have been obvious some time ago.Another post appeared from Dave Winer where he mentioned that we are inching toward federation.
Right now, today I'm using an approximation to the ideal system. I try to enter my original post on FriendFeed, then I have an agent script running on one of my machines that routes it to Twitter and Identi.ca, with a pointer to the discussion thread on FriendFeed, shortened by bit.ly.Obviously, this is not a good solution to the Twitter problem either. There are also multi-posting services like Ping.fm and Posty which allow a user to update each of their micro-blogging services with the same message. So, with one click you can update Twitter, Identi.ca, Pownce, Jaiku and Plurk. While that sounds really cool, they only provide one-way updates. Tools like Twhirl are starting to allow posting and retrieving of tweets and dents in one application. This gets a little better, but you still have multiple accounts to deal with.
Federated Microblogging
Federation is the real answer. What does federation do and why is it different from Twitter creating a distributed architecture? A distributed architecture means you have various pieces of the application sitting on different servers. Typically, several parts are replicated or redundant and there are various load balancing devices thrown in to complete the architecture. I am not going to go into tons of detail as books have been written about distributed architectures.
Federation is different from distributed in one simple sense. Federation requires a full copy of the entire system. Federation is the cooperation between various systems to act as one. Wikipedia has a good description of a federated database system.
A federated database system is a type of meta-database management system (DBMS) which transparently integrates multiple autonomous database systems into a single federated database. The constituent databases are interconnected via a computer network, and may be geographically decentralized. Since the constituent database systems remain autonomous, a federated database system is a contrastable alternative to the (sometimes daunting) task of merging together several disparate databases.If you replace all of the database and database system terminology with microblogging and microblogging system you will understand a federated system. So, how does this help Twitter? It does not, at least not directly. By having a federated system, people can move to Identi.ca or the next clone that appears and not lose their followers. If you think about the concepts of email, you will understand how the federated system would work. My account in the federated system would be robdiana@identi.ca. A Twitter user could reply from within Twitter directly to me (robdiana@identi.ca) and I would receive it in whatever client application I had. I could also follow Twitter users, Pownce users or any other microblogging user. By enabling people to communicate across platforms would mean that the server load would be spread across various services, thus decreasing the amount of traffic on Twitter directly.
The one part of the federation that is missing is the routing between servers. This can be accomplished by following the DNS model. A local DNS server has a reference to its parent or master server. This allows new servers to be built and their location and IP address propagated to other servers. This is a very effective solution and it has worked for several years.
Because Twitter and Identi.ca follow the same API, they could act as the root servers in the system. The other option to the Twitter API is the XMPP protocol, which Twitter is using for their realtime API. The protocol choice is important because a federated system requires that all of the systems speak the same language. A federated system would also require a different client application than we have now. As I mentioned earlier, Twhirl fully supports Identi.ca and Twitter. However you are dealing with two different accounts. How cool would it be if you had one microblogging client to chat with people on any service?