August 17, 2009

Progress Being Made On Twitter's Network After 10 Days of Attacks

After an initial wave of attacks brought the popular micromessaging service to a standstill, as well as a host of other social networking sites, on August 7th, the Twitter team has been fighting to turn the tide and restore access for the service's customers and developers to near-normalcy. Ten days into the attacks, they have not disappeared, according to postings from Twitter, but it looks like the company may have turned a corner in dealing with the additional stress from the distributed denial of service.

In a post to the company's Twitter Development Talk forum titled "Platform Status Update" entered at 4:30 this afternoon, Twitter's Ryan Sarver said:
"Over the past ten days we have been dealing with a lot of stress on our network and that has caused a number of our partners to be knocked offline for extended periods of time. This is obviously not something we want to happen and the platform and ops teams have been working hard throughout that time to address the needs of our ecosystem while protecting the system as a whole. We have made some great progress today in tuning the system to a point that should allow our partners to operate as they were before the recent issue began."
He added "the system is still under stress", and asked developers who might be continuing to have issues to work with them and provide information around what IP addresses their machines use to access the Twitter API, the method they are using to make requests, and any other information - including data as granular as their host operating system, browser, cookies and network connection.

Ryan's comments followed a note from earlier in the afternoon by Alex Payne, the company's platorm lead, where he also said the company continues to "fend off" the attack, saying that without specifics from developers (like those listed by Ryan), they could not adequately troubleshoot any issues.

While the majority of Twitter users are not experiencing the widespread outages that we saw a week and a half ago, it is still not uncommon for applications utilizing the Twitter API, and specifically OAuth, to see failures. There's also no doubt the ongoing attacks have led to a fair share of fail whale sightings. (I've encountered a few myself) Twitter posted a note to the company's status blog Sunday acknowledging the problems, and said they are "working to resolve it".

It has been tempting at times to poke at Twitter for its struggles during this period, but as I mentioned on August 7th, when the troubles first arose, the company is being much more visible and fast to respond to developers and to provide status, as they show signs of maturing. Their operations teams look to have been busy around the clock for the last two weeks, and they will need to keep fighting - as it looks like this battle may go on for some time.