August 07, 2009

Twitter's "Harsh and Cold" Honesty Tells Devs No ETA for Fixes

The much-discussed distributed denial of service attacks over the last two days which hobbled Twitter and also impacted a number of other sites, including Facebook and LiveJournal, has also had a tremendous negative impact downstream, bringing many third party applications and services to a halt. With the sluggishness and non-responsiveness stretching into what will be the third calendar day tomorrow, Twitter told developers, heading into the weekend, that not only was the attack ongoing, and not lessening, but they had no ETA for anything to get better.

While many developers are understandably sympathetic toward Twitter's troubles, even as the popular microblogging service's network architecture is being analyzed for weaknesses, the strain many of them were feeling without receiving answers started to be on display this evening as hours went on without an update - while at the same time, key Twitter employees happily updated that they were going out for sushi or enjoying tonight's Underworld concert.

One developer, in the company's Google Groups forums, wrote:
"There's been lots of posts from Twitter saying they plan for better communication. Okay, cool. But it's Friday night and I am pulling my hair out with no new information. I am sure they are doing all they can... but many third-party apps have been broken for 2 days now. And just as Twitter is getting bombarded with emails from developers... developers, too, are getting bombarded with complaints/questions from customers."
After the same developer noted that employees had gone out to catch dinner, he exclaimed frustratedly, "While food is important, would be helpful to also update devs what's going on, too, before a night on the town?!?!?!" This led to another developer's stating he too agreed with the outrage, adding:
"Thanks for your outrage. I thought i was one of the only few that have issues with how things are being handled."
But shortly after the conversation threatened to get ugly, Twitter's Chad Etzel posted an update, around 8 p.m. Pacific time, catching up the community, and he didn't have much good news.

His major points:
  • "The DDoS attack is still ongoing"
  • "the intensity has not decreased at all"
  • "interaction with the site and with the API will continue to be shaky"
  • *There is no ETA on fixing any of this* (repeated twice)
He told developers that the situation at Twitter will "continue to be rocky" and may actually get worse, but also said that the company's operations team will be working "around the clock" this weekend, adding, "as much as you want it to be fixed, we want it to be fixed more."

With past outages, Twitter has had a mixed bag in terms of communicating to the developer community. Had the initial complaints gone unanswered, it's likely Twitter would have transitioned further away from being the victim to being seen as part of the problem. Yes, applications are broken. Yes, access is still slow, and as the site has become a major piece of infrastructure in today's social Web, its downtime is more than trivial. But this isn't a situation of the company's employees partying on a sinking ship. It sounds like the team is strained, and being realistic about what will continue to be a long process of restoring normality.