The final panel at Friday's CrunchUp focused on the phenomenon of real-time, featuring a high-profile panel complete with representatives from Google, Microsoft, TweetDeck, TweetMeme, Seesmic, FriendFeed, Stanford University and a pair of venture capitalists. The discussion ranged from the opinion that real time was simply yet another feature, or a revolution in terms of application and Web service development, while the panelists discussed revenue opportunity or how large companies would try and control the data from being shared with competition.Iain Dodsworth of TweetDeck said that his own experiences using his Twitter application had a dramatic impact on how he was using the Web. "I was using TweetDeck more than anything else," he said. "I wasn't using e-mail or RSS and this to me was a massively big deal. I wasn't going to Web pages any more. I was going to stream data and that was what I was consuming now."
And while Dodsworth and many of us early adopters may have gotten to this stage before a more mainstream audience, the trend seemed to be recognized across the board. Google and Microsoft representatives talked about how they were working to best harness the real-time phenomenon, with the advent of Google Wave, and in Microsoft's case, trying to find ways to improve the user experience for hundreds of millions simultaneously, all while maintaining a stable infrastructure.
David Hornik of August Capital Venture Partners said he believed the advent of real time was "an important piece of the evolution", harkening back to what he called the "dark days of RSS" where that was considered real-time. He said that the problems faced in the early days of RSS, around sorting and filtering, had only escalated since, saying "we have that problem in spades".
A relative newcomer in the shadows of monoliths like Google and Microsoft, FriendFeed has helped push the envelope on the world of real-time as much as practically any other service, spurring on social networks like Facebook to do the same. Representing the company, co-founder Bret Taylor said real-time was no longer becoming an oddity, but instead a core aspect of the Web experience.
"Real time is an important feature of every site. It can now be said that every product evolves until it has e-mail, a social network and real time. It is something that all products will incorporate," Bret said, adding that one of the biggest challenges in this new realm was how to display the stream of data. "There is a tension between the simplicity of chronology and the need for filtering and ranking. You don't want to lose the feeling that you know something when it comes out. You need a mix of things that recently happened with things you need to see, and get a sense of both. If something is missing, users get an uncomfortable, anxious feeling."
And it is the user experience that will likely make or break services, in terms of how they can master the flow. Kevin Marks, who recently left Google, and announced that he will be joining British Telecom to work on standards, said it is about finding value in the data.
"It is about the flow and sense of flow," said Marks. "Real-time is one piece of that dimension and preserving history, so it can be found again and discovered later. This is something that FriendFeed is doing better than Twitter. Those things that used to be real time are now stored."
And storing the data for later consumption is getting increasingly difficult as the total volume continues to skyrocket. Andreas Weigend of Stanford noted the total volume of data had increased an order of magnitude just in the last five years.
"The real-time aspect is one way to incentivize people to make actions with their data," Weigend said. "It is not just about the data, but what question you ask, and the experience. It is how fast it is for me to get a question answered. The real-time aspect is the timescale of innovation being an order of magnitude faster."
But while everyone on the panel clearly recognized that real-time was being integrated into many services, and users were looking to make sense of the data explosion, there were clearly concerns about interoperability, and having services communicate well with others. Often, the current situation was compared to the wars that once were rife in the Instant Messaging world, where different providers simply did not communicate - such as from AOL to Yahoo! or MSN. Not surprisingly, much of that debate was on the two largest providers - Twitter and Facebook.
Bret Taylor hoped that the battle would be different this time around, pointing specifically to the work Kevin Marks and others have pursued around standards.
"Users will demand interoperability," Bret said. "If your friend uses Twitter for broadcasting shared links and if you use Facebook, it is reasonable to be frustrated about that. If (somebody like) Yahoo! wanted to compete, they wouldn't need to compete, they would have to set up ad hoc or formal standards and it would just work."
Daniel Lewin of Microsoft agreed, saying, "As long as you adhere to core standards, which we are committing to, there is an evolution of use cases. People experiment, interesting things will happen, and as time rises, there will be capabilities where end-users will program what they want."
In many cases, the move to real-time came as developers and users grew tired with slowness elsewhere. As we have discussed a few times, in terms of the speed of RSS versus that of Twitter, FriendFeed and other networks, there can be a gulf between what was previously acceptable and what is expected now. It's a major reason why, for instance, Nick Halstead and the team at TweetMeme started to focus on their current product more, and less on their original site, Favorit.
"Real time is about collecting news in real time," Nick said. "The slowness of Feedburner is quite well known and we didn't realize it until we ran TweetMeme and we would see it two seconds later. We would go fetch it and find it 20 minutes later on Google. The important thing is instant. You press the button and it goes back to Twitter. It is fun for me to watch FriendFeed. I push retweet and it shows up on seconds in FriendFeed. The trends we see appear and something might get retweeted 50-60 times in 20 seconds and we can bubble it up in front of users interested in those things, get the content in front of people who want to see it."
And it's not just Twitter that's pushing the gas pedal. Loic Le Meur of Seesmic said the company is pushing more than 4 million API calls to Facebook per day. Just yesterday, Loic added the number of active users on Facebook had doubled in a single week.
Loic added, "It's not about the tool. It's about the people, realizing they have communities around themselves. It will go from tool to tool. In a few years, it might be our tools, and it might not. What matters is how it transfers to real life."
The conclusions? We are at the very beginning, still, of determining how to best harness the firehose of real time data. Tools like Twitter Search, FriendFeed, TweetDeck, Seesmic, TweetMeme and others are working to parse the signal from the noise. It's suggested that real-time will become a major part of many applications going forward, and users are still going to be in the experimentation phase of how they imbibe the data, or in terms of what kind of interfaces they will use to be best satisfied. There remain concerns about openness and how companies will play well with each other, especially if they are in a leadership position. And it's very likely that in five years, many of the names we consider household names today may be gone, replaced with others. It's all going to play out in real-time in front of our eyes.