Splunk Sorts Sandy Social Data

Posted on Feb 25, 2013 in Get Involved, Get to Know, Other Companies

In disaster situations, an overwhelming amount of data contributes to the chaos that responders must face. Splunk (one of GWOB’s partner organizations) is a real time engine for mission data, a powerful tool for turning the noise into usable information. In the aftermath of Hurricane Sandy, the Splunk team put together a series of reports that demonstrated exactly how useful the sea of social media posts can be.

Splunk purchased millions of tweets mentioning certain key words from Social Media Data company DataSift. Using that data, they created a series of charts and graphs to visualize when and where people were facing specific situations. They published a page with insights about the mental health of people in the path of Sandy with timelines of tweeted calls for help, tweets describing fear levels, and a breakdown of when people said they were evacuating the area. You can see examples of the reports they generated at Splunk4Good.

This time, the tweet analysis happened after the storm was over. In the future, the lessons learned from this experience can be used to process the stream of data as it happens. When members of the Splunk research team met with the FEMA Innovation team in Washington, DC they were told that their charts could have helped tremendously. One FEMA official said that he was responsible for delivering 1 million gallons of water to Sandy affected areas. If he’d had the Splunk analysis at the time, he felt he would have been able to make better decisions about where to send that water.

Another experiment the research team ran was an aggregation of Instagram photos from the area affected by the storm. This stream was taken directly from Twitter using the API as it happened. The results were a compelling, constantly scrolling page of photos of flood damage, people in their homes and outside, and a few comedic memes. Theoretically, a stream like this could be used together with the geotagging on the images to figure out where assistance is most needed.

A lot more work has to be done to understand how we can best use large amounts of non-alpha-numeric datasets for realtime disaster management and recovery. This may include a crowdsourced element a-lá GalaxyZoo You can help by experimenting with the tools yourself. Splunk follows a freemium service model, so you can work with 500MB of daily index volume for free. You can also contact them to get additional free access for development purposes. Learn how to create applications with Splunk at their developers’ site.