Recently, I saw firsthand how a new “universal identification” program called Aadhar is taking shape in India. It has potential to improve the lives of millions of poor people via Big Data.
Aadhar is an ambitious government Big Data project aimed at becoming the world’s largest biometric database by 2014, with a goal of capturing about 600 million Indian identities. This could help India’s government and businesses deliver more efficient public services and facilitate direct cash transfers to some of the world’s poorest people — while saving billions of dollars each year.
Many of the core ideas surrounding Big Data have been around for awhile, such as traditional data mining and analytics. But new technology enables the collection and analysis of, until recently, unimaginable data volumes at extremely high speeds.
“Big Data” refers to methods and technologies that help businesses and individuals make better decisions by analyzing large data volumes and predicting probable outcomes. The term has been around for a few years, but 2013 may be a year when Big Data moves from the technical to the practical, as real consumers and citizens start seeing its impact.
1. How we spend: Traditional and online retailers typically spent resources building huge datasets trying to understand their customer’s buying patterns using programs such as loyalty points. They offered big discounts on certain shopping days, such as Black Friday. New technologies help companies provide real-time offers to customers based on the date, the time of the day and the location of their shopping. As companies use Big Data to store and analyze more and more information about customers and competition, shopping will become more personalized and marketing more targeted. In short, you may get a better deal than someone sitting right next to you!
2. How we vote: If there was one area outside of business where Big Data had an enormous impact in 2012, it was in the U.S. presidential election. President Barack Obama’s campaign ran what has been referred to as the first Big Data-powered campaign that could micro-target individual voters most likely to be persuaded. The basic idea was to analyze every individual voter’s preferences instead of relying on traditional methods of taking polls with small sample sizes and extrapolating. This was historic because it upended traditional methods of running campaigns. Mounds of data from surveys, phone calls, external voter lists and past voting patterns drove real-time voter outreach and get-out-the-vote efforts. But Big Data was not limited to campaigns with huge technology infrastructure, as Nate Silver of The New York Times famously predicted the 2012 election outcome by applying statistical models to aggregate existing polling data.
3. How we study: A number of academic institutions are employing Big Data to address dual challenges of high dropout rates and the ensuing decline in state funding. The basic approach is to ensure that students select the majors that are best suited for them and nudging them to take classes that increase their chances of successfully graduating. Even the course material can be personalized for the students based on their interest, prior courses and the medium they find easiest to learn from (video, text, etc.). This is all made possible by analyzing vast amounts of student data, such as standardized test scores, previous grades and even real-time data points like clicks in an online class. Applying statistical models to each student’s profile and comparing results to similar students can predict the most likely outcomes (like succeeding in a class or completing a major) and offer constructive recommendations.
4. How we stay healthy: Healthcare has been a particularly difficult domain for analytics because of myriad privacy and regulatory restrictions that prevent the usage of data for research purposes. However the proliferation of smartphones and other “self-tracking” devices is fast changing the landscape. It is now possible to collect data from healthy individuals by constantly monitoring their vital information 24 hours a day, creating a very large unbiased control group that can be segmented by demographics such as age, sex and race. Analyzing large volumes of historical and real-time data can help individuals make healthy lifestyle choices, take preventive measures (e.g., flu vaccinations), predict their chances of being inflicted with a certain disease and possibly even provide personal analytics on their daily activities and how it impacts their health.
5. How we keep (or lose) our privacy: With all this data collection and analysis, privacy has rightly been a paramount concern with Big Data. Often individuals fear Big Data becoming the Big Brother (or Big Boss!)watching their every move and knowing the most intimate details about their life. An increasing amount of data — especially online and on smartphones — can be collected without the user’s knowledge or consent. Collection, analysis and sale of personal data on the Web can range from your search habits to shopping preferences to personal health issues, and it is a booming business, according to a recent Wall Street Journal investigation. Still consumers and citizens willingly share much of the data collected today.
India’s Aadhar collects sensitive information, such as fingerprints and retinal scans. Yet people volunteer because the potential incentives can make the data privacy and security pitfalls look miniscule — especially if you’re impoverished.
Big Data is quickly becoming a vast goldmine for businesses, governments and even law-enforcement agencies, but it also attracts hackers and identity thieves. Savvy consumers will understand how and where to best share their data, and what they get in return.
Throughout 2013 we are sure to see more and more impact of Big Data in other aspects of our daily lives, such as how we bank, watch TV and even stay safe. Consumers would do well to weigh the cost and benefits before allowing access to their data.
Follow Siddharth Taparia on twitter @siddharth31Comments