Big Data 101

By Zhikica Pagovski
posted on May 2, 2014 in Leadership

We live in an era of increasingly diverse means of drawing on the Internet, sensing technologies, and social media. As a result, more data is available than ever before – and what to do with these resources is a question of mounting complexity.

For some, the opportunities offered by this trend will undoubtedly translate to a new avenue for profit. McKinsey Global Institute claims that due to this explosion of data, the “large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.” The ability to process more data may lead to more accurate analyses and thus more confident decision making, which in turn means greater operational efficiencies, cost reductions and reduced risk at any functional area.

What is “big data”?

Big data usually refers to data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. But this definition is both subjective and dynamic, in that technological advances will allow the means to compute and process this information will evolve with datasets themselves. Generally speaking, the three criteria of big data involve: (1) the degree of complexity within the data set, (2) the amount of value that can be derived from innovative vs. non-innovative analysis techniques, and (3) the use of longitudinal information supplements the analysis.

Where to use it

Big data is present in almost all parts of the human living in today’s world. There are variety of sectors that are already reaping the benefits of big data including, but certainly not limited to, government, private sector, science and health, research and technology, military and security, and international development. As acknowledged by the White House, government and policy actors are able to analyze the information contained in big data to address important problems and make informed and effective decisions. Hence, in 2012, president Obama announced the Big Data Research and Development Initiative (composed of 84 different big data programs within six departments), with the goal to use big data in identifying and resolving issues of governance and government.

The private sector, already responsible for storing a vast number of large datasets, is currently sponsoring big data competitions and providing funding for research utilizing these vast troves of information. Science, research, technology, architecture, and other sectors aim to turn benefits by accelerating new innovations and improving current systems and mechanisms. Also, the effective processing of big data could provide for better assessment of security risks and envision more appropriate defense mechanisms. Finally, in international development, researchers and policymakers are “beginning to realize the potential for channeling these torrents of data into actionable information that can be used to identify needs & provide services for the benefit of low-income populations.”

Still, by and large, the foreign policy community lacks a deep understanding of the operationalization of big data. With a wealth of data available and a burgeoning set of tools with which to analyze it, it is imperative that world leaders in Washington and beyond grapple with the opportunities and the challenges inherent to the adoption of big data.