What is Big Data?

Big data is a result or outcome of technological innovation and refers to extremely large sets of data, such as from sensors and smart devices that make up the Internet of Things. While big data is sometimes used interchangeably with data science, the latter represents the processes and techniques of working with data.

Big Data refers to expansive sets of data that are too large and complex to be analyzed through traditional data-processing and analysis methods.

The 5 V’s of Big Data

The 5 V’s provide a helpful framework for understanding the opportunities and challenges of big data today. These characteristics of big data are: volume, variety, velocity, veracity, and value.

Volume: Volume, or the size and breadth of the data, is often what first comes to mind when trying to understand “big” data. According to research firm Statista, in 2020, the total amount of data created, captured, copied, and consumed globally reached around 64.2 zettabytes. By 2025, the total amount of data is projected to grow to more than 180 zettabytes. The size of a zettabyte is nearly impossible to comprehend: One zettabyte is equal to 1 billion terabytes or one sextillion bytes (10^21, or 1,000,000,000,000,000,000,000 bytes).

However, only a fraction of this data can be captured and stored today, presenting a huge storage capacity opportunity for organizations looking to derive new insights from data, particularly from IoT connections. In 2020, the installed base of storage capacity reached 6.7 zettabytes, about 10.4% of the available big data.

Variety: There isn’t just one type of data. Data typically shows up to Data science, analytics, and engineering teams as structured, semi-structured, or unstructured.

• Structured data is the traditional data type: organized data stored in a relational database, typically with defined properties, formats, timestamps and metadata.
• Semi-structured data may have some structure and hierarchy from metadata tags, but is not neatly queried from a database.
• Unstructured data is most associated with big data today: high volumes of low-density data types, such as text, audio, and video from across the internet, with unknown value. Like searching for buried treasure across the world’s expansive oceans, unstructured data requires extensive preprocessing to clean and attempt structure, apply metadata, and try to identify patterns for the purpose of ultimately deriving meaning in the form of insights. In this way, Data scientists and Big data analyses are explorers in search of their organization’s next discovery.

Velocity: Today, data–especially unstructured data–is generated at unprecedented speed. For example, every millisecond, millions of social posts are published, and that high-velocity of data generation requires distributed processing techniques that won’t overwhelm storage tools or slow down computing.

Veracity: Not all data provides meaning or insight. With zettabytes of data created, how are organizations to wade through the clutter to derive signals from noise? Veracity is about the quality of the data and the likelihood that certain data will add value vs. being meaningless. Data scientists and Data engineers play an important role in prioritizing the veracity of data inputs into machine learning algorithms, so the analysis avoids “garbage in, garbage out.”

Value: There is enormous intrinsic value in data of all shapes and sizes. However, the value of that data must be unlocked with the right human capital (data scientists and analysts), the right culture and mindset, the right processes and tools, and a commitment to discovery through the scientific method.

Why is Big Data Important?

Big data is important to organizations who consider data to be a new form of capital and competitive advantage. Alphabet, Amazon, Meta and today’s other leading tech companies understand why big data is important–a large part of their financial value is driven by their advanced data capabilities as a differentiator. These companies are able to generate operational efficiencies (for example, Amazon’s logistics precision), to better understand their users and customers (for example: Netflix’s personalized recommendations algorithm), and to develop and launch new products and services (for example, ChatGPT).

In short, the process of transforming big data into valuable insights allows organizations to make better decisions and prioritize resources more effectively, which leads to a range of positive outcomes. These benefits include:

• Insights generation - better understanding markets, customer segments and needs
• Cost savings
• Revenue creation
• Customer satisfaction
• Employee purpose and satisfaction, or the belief that they’re working on something that matters

How is Big Data Stored and Processed?

Still the current state of data management in many organizations, data warehouses (or RDMS, relational database management systems) like Oracle or MySQL store relational, structured data automatically ingested from various applications, platforms and systems. In most cases, data structures and schemas are predefined to allow for ease of SQL queries and data analysis. Extensive consideration is given to accurate, “clean” data and data management, but structured, manageable amounts of data make end-to-end processes of data capture and analysis more straightforward than is the case with big data management. In the past, most of this data lived on-premise (“on-prem”) or was hosted at data centers.

Now that we’ve considered how data is traditionally stored, let’s look at how big data is stored and processed. Whether the “raw” big data comes from on-prem storage, cloud storage, or other edge-computing systems, vast amounts of big data can be ingested, processed, and stored in a centralized repository called a data lake. Data lakes allow for a hybrid, or blended, form of data management: the storage of relational, structured data from databases as well as the storage of non-relational, unstructured data from IoT devices, across the internet and from social media. Data lakes can be stored in cloud platforms like AWS and Microsoft Azure.

This fast-evolving capacity of data storage precipitates new techniques for data processing and different types of big data analytics. Data scientists, big data analysts and data engineers can use SQL, Python, R, or other languages to wrangle and structure data, create statistical machine learning models to curate and analyze data at scale, identify patterns, and visualize insights. They can also use open source data processing frameworks like Apache Hadoop or Spark to make big data easier to work with.

For example, data mining is one method of processing big data in data science. It’s a technique that uses intelligent methods like machine learning and statistics to find patterns in large amounts of data, then to structure that meaningful information so it’s accessible for future use or study.

Who Works with Big Data?

The answer to ‘who uses big data’ varies for each organization depending on data management maturity, complexity, and size. In general, different roles are typically involved in different stages of the data lifecycle because of their unique data skills.

Data architecture, storage and processing:

• Data architects
• Data engineers
• Software engineers
• DevOps engineers
• Data scientists

Data processing, wrangling and curating:

• Data scientists: Some estimate that Data scientists spend 50-80% of their time curating and preparing data before it can actually be used.
• Data engineers
• Machine learning engineers
• DevOps engineers
• Data developers

Data analysis, intelligence and visualization:

• Data scientists
• Machine learning engineers
• BI (Business Intelligence) developers
• Data developers
• Data analysts
• Business analysts
• Research analysts
• Operations analysts
• Financial analysts

Challenges with Big Data

Some of the challenges with big data include infrastructure and architecture, talent, and data governance.

• Big data infrastructure and architecture: On-prem vs. cloud vs. edge computing? Data warehouse vs. Data lake? Hadoop vs. Spark? Typically focused on the bottom line and customers, some business executives struggle to wrap their minds around legacy data management at their organizations, relative to the promises of big data storage capacity, big data processing power, cloud computing, and more. Managing significant amounts of data is also expensive, with the operational costs of cloud infrastructure and other data processing tools often outweighing the tangible value of the data, which is initially unknown. Organizations must be ready to commit to exploration and discovery of the intangible value of data without guarantee of “striking gold.”
• Big data talent: Data scientist is a new occupation, largely born out of tech breakthroughs in algorithms and AI/machine learning. Becoming proficient in optimizing and modeling unstructured data is very difficult and acts as a barrier for many budding Data scientists. Also, many have either the math or programming backgrounds to succeed – typically not both.
• Big data governance: The phrase “garbage in, garbage out” is well known in traditional data management. Now, what happens if an organization’s big data grows from bytes to terabytes–becoming exponentially larger and more complex? How can an organization effectively put controls and governance around their data warehouses, data lakes, and other repositories to help extract the data’s substantial potential value and eliminate noise? AI/machine learning and Data science professionals also need to consider “AI explainability,” or the ability to clearly explain to executives, regulators, and others how the machine learning algorithms work, to help identify issues of bias or eliminate “unclean” inputs.

Are Big Data and Data Science the Same?

Data science and big data are two different concepts, but they’re related in that data science is needed to process and utilize big data efficiently. The following points may help you further understand key differences and how big data relates to data science:

• Organizations use big data to be more efficient, understand markets, and maintain competitiveness, while data scientists provide the means to identify and utilize big data’s full potential.
• It’s significantly challenging to extract all valuable information from big data, but data scientists assume the responsibility of finding all useful information within big data through the development of theoretical and experimental approaches, as well as inference and deduction.
• Big data analytics involves identifying relevant information in expansive datasets. It usually has a specific question or goal in mind, and it analyzes the data to find a solution.
• Data science, on the other hand, aims to extract all useful information from datasets; it’s not limited to one particular goal or problem. Data scientists engage in machine learning and statistical methods to teach computers how to make predictions from the data, and it develops new ways to process and model it.

Big data is used in tools and software for distributed computing, analytics, and technology (like Hadoop, which is an open-source framework that aids in the storage and analysis of big data). Data science is used to develop business strategies and guide decisions while using disciplines like mathematics, statistics, data capturing and mining, and computer programming.

The table below further evaluates the fundamental differences between big data and data science:

Learn More About Big Data and Data Science

Data science is a broad field that encompasses a multitude of functions, specialties, and processes to help understand and utilize big data. Those with data science degrees and skillsets, including machine learning, are in high demand across numerous industries. Big data has become ubiquitous across industries and continues to drive business growth and advancement.

If you’re interested in pursuing a career in data science or machine learning that involves big data, learn more about data science and how Rice’s Master of Data Science degree program can help you stand out among other candidates in this competitive and demanding field. The MDS@Rice program offers specializations in related fields, an interdisciplinary curriculum with a course dedicated solely to learning about the theory and practice of big data, world-class faculty, and access to the Data To Knowledge (D2K) Lab and Capstone, where you’ll work on real-world projects that aid society through the use of big data. Learn more about the MDS@Rice degree program and the various specializations available to launch or advance your career in the exciting, innovative field of data science.