Big Data
● Definition: Big Data refers to datasets that are so massive, complex, and fast-moving that they cannot be stored, processed, or
analyzed using traditional data processing software (like Excel or standard SQL databases).
● Nature of Data: It is not just "a lot of data"; it is a mix of:
○ Structured Data: Organized data like bank transactions or Excel sheets.
○ Unstructured Data: Messy data like emails, videos, tweets, and sensor logs.
● Core Objective: The goal of Big Data analytics is to process this mess to uncover hidden patterns, market trends, and customer
preferences that lead to smarter business decisions.
● Where does it come from? (Sources):
○ Social Media & Web: Every like, share, tweet, or Google search generates a digital footprint. Example: 500 hours of video
uploaded to YouTube every minute.
○ IoT & Machines: Smart devices (Smartwatches, Traffic sensors, Satellites) continuously transmit real-time usage data.
○ Transactional Data: Every credit card swipe, Amazon purchase, or flight booking adds to the massive database of consumer
behavior.
[Link]
Characteristics of Big Data
Big Data is defined by five key characteristics, commonly known as the "5 V's":
● 1. Volume (Size): Refers to the massive scale of data generated every second, often measured in Terabytes or Petabytes. Example:
Facebook processing 500+ Terabytes of data daily.
● 2. Velocity (Speed): Refers to the speed at which data is generated and needs to be processed (often in real-time). Example: Stock
market data changing every millisecond or live video streaming.
● 3. Variety (Diversity): Refers to the different forms of data, including structured (tables), semi-structured (emails), and unstructured
(audio/video). Example: Analyzing text messages, photos, and GPS logs simultaneously.
● 4. Veracity (Accuracy): Refers to the trustworthiness and quality of the data. Poor quality data leads to wrong analysis. Example:
Filtering out fake news or bot accounts from social media data.
● 5. Value (Utility): The most important 'V'. Data is useless unless it provides meaningful insights for decision-making. Example: Using
customer data to offer a discount that increases sales.
[Link]
Data Analytics
Definition: Data Analytics is the science of examining raw datasets to draw conclusions, identify
patterns, and support decision-making using specialized software.
Core Objective: Unlike simple data storage, analytics focuses on deriving value—verifying scientific
theories or helping businesses optimize performance. Example: E-commerce companies analyzing
user clicks to decide which products to put on sale.
The Key Tool: Python Pandas:
● What is it? Pandas is the most popular open-source Python library used for data
manipulation and analysis. It is specifically designed to work with Structured Data (tables with
rows and columns).
● Why use it? It replaces complex Excel tasks with simple code, handling large datasets
efficiently. Example: A Data Scientist using Pandas to load a 1-million-row Excel file and
calculate the average sales in seconds.
Key Functions of Analytics (using Pandas):
● Data Cleaning: Fixing broken data or removing missing values to ensure accuracy. Example:
Removing rows where user age is listed as "0" or "null".
● Data Manipulation: Reorganizing data to make it easier to read. Example: Sorting a list of
students by "Marks" in descending order.
● Visualization: Converting numbers into graphs for easy understanding. Example: Plotting a
line graph showing the rise in stock prices over a year.
[Link]