Big Data Analytics 21CS71

MODULE 1
INTRODUCTION TO BIG DATA ANALYTICS
Data
Data has multiple definitions and can be used in both singular and plural forms:
1. "Data is information, usually in the form of facts or statistics that one can analyze or
use for further calculations."
2. "Data is information that can be stored and used by a computer program."
3. "Data is information presented in numbers, letters, or other forms."
4. "Data is information from a series of observations, measurements, or facts."
5. "Data is information from a series of behavioral observations, measurements, or
facts."
Web Data
Web data refers to the information available on web servers, including text, images, videos,
audio, and other multimedia content accessible to web users. A user (client software) interacts
with this data in various ways:
 Pull: Clients retrieve data by sending requests to the server.
 Push/Post: Servers can also publish or push data, or users can post data after
subscribing to services.
Examples of Internet Applications:
 Websites, web services, and web portals
 Online business applications
 Emails, chats, tweets
 Social networks
Classification of Data:
Structured, Semi-Structured, Multi-Structured and Unstructured
Data can be broadly classified into the following categories:
1. Structured Data
Structured data conforms to predefined data schemas and models, such as relational tables
with rows and columns. Around 15-20% of data is either structured or semi-structured.
Characteristics of Structured Data:
 Supports CRUD operations: Enables data insertion, deletion, updating, and
appending.
 Indexing: Facilitates faster data retrieval through indexing.

1
Big Data Analytics 21CS71

 Scalability: Allows for increasing or decreasing storage capacity and processing

power for data operations like storage, analytics, and processing.
 Data Security: Supports encryption and decryption for secure data management.
2. Semi-Structured Data
Semi-structured data does not strictly adhere to a formal data model like relational databases
but contains organizational markers (tags) that distinguish different elements within the data.
Examples:
 XML (eXtensible Markup Language)
 JSON (JavaScript Object Notation)
Characteristics of Semi-Structured Data:
3. Multi-Structured Data
Multi-structured data encompasses data in multiple formats, which can include structured,
semi-structured, and unstructured data.
Characteristics of Multi-Structured Data:
 Found in non-transactional systems and across various platforms.
 Examples include streaming data from customer interactions, sensor data, or data
from web servers.
 Combines features of structured and unstructured data, sometimes with semantic
meanings attached.
4. Unstructured Data
Unstructured data does not conform to any predefined data models or schemas and lacks an
organized structure, such as tables or databases.
Characteristics of Unstructured Data:
 Can be found in file types like TXT, CSV, etc.
 Unstructured data may still have internal structures, as seen in emails, where headers
and bodies are distinguished.
 Establishing relationships, schemas, and structures for this data type often requires
additional processing.
Big Data
Big Data refers to large, complex data sets that traditional data processing systems cannot
efficiently handle. It is characterized by high volume, high velocity, and high variety,
requiring new processing methods to extract valuable insights and optimize processes.

2
Big Data Analytics 21CS71

Definitions of Big Data

1. Large and Complex Data Sets: A collection of data so large or complex that
traditional data processing applications are inadequate.
2. Logistical Challenges: Data that is too large for typical database software to capture,
store, manage, or analyze.
3. Large Size: Data of a very large size, making its manipulation and management
logistically challenging.
Characteristics of Big Data (4Vs)
Big Data has four key characteristics known as the 4Vs:
1. Volume
o Refers to the size of the data, often measured in terabytes or petabytes. The
sheer quantity of data generated from applications contributes to this
characteristic.
2. Velocity
o Refers to the speed at which data is generated and processed. Big Data often
needs to be processed in real-time or near real-time to meet decision-making
demands.
3. Variety
o Refers to the different types and formats of data generated from multiple
sources. Data can include structured, semi-structured, and unstructured
formats, making it more complex to manage.
4. Veracity
o Refers to the quality and reliability of the data. Big Data can come with
inconsistencies or inaccuracies, and ensuring data accuracy is critical for
meaningful analysis.
Big Data Types
Big Data comes from a wide variety of sources, both human and machine-generated. These
types of data contribute to its complexity and the need for specialized handling and
processing techniques.
1. Social Networks and Web Data
 Examples: Facebook, Twitter, YouTube, emails, blogs
 Description: Data from social media platforms, user interactions, and content
creation on the web. This includes textual data, multimedia content, and user
engagement metrics.
2. Transactional and Business Process Data

3
Big Data Analytics 21CS71

 Examples: Credit card transactions, flight bookings, public agency records (e.g.,
medical records, insurance data)
 Description: Data generated through business transactions and operational processes.
This includes financial transactions, service bookings, and records from public
agencies.
3. Customer Master Data
 Examples: Facial recognition data, personal information (e.g., name, date of birth,
marriage anniversary, gender, location, income category)
 Description: Data related to customer identity and demographics, often used for
personalized services, marketing, and authentication (e.g., facial recognition).
4. Machine-Generated Data
 Examples: Internet of Things (IoT) data, sensors, trackers, web logs, computer
system logs
 Description: Data generated from machines, devices, and sensors in an automated
manner. This type includes data from sensors in IoT devices, system logs from
servers, and data from machine-to-machine communication.
5. Human-Generated Data
 Examples: Biometrics, human-machine interaction data, email records, student
grades (stored in databases like MySQL)
 Description: Data that is generated by human interaction with machines. This
includes biometric data (e.g., fingerprints, facial scans), emails, and data stored in
databases for academic or business purposes.
Big Data Classification
Big Data can be classified based on various criteria such as data sources, formats, storage
structures, processing rates, and analysis methods. This classification helps understand how
Big Data is sourced, stored, processed, and analyzed.
1. Data Sources (Traditional)
Traditional data sources include:
 Records, Relational Database Management Systems (RDBMS): Structured data
storage in tables.
 Distributed Databases: Data spread across multiple systems for redundancy and
performance.
 In-memory Data Tables: Data stored directly in memory for fast processing.
 Data Warehouse: Centralized repositories for structured data.
 Servers: Data generated from machine interactions and operations.

4
Big Data Analytics 21CS71

 Business Process (BP) Data and Business Intelligence (BI) Data: Business
operation records and data for decision-making purposes.
 Human-Sourced Data: Data generated by human activities, such as emails, social
media, and transactions.
2. Data Formats (Traditional)
 Structured and Semi-Structured Data: Data stored in predefined formats, like
tables, XML, or JSON, making it easier to retrieve and analyze.
3. Big Data Sources
Big Data is sourced from a variety of places:
 Data Storage Systems: Distributed file systems, Operational Data Stores (ODS), data
marts, data warehouses, and NoSQL databases (e.g., MongoDB, Cassandra).
 Sensor Data: Data from IoT devices, monitoring systems, and sensors.
 External Data Sources: Web data, social media activity, weather data, and health
records.
 Audit Trails: Logs from financial transactions and other system operations.
4. Big Data Formats
Big Data comes in various formats:
 Unstructured, Semi-Structured, and Multi-Structured Data: Data without a
predefined schema, such as images, videos, and text, along with semi-structured
formats like XML and JSON.
 Data Stores Structure: Includes row-oriented data (used for OLTP systems),
column-oriented data (used for OLAP systems), graph databases, and hashed
key/value pairs.
5. Processing Data Rates
Big Data processing can happen at different speeds:
 Batch Processing: Large volumes of data are processed in chunks (e.g., using
MapReduce).
 Near-Time Processing: Data is processed almost immediately after it's received.
 Real-Time and Streaming Processing: Data is processed as it arrives (e.g., using
Spark Streaming).
6. Big Data Processing Methods
 Batch Processing: Includes tools like MapReduce, Hive, and Pig for processing large
data sets over time.
 Real-Time Processing: Uses tools like SparkStreaming, Apache Drill, and SparkSQL
for immediate data analysis and decision-making.

5
Big Data Analytics 21CS71

7. Data Analysis Methods

Big Data analysis can be performed using a variety of methods:
 Statistical and Predictive Analysis: Uses historical data to predict future outcomes.
 Regression and Machine Learning Algorithms: Includes clustering, classification,
and text analysis.
 Social Network and Location-Based Analysis: Analyzes social media networks or
geographical data.
 Cognitive and Diagnostic Analysis: Focuses on understanding patterns in data and
diagnosing problems.
8. Data Usages
Big Data is used in various fields:
 Human Interaction Data: Includes social media, customer feedback, etc.
 Business Processes and Knowledge Discovery: Helps businesses discover patterns
and optimize operations.
 Enterprise Applications: Big Data is utilized in decision-making, predictive
analytics, and improving business intelligence systems.
Scalability and Parallel Processing in Big Data
Big Data requires processing large volumes of data, often necessitating complex
computations that span across numerous computing nodes. Scalability and parallel processing
are fundamental to handling the massive data workloads that Big Data generates. Here's a
breakdown of how scalability and parallel processing are achieved:
Convergence of Data Environments and Analytics
 Scaling Up and Scaling Out: Big Data processing and analytics require both vertical
(scaling up) and horizontal (scaling out) scalability to handle massive data sets.
o Scaling Up (Vertical Scalability): Refers to improving a single system's
capacity by upgrading resources like CPUs, RAM, and storage. This enhances
analytics, reporting, and visualization capabilities.
o Scaling Out (Horizontal Scalability): Involves adding more systems (nodes)
to handle workloads. Multiple systems work together, distributing tasks and
increasing overall system capacity.

Scalability
 Capacity Increase: Scalability allows a system to grow or shrink in capacity as data
and processing demands change.
o Vertical Scalability: Increases a single system's resources to improve its
processing power and efficiency.

6
Big Data Analytics 21CS71

o Horizontal Scalability: Adds more systems to distribute tasks, enhancing the

ability to handle large datasets in parallel.
Analytics Scalability to Big Data
 Vertical Scalability: Focuses on maximizing the performance of a single machine.
By designing algorithms that efficiently use system resources (CPU, memory,
storage), Big Data analytics can handle larger datasets and more complex queries.
 Horizontal Scalability: Distributes workload across multiple machines or nodes,
enabling systems to process large datasets simultaneously and in parallel, improving
efficiency for large-scale data analysis.
Massively Parallel Processing (MPP) Platforms
 Parallel Processing: Involves dividing a computational problem into smaller sub-
tasks that are executed simultaneously across multiple CPUs or computers. This can
be achieved by:
o Task Distribution within the Same CPU: Using multiple threads to perform
different tasks on the same processor.
o Task Distribution Across CPUs in a Single Machine: Allocating separate
tasks to different processors within the same machine.
o Task Distribution Across Multiple Machines: Leveraging several computers
to process different parts of the dataset in parallel.
Advantages of Parallel Processing:
o Tasks are completed faster by distributing them across multiple compute
resources.
o Massive Parallel Processing (MPP) platforms enable scaling by distributing
computations across many systems or CPUs.
Distributed Computing Model
 Distributed Computing: Utilizes cloud, grid, or cluster computing environments,
where datasets are processed across several computing nodes connected by high-
speed networks.
o Key Characteristics:
 Efficiently processes large datasets by leveraging the collective
computing power of multiple systems.
 Scalable and parallel processing models like MapReduce are used to
break down large data into smaller, manageable chunks, processed in
parallel across nodes.
o No-Sharing Program Model: Ensures each node works independently,
reducing the complexity and improving performance.
Scalability and Efficient Design

7
Big Data Analytics 21CS71

 Scaling software to run on larger machines with more resources can enhance
performance, but the efficiency of the algorithm plays a significant role.
o Simply adding more CPUs or memory without optimizing the software's
ability to leverage these resources won't provide substantial performance
gains.
o Algorithm Design: Properly designed algorithms exploit additional resources
like extra CPUs and memory, enabling efficient use of parallel computing
environments.
Cloud Computing
Cloud computing is an internet-based service that allows on-demand access to shared
resources and data. It provides flexible and scalable computing power, data storage, and
services without requiring users to invest in their own physical infrastructure.
Features of Cloud Computing:
1. On-Demand Service: Users can access computing resources (such as storage,
processing power, or software) whenever needed without human interaction with
service providers.
2. Resource Pooling: Cloud providers use multi-tenant models to pool resources,
dynamically allocating them to meet the demands of multiple customers.
3. Scalability: Cloud resources can be scaled up or down based on demand, ensuring
flexibility.
4. Broad Network Access: Cloud services are accessible via the internet, meaning users
can access their resources from any location, using various devices.
5. Accountability: Cloud providers ensure transparent usage metrics and billing, giving
users clear insight into their resource consumption.
Types of Cloud Computing Services:
1. Infrastructure as a Service (IaaS):
o Provides access to computing resources such as virtual machines, storage, and
network infrastructure.
o Users can rent infrastructure on a pay-as-you-go basis.
o Examples:
 Amazon EC2: Virtual server space for scalable computing power.
 Tata CloudStack: Open-source software for managing virtual
machines, offering public cloud services.
2. Platform as a Service (PaaS):
o Provides a platform allowing developers to build, deploy, and manage
applications without worrying about the underlying infrastructure.

8
Big Data Analytics 21CS71

o Examples:
 Microsoft Azure HD Insights: Offers cloud-based Hadoop services.
 IBM BigInsight and Oracle Big Data Cloud Services: Provide big
data platforms for analytics and application development.
3. Software as a Service (SaaS):
o Delivers software applications over the internet. Users access software without
installing or maintaining it on their own computers.
o Examples:
 GoogleSQL, IBM BigSQL, HPE Vertica: Cloud-based SQL services.
 Microsoft Polybase and Oracle Big Data SQL: Cloud solutions for
data analytics and querying large datasets.
Cloud Computing in Big Data Processing
Cloud computing is a powerful environment for handling Big Data as it allows for both
parallel and distributed computing across multiple nodes. Big data solutions leverage the
cloud for:
 Data Storage: Cloud platforms such as Amazon S3 provide scalable storage for large
datasets.
 Data Processing: Cloud-based services like Microsoft Azure, Apache CloudStack,
and AWS EC2 facilitate the parallel processing of large-scale datasets.
Grid Computing:
Grid computing is a form of distributed computing where computers located in different
locations are interconnected to work together on a common task. It allows the sharing of
resources across various organizations or individuals for achieving large-scale tasks,
particularly data-intensive ones.
Features of Grid Computing:
1. Distributed Network: Grid computing involves a network of computers from
multiple locations, each contributing resources for a common goal.
2. Large-Scale Resource Sharing: It enables the flexible, coordinated, and secure
sharing of resources among users, such as individuals and organizations.
3. Data-Intensive Tasks: Grid computing is particularly well-suited for handling large
datasets that can be distributed across grid nodes.
4. Scalability: Grid computing can scale efficiently by adding more nodes to
accommodate growing data or processing needs.
5. Single-Task Focus: At any given time, a grid typically dedicates its resources to a
single application or task.
Drawbacks of Grid Computing:

9
Big Data Analytics 21CS71

1. Single Point of Failure: If one node underperforms or fails, it can disrupt the entire
grid, affecting overall performance.
2. Variable Performance: The performance and storage capacity of the grid can
fluctuate depending on the number of users, instances, and data transferred.
3. Resource Management Complexity: As resources are shared among many users,
managing and coordinating them can be challenging, especially with large volumes of
data.
Cluster Computing:
Cluster computing refers to a group of computers connected by a local network that work
together to accomplish the same task. Unlike grid computing, clusters are typically located in
close proximity and used primarily for load balancing and high availability.
Key Features of Cluster Computing:
1. Local Network: The computers in a cluster are interconnected locally and operate as
a single system.
2. Load Balancing: Clusters distribute processes among nodes to ensure that no single
computer is overloaded. This allows for better resource utilization and higher
availability.
3. Fault Tolerance: Clusters often provide redundancy, where if one node fails, others
can take over, minimizing the risk of downtime.
4. Application: Cluster computing is commonly used in high-performance computing
(HPC), scientific simulations, and business analytics.
5. Hadoop Integration: The Hadoop architecture follows cluster computing principles
by distributing tasks across many nodes for large-scale data processing.
Volunteer Computing:
Volunteer computing is a type of distributed computing that uses the resources of volunteers
(organizations or individuals) to contribute to projects requiring computational power.
Key Features of Volunteer Computing:
1. Volunteer Resources: Volunteers donate the computing power of their personal
devices (computers, smartphones, etc.) to help process data or run simulations for
large-scale projects.
2. Distributed Network: Similar to grid computing, volunteer computing relies on a
network of geographically distributed devices.
3. Popular in Academia: Volunteer computing is often used for science-related projects
by universities or research institutions.
Examples of Volunteer Computing Projects:
 SETI@home: A project that uses idle resources from volunteers to analyze radio
signals for extraterrestrial life.

10
Big Data Analytics 21CS71

 Folding@home: A project aimed at understanding protein folding to help find cures

for diseases like Alzheimer's and cancer.
Designing Data Architecture:

Data Architecture Design involves organizing how Big Data is stored, accessed, and
managed in a Big Data or IT environment. It creates a structure that allows the flow of
information, security management, and utilization of core components in an efficient manner.
Big Data architecture follows a systematic approach, especially when broken down into
logical layers, each serving a specific function. These layers make it easier to design, process,
and implement data architecture.
Big Data Architecture Layers:
The architecture is broken down into five main layers, each representing a set of core
functions essential for handling Big Data:
1. Identification of Data Sources (L1):
o Purpose: Identify the sources of data, which could be both internal
(organization databases, ERP systems) and external (social media, IoT
devices, APIs).
o Key Task: Determine the relevant data sources to be ingested into the system.
2. Acquisition, Ingestion, and Pre-Processing of Data (L2):

11
Big Data Analytics 21CS71

o Purpose: Data ingestion is the process of importing and absorbing data into
the system for further use. This data may be ingested in batches or real-time.
o Key Task: Perform initial data transformation, cleaning, and standardization
to ensure data readiness for storage and processing.
3. Data Storage (L3):
o Purpose: Store data in a variety of storage environments, such as files,
databases, clusters, or cloud systems. This layer holds structured, semi-
structured, and unstructured data for future processing.
o Key Task: Choose appropriate storage systems based on scalability and
reliability (e.g., Hadoop Distributed File System (HDFS), cloud storage like
AWS S3, or distributed storage).
4. Data Processing (L4):
o Purpose: This layer focuses on processing the data using frameworks and
tools like MapReduce, Apache Hive, Apache Pig, and Apache Spark.
o Key Task: Implement large-scale distributed data processing to analyze and
extract meaningful insights from vast datasets.
5. Data Consumption (L5):
o Purpose: After data is processed, this layer delivers the insights and results to
end users through analytics, visualization, and reporting tools.
o Key Task: Use analytics for various applications such as business
intelligence, AI/ML models, predictive analytics, pattern recognition, and
data visualization tools.
Designing Data Architecture
Data Architecture Design involves organizing how Big Data is stored, accessed, and
managed in a Big Data or IT environment. It creates a structure that allows the flow of
information, security management, and utilization of core components in an efficient manner.
Big Data architecture follows a systematic approach, especially when broken down into
logical layers, each serving a specific function. These layers make it easier to design, process,
and implement data architecture.
Big Data Architecture Layers:
The architecture is broken down into five main layers, each representing a set of core
functions essential for handling Big Data:
1. Identification of Data Sources (L1):
o Purpose: Identify the sources of data, which could be both internal
(organization databases, ERP systems) and external (social media, IoT
devices, APIs).
o Key Task: Determine the relevant data sources to be ingested into the system.

12
Big Data Analytics 21CS71

2. Acquisition, Ingestion, and Pre-Processing of Data (L2):

o Purpose: Data ingestion is the process of importing and absorbing data into
the system for further use. This data may be ingested in batches or real-time.
o Key Task: Perform initial data transformation, cleaning, and standardization
to ensure data readiness for storage and processing.
3. Data Storage (L3):
o Purpose: Store data in a variety of storage environments, such as files,
databases, clusters, or cloud systems. This layer holds structured, semi-
structured, and unstructured data for future processing.
o Key Task: Choose appropriate storage systems based on scalability and
reliability (e.g., Hadoop Distributed File System (HDFS), cloud storage like
AWS S3, or distributed storage).
4. Data Processing (L4):
o Purpose: This layer focuses on processing the data using frameworks and
tools like MapReduce, Apache Hive, Apache Pig, and Apache Spark.
o Key Task: Implement large-scale distributed data processing to analyze and
extract meaningful insights from vast datasets.
5. Data Consumption (L5):
o Purpose: After data is processed, this layer delivers the insights and results to
end users through analytics, visualization, and reporting tools.
o Key Task: Use analytics for various applications such as business
intelligence, AI/ML models, predictive analytics, pattern recognition, and
data visualization tools.
Functions of Data Architecture Layers
 Data Ingestion and Pre-processing (L2): The ingestion layer handles transferring
data into the system, similar to the way the body ingests food. It cleans and transforms
the data to make it suitable for analysis. This could include batch or real-time
ingestion processes.
 Storage (L3): After data is ingested, it is stored in a suitable format, either on
traditional servers, in clusters, or cloud-based platforms depending on the size, type,
and access requirements.
 Processing (L4): The processing layer handles computation and data processing
using tools such as Spark or MapReduce. It is responsible for transforming raw data
into meaningful information.
 Consumption (L5): Finally, data is made available for use through business
intelligence tools, reporting applications, or visualization platforms.
Applications of Big Data Architecture:

13
Big Data Analytics 21CS71

 Business Intelligence (BI)

 Data Mining and Machine Learning
 Artificial Intelligence (AI)
 Predictive and Descriptive Analytics
 Data Visualization
Data Pre-Processing
Data pre-processing is an essential step before conducting data mining, analytics, or running
Machine Learning (ML) algorithms. This stage ensures that the data is of high quality and
suitable for further analysis, particularly when data is being exported to cloud services or data
stores.
Pre-Processing Needs:
1. Handling Outliers and Inconsistencies:
o Dropping out-of-range values: Identify and remove data points that fall
outside acceptable limits.
o Addressing inconsistencies: Ensure that data follows the same format and
structure throughout.
2. Filtering Information:
o Eliminating unreliable data: Remove any data that is deemed irrelevant or
redundant to improve the quality of the dataset.
3. Data Cleaning and Editing:
o Correcting inaccuracies: Identify and rectify incomplete or incorrect data
entries.
o Data editing controls: Review and adjust datasets to enhance quality, using
methods such as:
 Interactive editing
 Selective editing
 Automatic editing
 Aggregating data
 Distribution of data
4. Data Reduction:
o Simplifying datasets: Transform acquired information into a more
manageable form while retaining meaningful insights. This involves reducing
the volume of data while focusing on the most relevant parts.
5. Data Wrangling:

14
Big Data Analytics 21CS71

o Transforming and mapping data: This process involves restructuring data to

make it more suitable for analytics and visualization. For instance, mapping
data to a new format can enhance its value.
6. Data Validation and Transformation:
o Ensuring accuracy: Validate data to confirm that it meets the required
standards.
o Transformation or transcoding: Convert data from one format to another to
facilitate easier analysis.
7. ELT Processing (Extract, Load, Transform):
o Data pipeline: Involves extracting data from various sources, loading it into a
storage system, and then transforming it as needed for analysis.
Data Cleaning
Data cleaning is a critical component of data pre-processing, focusing on the identification
and correction of issues in the data, such as:
 Removing inaccuracies: Eliminate incorrect or irrelevant data points that could lead
to misleading insights or decisions.
 Correcting incomplete data: Fill in gaps where information is missing or
incomplete.
Data Cleaning Tools:
Various tools are available for data cleaning, which play a vital role in preparing data for
analysis. These tools help in structuring and refining data to make it usable for decision-
making.
Data Enrichment
Data enrichment refers to processes aimed at enhancing the quality and depth of raw data.
This can involve adding context or supplementary information to improve analysis outcomes.
Big Data Platform Overview
A Big Data platform is designed to handle the challenges associated with large datasets, high
velocity, diverse varieties, and varying veracity of data. These platforms leverage advanced
technologies and tools to store, process, and analyze vast amounts of information effectively.
Here are the key components and requirements for a robust Big Data platform:
Core Functions of a Big Data Platform:
1. Storage, Processing, and Analytics:
o Supports efficient data storage and retrieval mechanisms.
o Enables real-time and batch processing capabilities.
o Provides analytical tools for insights and decision-making.

15
Big Data Analytics 21CS71

2. Development and Management:

o Tools for developing, deploying, and managing Big Data environments.
o Streamlined processes for integrating various data sources into a cohesive
system.
3. Complexity Reduction:
o Simplifies the integration of multiple data sources and applications.
o Ensures a unified approach to data management.
4. Custom Development and Integration:
o Facilitates custom solutions for specific business needs.
o Enables querying and integration with existing systems.
5. Support for Traditional and Big Data Techniques:
o Combines conventional data management techniques with modern Big Data
practices.
Requirements for Managing Big Data:
1. Innovative Storage Methods: Adoption of non-traditional methods for storing and
processing data.
2. Distributed Data Stores: Utilization of distributed databases to manage large
datasets effectively.
3. Scalable and Elastic Platforms: Implementation of cloud computing to create
virtualized environments that can scale as needed.
4. High-Volume Data Stores: Infrastructure to handle massive amounts of data
efficiently.
5. Massive Parallelism: Use of high-speed networks and parallel processing to enhance
performance.
6. High-Performance Processing: Optimization and tuning for maximum efficiency in
data processing tasks.
7. NoSQL Data Management Models: Adoption of NoSQL databases for flexible data
storage solutions.
8. In-Memory Data Processing: Implementation of in-memory data processing for
faster transactions and analytics.
9. Advanced Analytics Capabilities: Tools for data retrieval, mining, reporting,
visualization, and analysis.
10. Graph Databases: Leveraging graph databases for analyzing complex relationships,
such as social networks.

16
Big Data Analytics 21CS71

11. Machine Learning Integration: Utilizing machine learning techniques for predictive
analytics and data insights.
Data Sources in a Big Data Environment:
 Data Storage Solutions:
Traditional data warehouses and modern NoSQL databases (e.g., Oracle Big
Data, MongoDB, Cassandra).
 Sensor Data: Data generated from various sensors, which can include IoT devices.
 Audit Trails: Records of financial transactions and other business processes.
 External Data Sources: Information from web platforms, social media, weather data,
and health records.
Big Data Analytics Applications and Case Studies:
1. Big Data in Marketing and Sales:
o Marketing revolves around delivering value to customers. Big Data plays a
vital role in customer value analytics (CVA), allowing companies like Amazon
to enhance customer experiences. It helps businesses understand customer
needs and perceptions, leading to effective strategies for improving customer
relationships and lifetime value (CLTV).
o Big Data in marketing also aids in lowering customer acquisition cost (CAC)
and enhancing contextual marketing by targeting potential customers based on
browsing patterns.
2. Big Data Analytics in Fraud Detection:
o Fraud detection is critical to avoiding financial losses. Examples of fraud
include sharing customer data with third parties or falsifying company
information. Big Data analytics help detect and prevent fraud by integrating
data from multiple sources such as social media, emails, and websites,
allowing faster detection of threats and preventing potential frauds.
3. Big Data Risks:
o While Big Data offers insights, it also introduces risks. Erroneous or
inaccurate data can lead to faulty analytics, requiring companies to implement
strong risk management strategies to ensure accurate predictions and reliable
data usage.
4. Big Data in Credit Risk Management:
o Financial institutions use Big Data to manage credit risks by analyzing loan
defaults, timely return of interests, and the creditworthiness of borrowers. Big
Data provides insights into industries with higher risks, individuals with poor
credit ratings, and liquidity issues, helping financial institutions make
informed lending decisions.

17
Big Data Analytics 21CS71

5. Big Data in Healthcare:

o Big Data in healthcare leverages clinical records, electronic medical records,
and other sources to enhance healthcare services. It facilitates value-based,
customer-centric healthcare, fraud prevention, real-time patient monitoring,
and reduces healthcare costs. The Internet of Things (IoT) is also integrated
with Big Data to improve patient care.
6. Big Data in Medicine:
o In medicine, Big Data uses large datasets to create predictive models that help
in disease research and understanding the biology of diseases. Wearable
devices provide continuous health data, allowing better disease risk profiling
and personalized treatments.
7. Big Data in Advertising:
o Big Data has transformed the digital advertising industry by enabling real-time
analytics, identifying emerging trends, and building customer relationships. It
captures data from various sources, enriching structured data and providing
insights for targeted digital advertisements via SMS, emails, social media, and
other platforms. The data helps advertisers discover less competitive markets
and personalize their advertising strategies.
-------------------------------------------END OF MODULE 1-------------------------------------------
Important Diagrams:

18
Big Data Analytics 21CS71

------
Hadoop Structure

19