Scribd
Upload
37 views21 pages

Introduction to Big Data Analytics

The document provides an overview of Module 1 on introduction to big data, discussing key concepts like the 3Vs of big data, characteristics of big data, differences between traditional business intelligence and big data, data warehouse and Hadoop environments, and how they can coexist. It also covers terminologies in big data like CAP theorem and BASE concepts, introduces NoSQL and Hadoop components, and describes MapReduce and Hadoop architecture.

Uploaded by

dhurgadevi
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
37 views21 pages

Introduction to Big Data Analytics

The document provides an overview of Module 1 on introduction to big data, discussing key concepts like the 3Vs of big data, characteristics of big data, differences between traditional business intelligence and big data, data warehouse and Hadoop environments, and how they can coexist. It also covers terminologies in big data like CAP theorem and BASE concepts, introduces NoSQL and Hadoop components, and describes MapReduce and Hadoop architecture.

Uploaded by

dhurgadevi
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd

SRI KRISHNA COLLEGE OF TECHNOLOGY

[An Autonomous Institution | Affiliated to Anna University and


Approved by AICTE | Accredited by NAAC with ‘A’ Grade]

KOVAIPUDUR, COIMBATORE – 641 042.

21ITE06/BIG DATA ANALYTICS


III YEAR /CSE/VI SEMESTER
MODULE-1
Module 1- Introduction

Session Topic
1.1 Types of Digital Data-Characteristics of Data – Evolution of Big Data - Definition of Big Data –
Challenges with Big Data.
1.2 3Vs of Big Data – Non-Definitional traits of Big Data – BI vs. Big Data - Data warehouse and Hadoop
environment – Coexistence.
1.3 Big Data Analytics: Classification of analytics – Data Science
1.4 Terminologies in Big Data – CAP Theorem – BASE Concept.
1.5
NoSQL: Types of Databases – Advantages – NewSQL - SQL vs. NOSQL vs NewSQL.

1.6
Introduction to Hadoop: Features – Advantages - Versions – Overview of Hadoop Eco systems –

1.7 Hadoop distributions – Hadoop vs. SQL – RDBMS vs. Hadoop


1.8 Hadoop Components – Architecture-HDFS
1.9
Map Reduce: Mapper – Reducer - Combiner -Partitioner – Searching – Sorting – Compression

1.10
Hadoop 2 (YARN): Architecture – Interacting with Hadoop Eco systems.

MODULE 1 Introduction to Big Data


1.2 Introduction to Big Data:
3Vs of Big Data – Non-Definitional traits of Big Data – BI vs. Big Data - Data
warehouse and Hadoop environment – Coexistence..

Course Outcome:
Upon completion of the session, students shall have ability to
[U]
CO1 Understand evolution of big data with its characteristics and

challenges with traditional business intelligence.


CO2 Distinguish big data analysis and analytics in optimizing the business [AP]
decisions.
3V’s of Big Data

[Link]
[Link]
[Link]
1. Volume:
Volume means “How much Data is generated”. Now-a-days, Organizations or Human
Beings or Systems are generating or getting very vast amount of Data say TB(Tera Bytes)
to PB(Peta Bytes) to Exa Byte(EB) and more.
3V’s of Big Data
1. Volume:
Sources of data
• XLS,Doc,PDF->
Unstructured data
• Videos

Data Maybe from

•Internal data sources->Data stored and Archived Data


•External data sources->Public web data
•Both->Sensor data, Machine Log,Social media, Media,Business
Application.
2. Velocity:
Velocity means “How fast produce Data”. Now-a-days, Organizations or Human Beings or
Systems are generating huge amounts of Data at very fast rate.

3. Variety:
Variety means “Different forms of Data”. Now-a-days, Organizations or Human Beings or
Systems are generating very huge amount of data at very fast rate in different formats. Structured
data, Semi structured data and unstructured data.
Big data: Characteristics
Non-Definitional traits of Big Data
Other characteristics of data

4. Veracity
Veracity means “The Quality or Correctness or Accuracy of Captured Data”.
Out of 4Vs, it is most important V for any Big Data Solutions. Because without
Correct Information or Data, there is no use of storing large amount of data at fast rate and
different formats. That data should give correct business value.
Non-Definitional traits of Big Data
Other characteristics of data

5. Volatility
• How long is the data valid?
• How long it is stored?

6. Variability
• Data flows can be highly inconsistent with periodic peaks.
BI vs. Big Data

Traditional business intelligence Big data

Centralized database architecture Distributed File system is adopted


used to store and maintain the in big data
data in a fixed format.

Data scales in and out horizontally Vertical Scaling happens

Traditional data is typically It deals with large volume of both


structured, meaning it is structured, semi structured and
organized in a predefined manner unstructured data.
such as tables, columns, and
rows.
Data is analysed in Offline mode Both real time and offline mode
Data warehouse Environment
• Data Generation
• Sources are ER ,CRM,Legacy data
• Types are operational or transactional or day to day business data
• Data processed using ETL
• Data comes from same or different source, formats, locations.
• Data is integrated, cleaned, transformed and standardized.
• Extraction, Transformation and loading
• Loaded into enterprise ware house /data marts
• Apply Business Intelligence or Analytics tools to
process data
• Includes data mining, SQL etc.,
Hadoop Environment

• Different from data warehouse environment


• Data sources includes from weblogs to
images,audios,videos ,pdf,docs ,social media data etc.,
• Data may also resides outside the enterprise firewalls
• Hence all the data are placed in HDFS and whenever needed
for further processing it can repopulated back into data
warehouse, data marts ,operational systems, Operational
Data Store(ODS).
MODULE 1 Introduction to Big Data
Coexistence of Big data and Data warehouse
 Data warehouse continue with standard workload
from legacy operational systems, storing historical
data to provision traditional BI reporting and
analytics.
 Can’t ignore Hadoop also since different types of

table can be analyzed.


 Hence both work together none can be thrown

out.
Coexistence
Test your Knowledge

Big Data is high–Volume,high-Velocity,and high-variety


information assets that demand____________forms of
information processing for enhanced____________and
____________.
Test your Knowledge

Answer:
Cost- Effective, Innovative, Insight, Decision making
Test your Knowledge
Test your Knowledge
Next Session…

1.3 Big Data Analytics: Classification of


analytics – Data Science …………

Common questions

Powered by AI

Data velocity impacts real-time processing needs by demanding that systems process data as it is generated, which is crucial for operations like fraud detection where instant decision-making is paramount. For example, high-velocity data from financial transactions needs immediate processing to identify anomalies. Businesses in e-commerce may use real-time analysis of customer interactions to recommend products dynamically, enhancing customer experience and operational efficiency .

Variability refers to the unpredictable nature and inconsistent flows of data, which can complicate data analysis processes. This impacts analytics by making it challenging to maintain processing performance during data spikes or varied data input. Measures to manage variability include implementing robust data management and processing frameworks, like Hadoop, that can dynamically scale resources and using data cleaning techniques to ensure data consistency and reliability .

Transitioning from traditional RDBMS to Hadoop-based systems offers several benefits such as horizontal scalability, flexibility in schema-less data storage, and the capability to handle varied data types including unstructured data. However, challenges include the need for expertise in distributed computing paradigms, potential issues with data migration, and the requirement to adapt existing applications to a new processing framework. Organizations need to weigh these benefits and challenges carefully to make a strategic decision based on their specific data processing needs and resources .

Veracity refers to the quality, correctness, or accuracy of the data captured. Unlike Volume, Velocity, and Variety, which focus on measurable aspects of data, Veracity addresses the trustworthiness of data. It is crucial because incorrect data can lead to misguided business decisions, making it essential for ensuring that data storage and analysis lead to accurate and actionable insights .

Data warehouses continue to handle standard workloads from legacy systems by storing and processing structured historical data, typically used for traditional BI reporting. Hadoop environments, however, manage diverse data types, including unstructured data like weblogs, images, and social media, allowing for more complex analyses. Coexistence is beneficial because each system complements the other; data can flow from Hadoop to data warehouses for integrated analytics, permitting the use of both systems’ strengths without discarding either .

The 3Vs of Big Data are Volume, Velocity, and Variety. Volume refers to the amount of data generated, which often spans from terabytes to petabytes or even exabytes, necessitating scalable storage solutions. Velocity involves the speed at which data is generated and processed, prompting the need for real-time data processing solutions. Variety pertains to the different forms of data, including structured, semi-structured, and unstructured data, which requires flexible processing techniques to handle diverse data formats .

NoSQL databases offer schema flexibility, horizontal scalability, and the ability to handle diverse data types, including unstructured data. They are well-suited for Big Data management as they efficiently process large volumes of data with quick query response times, unlike traditional SQL databases which require predefined schema and struggle with scaling and performance at large data sizes .

The CAP Theorem states that a distributed data store can provide only two out of the following three guarantees at the same time: Consistency, Availability, and Partition tolerance. This theorem is significant for Big Data architectures as it guides the design choices when implementing distributed systems, like those found in Hadoop or NoSQL databases, influencing decisions on trade-offs between data consistency, system availability, and tolerance to network partitions .

MapReduce is a programming model used in Hadoop for processing large data sets across a distributed system. It consists of two main functions: the 'Map' function filters and sorts data, while the 'Reduce' function aggregates the results. This paradigm allows for parallel processing of data, significantly improving efficiency and scalability by dividing tasks and running them concurrently across multiple nodes, thus facilitating Big Data analytics by efficiently handling large volumes of data .

Traditional BI systems utilize a centralized database architecture that maintains data in a structured format, typically enabling analysis in offline modes. In contrast, Big Data environments adopt a distributed file system that scales horizontally and handles a mix of structured, semi-structured, and unstructured data for both real-time and offline analytics. This difference in architecture and data handling capabilities allows Big Data systems to manage more complex and varied data types than traditional BI .

You might also like