CABD0056:

BIG DATA MANAGEMENT

MCA 3rd Semester

Dr. Jonti Deuri, ADBU

 What’s Big Data?
• Big data is a collection of large datasets that cannot be processed using
traditional computing techniques.

• According to Gartner, the definition of Big Data – “Big data” is high-

volume, velocity, and variety information assets that demand cost-effective,
innovative forms of information processing for enhanced insight and
decision making.”

• The challenges include capture, curation, storage, search, sharing, transfer,

analysis, and visualization.

Big Data Management- Dr. Jonti Deuri, ADBU

Characteristics Of Big Data
1. Volume

2. Velocity

3. Variety

4. Veracity

Big Data Management- Dr. Jonti Deuri, ADBU

Big Data: 3V’s

Big Data Management- Dr. Jonti Deuri, ADBU

 Volume (Scale)
• Data Volume
• 44x increase from 2009 2020
• From 0.8 zettabytes to 35zb
• Data volume is increasing exponentially

Exponential increase in
collected/generated data

Big Data Management- Dr. Jonti Deuri, ADBU

 4.6
30 billion RFID billion
12+ TBs tags today
camera
of tweet data (1.3B in 2005)
phones
every day world wide

100s of
millions
data every day

of GPS
? TBs of

enabled
devices sold
annually

25+ TBs of 2+
log data
every day billion
people on
the Web
76 million smart meters by end
in 2009… 2011
200M by 2014

Big Data Management- Dr. Jonti Deuri, ADBU

 Variety (Complexity)
• Relational Data (Tables/Transaction/Legacy Data)
• Text Data (Web)
• Semi-structured Data (XML)
• Graph Data
• Social Network, Semantic Web (RDF), …

• Streaming Data
• You can only scan the data once

• A single application can be generating/collecting many

types of data

• Big Public Data (online, weather, finance, etc)

To extract knowledge➔ all these types of data

need to linked together

Big Data Management- Dr. Jonti Deuri, ADBU

 A Single View to the Customer

Social Banking
Media Finance

Our
Gaming
Customer Known
History

Entertain Purchase

Big Data Management- Dr. Jonti Deuri, ADBU

 Velocity (Speed)
• Data is generated fast and need to be processed fast
• Online Data Analytics
• Late decisions ➔ missing opportunities
• Examples
• E-Promotions: Based on your current location, your purchase history,
what you like ➔ send promotions right now for store next to you

• Healthcare monitoring: sensors monitoring your activities and body

➔ any abnormal measurements require immediate reaction

Big Data Management- Dr. Jonti Deuri, ADBU

 Real-time/Fast Data

Mobile devices
(tracking all objects all the
time)
Social media and networks Scientific instruments
(all of us are generating data) (collecting all sorts of data)

Sensor technology and

networks
(measuring all kinds of data)

• The progress and innovation is no longer hindered by the ability to collect data
• But, by the ability to manage, analyze, summarize, visualize, and discover
knowledge from the collected data in a timely manner and in a scalable fashion

Big Data Management- Dr. Jonti Deuri, ADBU

Some Make it 4V’s

Big Data Management- Dr. Jonti Deuri, ADBU

Types of Digital Data

[Link]
[Link]
[Link]-structured

Big Data Management- Dr. Jonti Deuri, ADBU

 Types of Digital Data
❑Structured
▪ Any data that can be stored, accessed and
processed in the form of fixed format is termed as
a 'structured' data.
▪ However, nowadays, we are foreseeing issues when a
size of such data grows to a huge extent, typical sizes
are being in the range of multiple zettabytes.

Big Data Management- Dr. Jonti Deuri, ADBU

 ➢ Do you know? Data stored in a relational database management system is one
example of a 'structured' data.

• Examples Of Structured Data

An 'Employee' table in a database is an example of Structured Data

Employee_ID Employee_Name Gender Department Salary_In_lacs

2365 Rajesh Kulkarni Male Finance 650000

3398 Pratibha Joshi Female Admin 650000
7465 Shushil Roy Male Admin 500000
7500 Shubhojit Das Male Finance 500000
7699 Priya Sane Female Finance 550000

Big Data Management- Dr. Jonti Deuri, ADBU

❑ Unstructured

▪ Any data with unknown form or the structure is classified as unstructured data.

▪ A unstructured data is a heterogeneous data source containing a combination

of simple text files, images, videos etc.

• Examples Of Un-structured Data

The output returned by 'Google Search'

Big Data Management- Dr. Jonti Deuri, ADBU

❑ Semi-structured
▪ Semi-structured data can contain both the forms of data.
▪ We can see semi-structured data as a structured in form but it is actually not defined
with e.g. a table definition in relational DBMS.
▪ Example of semi-structured data is a data represented in an XML file.

➢ Examples Of Semi-structured Data

Personal data stored in an XML file-

<rec><name>Prashant Rao</name><sex>Male</sex><age>35</age></rec>
<rec><name>Seema R.</name><sex>Female</sex><age>41</age></rec>
<rec><name>Satish Mane</name><sex>Male</sex><age>29</age></rec>
<rec><name>Subrato Roy</name><sex>Male</sex><age>26</age></rec>
<rec><name>Jeremiah J.</name><sex>Male</sex><age>35</age></rec>

Big Data Management- Dr. Jonti Deuri, ADBU

❑ Semi-structured
▪ Semi-structured data can contain both the forms of data.
▪ We can see semi-structured data as a structured in form but it is actually not defined
with e.g. a table definition in relational DBMS.
▪ Example of semi-structured data is a data represented in an XML file.

➢ Examples Of Semi-structured Data

Personal data stored in an XML file-

<rec><name>Prashant Rao</name><sex>Male</sex><age>35</age></rec>
<rec><name>Seema R.</name><sex>Female</sex><age>41</age></rec>
<rec><name>Satish Mane</name><sex>Male</sex><age>29</age></rec>
<rec><name>Subrato Roy</name><sex>Male</sex><age>26</age></rec>
<rec><name>Jeremiah J.</name><sex>Male</sex><age>35</age></rec>

Big Data Management- Dr. Jonti Deuri, ADBU

 Why is Big Data Important ?
• Cost Savings
• Time Reductions.
• Understand the market conditions
• Control online reputation
• Boost Customer Acquisition and Retention
• Solve Advertisers Problem and Offer Marketing
Insights
• Big Data Analytics As a Driver of Innovations and
Product Development

Big Data Management- Dr. Jonti Deuri, ADBU

 Business Intelligence vs Big Data
• Traditional BI methodology is based on the principle of
grouping all business data into a central server.
• Typically, this data is analyzed in offline mode, after
storing the information in an environment called Data
Warehouse.
• The data is structured in a conventional relational
database with an additional set of indexes and forms of
access to the tables (multidimensional cubes).

Big Data Management- Dr. Jonti Deuri, ADBU

 Data Warehouse
• A data warehouse (DW) is a collection of corporate information
and data derived from operational systems and external data
sources.
• A data warehouse is designed to support business decisions by
allowing data consolidation, analysis and reporting at different
aggregate levels.
• Data is populated into the Data Warehouse through the processes
of extraction, transformation and loading (ETL tools).
• Data analysis tools, such as business intelligence software, access the
data within the warehouse.

Big Data Management- Dr. Jonti Deuri, ADBU

 Key characteristics data warehouse

• Integration

• Time-Variant

• Non-Volatile

• Subject-Oriented

• Designed for Query Performance

Big Data Management- Dr. Jonti Deuri, ADBU

 Harnessing Big Data

• OLTP: Online Transaction Processing (DBMSs)

• OLAP: Online Analytical Processing (Data Warehousing)
• RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

Big Data Management- Dr. Jonti Deuri, ADBU

❖ Application of Big Data
Big Data Management- Dr. Jonti Deuri, ADBU

✓ Here is the list of top Big Data applications in today’s world:

• Big Data in Healthcare

• Big Data in Education
• Big Data in E-commerce
• Big Data in Media and Entertainment
• Big Data in Finance
• Big Data in Travel Industry
• Big Data in Telecom
• Big Data in Automobile
Big Data Management- Dr. Jonti Deuri, ADBU

✓ Let’s discuss the applications of Big Data in detail.

[Link] Data in Retail

▪ The retail industry is the one that faces the most fierce competition of all.
Retailers constantly hunt for ways that will give them a competitive edge over
others. Customers are the real king sounds legit for the retail industry in
particular.
▪ For retailers to thrive in this competitive world, they need to understand
their customers in a better way. If they are aware of their customers’ needs
and how to fulfill those needs in the best possible way, then they know
everything.
▪ Through advanced analysis of their customer’s data, retailers are now able to
understand them from every angle possible.
▪They gather this data from various sources such as social media, loyalty programs,
etc.
Big Data Management- Dr. Jonti Deuri, ADBU

▪ Even a minute detail about any customer has now become significant for them. They are
now closer to their customers than they have ever been. This empowers them to provide
customers with more personalized services and predict their demands in advance.
▪ This helps them in building a loyal customer base. Some of the biggest names in the retail
world like Walmart, Sears and Holdings, Costco, Walgreens, and many more now have Big
Data as an integral part of their organizations.
▪ A study by the National Retail Federation estimated that sales in November and December
are responsible for as much as 30% of retail annual sales.
 2. Big Data in Healthcare
▪ Big Data and healthcare are an ideal match. The amount of data the healthcare
industry has to deal with is unimaginable.
▪ From finding a cure to cancer to detecting Ebola and much more, Big Data has
got it all under its belt and researchers have seen some life-saving outcomes
through it.
▪ Big Data and analytics have given them the license to build more personalized
medications.
▪ Data analysts are harnessing this data to develop more and more effective
treatments. Identifying unusual patterns of certain medicines to discover ways for
developing more economical solutions is a common practice these days.

Big Data Management- Dr. Jonti Deuri, ADBU

Big Data Management- Dr. Jonti Deuri, ADBU

▪ Explore how Big Data helps to speed up the treatment process – Big Data in
Healthcare.
▪ Smart wearables have gradually gained popularity and are the latest trend among
people of all age groups. This generates massive amounts of real-time data in the
form of alerts which helps in saving the lives of the people.
3. Big Data in Education
▪ When you ask people about the use of the data that an educational institute gathers, the
majority of the people will have the same answer that the institute or the student might
need it for future references.
▪ Even you had the same perception about this data, didn’t you? But the fact is, this data
holds enormous importance. Big Data is the key to shaping the future of the people
and has the power to transform the education system for better.
▪ Some of the top universities are using Big Data as a tool to renovate their
academic curriculum. Additionally, universities can even track the dropout rates of
the students and are taking the required measures to reduce this rate as much as
possible.

Big Data Management- Dr. Jonti Deuri, ADBU

Big Data Management- Dr. Jonti Deuri, ADBU

4. Big Data in E-commerce

▪ It is now part and parcel of our routine life. Whenever we need to buy something, the first
thought that provokes our mind is E-commerce. Big Data has been the face of it.
▪ Some of the biggest E-commerce companies of the world like Amazon, Flipkart, Alibaba, and
many more are now bound to Big Data and analytics is itself an evidence of the level of
popularity Big Data has gained in recent times.
▪ Big Data is now as important as anyone else in these organizations. Amazon, the biggest E-
commerce firm in the world and one of the pioneers of Big Data and analytics, has Big Data as
the backbone of its system. Flipkart, the biggest E-commerce firm in India, has one of the most
robust data platforms in the country.
▪ See how Flipkart used Big Data to have one of the most robust data platforms.
▪ Big Data’s recommendation engine is one of the most amazing applications the Big Data world
has ever witnessed. It furnishes the companies with a 360-degree view of its customers.
▪ Companies then suggest customers accordingly. Customers now experience more personalized
services than they have ever had. Big Data has completely redefined people’s online shopping
experiences.
Big Data Management- Dr. Jonti Deuri, ADBU

5. Big Data in Media and Entertainment

▪ Media and Entertainment industry is all about art and employing Big Data in it is a sheer piece of art.
Art and science are often considered to be the two completely contrasting domains but when
employed together, they do make a deadly duo and Big Data’s endeavors in the media industry are a
perfect example of it.

▪ Viewers these days need content according to their choices only. Content that is relatively new to
what they saw the previous time. Earlier the companies broadcasted the Ads randomly without any
kind of analysis.
▪ But after the advent of Big Data analytics in the industry, companies now are aware of the kind of Ads
that attracts a customer and the most appropriate time to broadcast it for seeking maximum attention.
▪ Customers are now the real heroes of the Media and entertainment industry - courtesy to Big
Data and Analytics.
Big Data Management- Dr. Jonti Deuri, ADBU

6. Big Data in Finance

▪ The functioning of any financial organization depends heavily on its data and to safeguard that
data is one of the toughest challenges any financial firm faces. Data has been the second most
important commodity for them after money.
▪ Even before Big Data gained popularity, the finance industry was already conquering the
technical field. In addition to it, financial firms were among the earliest adopters of Big Data
and Analytics.
▪ Digital banking and payments are two of the most trending buzzwords around and Big data
has been at the heart of it. Big Data is bossing the key areas of financial firms such as fraud
detection, risk analysis, algorithmic trading, and customer contentment.
▪ This has brought much-needed fluency in their systems. They are now empowered to focus
more on providing better services to their customers rather than focussing on security issues.
Big Data has now enhanced the financial system with answers to its hardest of the challenges.
Big Data Management- Dr. Jonti Deuri, ADBU

7. Big Data in Travel Industry

▪ While Big Data is spreading like wildfire and various industries have been cooking its food
with it, the travel industry was a bit late to realize its worth. Better late than never though.
Having a stress-free traveling experience is still like a daydream for many.
▪ And now Big Data’s arrival is like a ray of hope, that will mark the departure of all the
hindrances in our smooth traveling experience.

▪ See how Big Data is revolutionizing the travel & tourism sector.
▪ Through Big Data and analytics, travel companies are now able to offer more customized traveling
experience. They are now able to understand their customer’s requirements in a much-enhanced
way.
▪ From providing them with the best offers to be able to make suggestions in real-time, Big Data is
certainly a perfect guide for any traveler. Big Data is gradually taking the window seat in the
travel industry.
Big Data Management- Dr. Jonti Deuri, ADBU

8. Big Data in Telecom

▪ The telecom industry is the soul of every digital revolution that takes place around the world.
With the ever-increasing popularity of smartphones, it has flooded the telecom industry with
massive amounts of data.
▪ And this data is like a goldmine, telecom companies just need to know how to dig it properly.
Through Big Data and analytics, companies are able to provide the customers with smooth
connectivity, thus eradicating all the network barriers that the customers have to deal with.
▪ Companies now with the help of Big Data and analytics can track the areas with the lowest as
well as the highest network traffics and thus doing the needful to ensure hassle-free network
connectivity.
▪ Big Data alike other industries have helped the telecom industry to understand its customers
pretty well.
▪ Telecom industries now provide customers with offers as customized as possible.
▪ Big Data has been behind the data revolution we are currently experiencing.
Big Data Management- Dr. Jonti Deuri, ADBU

9. Big Data in Automobile

▪ “A business like an automobile, has to be driven, in order to get results.” B.C. Forbes
▪ And Big Data has now taken complete control of the automobile industry and is driving it
smoothly. Big Data is driving the automobile industry towards some unbelievable and never
before results.
▪ The automobile industry is on a roll and Big Data is its wheels or I must say Big Data has given
wings to it. Big Data has helped the automobile industry achieve things that were beyond our
imaginations

▪ From analyzing the trends to understanding the supply chain management, from taking care
of its customers to turning our wildest dream of connected cars a reality, Big Data is well
and truly driving the automobile industry crazy.
 The Model Has Changed…
• The Model of Generating/Consuming Data has Changed

Old Model: Few companies are generating data, all others are consuming data

New Model: all of us are generating data, and all of us are consuming data

Big Data Management- Dr. Jonti Deuri, ADBU

What’s driving Big Data
- Optimizations and predictive analytics
- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time

- Ad-hoc querying and reporting

- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets

Big Data Management- Dr. Jonti Deuri, ADBU

 The Evolution of Business Intelligence
Interactive Business
Intelligence &
Speed In-memory RDBMS Scale
QliqView, Tableau, HANA
Big Data:
BI Reporting Real Time &
OLAP & Single View
Dataware house Graph Databases

Business Objects, SAS,

Informatica, Cognos Big Data: Speed
other SQL Reporting
Tools Scale Batch Processing &
Distributed Data Store
Hadoop/Spark;
HBase/Cassandra

1990’s 2000’s 2010’s

Big Data Management- Dr. Jonti Deuri, ADBU

 Big Data Analytics
• Big data is more real-time in nature
than traditional DW applications
• Traditional DW architectures (e.g.
Exadata, Teradata) are not well-suited
for big data apps
• Shared nothing, massively parallel
processing, scale out architectures are
well-suited for big data apps

Big Data Management- Dr. Jonti Deuri, ADBU

Big Data Management- Dr. Jonti Deuri, ADBU
Big Data Technology

Big Data Management- Dr. Jonti Deuri, ADBU