Big Data

-PROF. PRAJAKTA SOMAN

19-01-2026 MBA(IT) 1
Traditional Data & Database Systems
❑Structured data, maintained by ❑Databases
organizations ◦ Oracle
❑Data is stored in fixed format, ◦ Microsoft SQL Server
accessed using SQL ◦ MySQL
❑Records stored in tables, Relations ◦ PostgreSQL
are defined
❑Example
❑Data about employees, students,
customers, patients
❑Banking & mortgage data, RTO data
❑Purchase transactions, inventory

19-01-2026 MBA(IT) 2
Different Types of Data Today
❑Transaction Data – Big Bazaar ❑Image Data – Photos clicked and uploaded by
Users, X-Rays, MRI Scans
❑Log Data – Server Log Files
❑Video Data – CCTV footage
❑Streaming Sensor Data (IoT) –Humidity,
Soil moisture, Pulse beat ❑Audio Data – voice clips

❑Text Data – Tweets, Reviews, Blogs, Posts ❑Time-series data – temperature and weather
related data, medical parameters
❑Event Data – Book a cab, Web click
❑Web documents – Wikipedia
❑Spatial (GPS) Data – Latitude, Longitude
❑Genomics data– DNA and protein sequence
❑Graph Data – Social Network, Connections

19-01-2026 MBA(IT) 3
 How BIG can data be?
❑1 Kilobyte (KB) ❑1 Petabyte = 1000 TB
o 2 or 3 paragraphs of text o Walmart generates 40 Petabytes of data per day
o Size of small files is in KBs o Google and FB store > 100 PB data on their servers
❑1 Megabyte (MB) ❑1 Exabyte = 1018 bytes = 1000 PB
o An image, a PowerPoint presentation o Google processes 3.5 billion requests per day, storing
o A song of 2-3 min 10 Exabytes of data
❑1 Gigabyte (GB) ❑1 Zettabyte = 1000 Exabytes
o 256 MP3 files o All data in the world is made up of few ZB
o 15 GB space available on google drive and Gmail ❑1 Yottabyte = 1024 bytes
per user o Not yet in use
o Storage capacity of pen drive o Millions of datacenters would together hold
❑1 Terabyte (TB) Yottabyte of data
o Facebook generates 500+ Terabytes of data per
day
o Storage capacity of large storage devices
o Chatgpt-3 was trained on 45 TB of data

19-01-2026 MBA(IT) 4
Big Data - Definition
Data of a very large size, typically to the extent that its manipulation and management
present significant logistical challenges – Oxford English Dictionary
An all-encompassing term for any collection of data sets so large and complex that it
becomes difficult to process using on-hand data management tools or traditional data
processing applications - Wikipedia
Datasets whose size is beyond the ability of typical database software tools to capture,
store, manage, and analyze – Mckinsey

Big Data Analytics – Process of examining large amounts of data to gain insights. This
involves using relevant tools, algorithms and technology to discover value from data

19-01-2026 MBA(IT) 5
What do we do with Big Data?
- Businesses
- Government
- Society
- Individual
- Building a Mini-ChatGPT for a company

19-01-2026 MBA(IT) 6
What do we do with Big Data?
- Businesses – Customer churn analysis, Pattern finding, Recommendations,
Sentiment analysis, KPI tracking
- Government – better governance, safety of citizens, health services, city
transportation, emergency response for natural disaster, applications in
agriculture, spending patterns, eradicating poverty, maximize tax collection
- Society – Smart cities, Employment, Environment
- Individual – Fitness tracking, Recommendations

19-01-2026 MBA(IT) 7
Discussion
1. Any 5 companies which generate Big Data
2. What kind of data is generated at Amazon?
3. How do companies use data to grow their business?
4. Can data be used for societal good?
5. Which company has largest data center in the world?

19-01-2026 MBA(IT) 8
V’s of Big Data

19-01-2026 MBA(IT) 9
 ✓ 138.9 Million Reels (across FB & Insta) per
minute.
✓ Athletes share 4K activities on Strava per min!
Volume ✓ 350K posts per min on twitter!
✓ Facebook generates 5-7 PB of data per day!
❑Size of the dataset that needs to be analysed ✓ 6000+ prompts on ChatGPT per min

❑Number of images that FB needs to store

19-01-2026 MBA(IT) 10
Velocity
❑Data at a very high speed, low latency
❑Streaming data – sensors, stock market, tweets, website clicks, CCTV
❑Challenges
❑Data becomes stale immediately
❑Algorithms need to work on data in memory to quickly come up with insights
❑More storage capacity or more processing power or more RAM?

❑Detect health abnormalities in real time and notify the health care provider
❑Determining trading strategy real time
❑Emergency response planning in tornado, wildfire

19-01-2026 MBA(IT) 11
 Variety is the spice of life!
➢Structured, semi-structured, un-structured
➢Online images and videos, GPS data, SMS, pdfs
➢Formats – json, yaml, xml, csv, tab separated
➢Text generated by humans – blogs, posts, reviews,
articles
➢Emails, voice mails, audio recordings, ECG reading
➢More variety = more complexity = increased
diversity

19-01-2026 MBA(IT) 12
Axes of Data Variety (Heterogeneity)
•Structural Variety – difference in representation (satellite image of wildfire is
different from tweet, EEG signal is different from news paper article)
•Media Variety – audio of script vs transcript of the same speech
•Semantic Variety – qualitative vs quantitative
•Availability Variations – real-time, intermittent, history (what operations one can
do differs)

19-01-2026 MBA(IT) 13
Variety within an email

19-01-2026 MBA(IT) 14
Veracity
❖Veracity = Quality
❖Uncertainty, full of biases, abnormal, imprecise, inaccurate
❖Why is data uncertain?
❖Results of Big Data Analysis are only as good as the data being analysed
❖How to decide if the data is of good quality?
❖Fake reviews can result in incorrect sentiment analysis
❖Data provenance – record trail that allows for the origin of piece of data

19-01-2026 MBA(IT) 15
 Value
•How useful the data can be? How can it generate value for business?
•What useful insights are generated? What new knowledge is
discovered?
•Any strategic decisions taken? How will it impact future gains?
•Customer Churn Analysis – Using traditional DB, social media data,
web mining
•Healthcare – wearable device data, understanding links between
lifestyles and diseases
•Data Science => Big Data + Analysis (Questions) + Action

19-01-2026 MBA(IT) 16
Other V’s
❑Variability – inconsistency, outliers, anomalies, high dimensions, inconsistent speed
❑Validity – how correct the data is for intended use , pre-processing is important
❑Vulnerability – privacy and security concerns, security breach
❑Volatility – data becomes stale (invalid) suddenly (when the stock market rises and falls more
than one percent over a sustained period of time, it is called a "volatile" market)
❑Visualization – using meaningful visuals, with complex relationships, non-traditional: sunbursts,
cone trees, candle stick

19-01-2026 MBA(IT) 17
Technology Directions for Big Data
1. Data Stream Management – supports characteristics of stream as compared
to discrete transactions, parallelism, in-memory processing
2. Cloud – PAAS, SAAS, IAAS to store and process huge data
3. NoSQL Databases – support unstructured and semi-structured formats,
schema less data, denormalized data
4. Bandwidth – huge increase in bandwidth as a requirement to transfer video
data (Zoom platform, proctoring software)
5. Mobility / Wireless – Mobile access to all forms of data , increase in demand
and supply of data and information
6. New forms of Data – Text , Speech, Audio, Image – availability of technology
to manipulate and analyze such data

19-01-2026 MBA(IT) 18
Datasets

19-01-2026 MBA(IT) 19
 Kaggle Datasets
Over 50,000 public datasets
Competitions to solve real world machine learning problems
Challenges:
National Football League (NFL ) Big Data Bowl 2023 – Around 1 GB of data
(csv files)
Goal – use player tracking data to generate actionable and creative stats,
which NFL teams can use to evaluate their offensive and defensive lines
Image Matching Challenge 2022 – 2.6B data in csv, jpg and png format
Aligning photographs of the same scene to reconstruct 3D objects and
buildings from images. Registering 2 images from different viewpoints
from among thousands of images uploaded

19-01-2026 MBA(IT) 20
Challenges

19-01-2026 MBA(IT) 21
Challenges related to Big Data (pre-AI)
❑Insufficient understanding and acceptance of big data
❑Confusing variety of big data technologies
❑Analyzing data from different data sources
❑Complexity of managing data quality
❑Dangerous big data security holes
❑Tricky process of converting big data into valuable insights
❑Troubles of up-scaling
❑Lacking a clearly defined BI strategy, Poor BI functionality and interactivity

19-01-2026 MBA(IT) 22
Data from different Sources
Data from different sources may be stored in different formats, making it difficult to create a
seamless integration.
This may require additional time and resources for data cleaning and validation.
Performance issues: As the number of data sources increases, the complexity of your dashboard
may also increase

19-01-2026 MBA(IT) 23
Scalability challenge
❑Big data projects can grow and evolve rapidly. The scalability issue of Big Data
has lead towards cloud computing.
❑It leads to various challenges like how to run and execute various jobs so that
goal of each workload can be achieved cost-effectively.
❑It also requires dealing with the system failures in an efficient manner. This
leads to a big question again that what kinds of storage devices are to be used.

19-01-2026 MBA(IT) 24
Quality of Data
oWhen there is a collection of a large amount of data and storage of this data, it
comes at a cost. Big companies, business leaders and IT leaders always want
large data storage.
oFor better results and conclusions, Big data rather than having irrelevant data,
focuses on quality data storage.
oThis further arise a question that how it can be ensured that data is relevant,
how much data would be enough for decision making and whether the stored
data is accurate or not.

19-01-2026 MBA(IT) 25
Analytical Challenges
❑There are some huge analytical challenges in big data which arise some main
challenges questions like how to deal with a problem if data volume gets too
large?
❑Or how to find out the important data points?
❑Or how to use data to the best advantage?
❑These large amount of data on which these type of analysis is to be done can
be structured (organized data), semi-structured (Semi-organized data) or
unstructured (unorganized data). There are two techniques through which
decision making can be done:
❑Either incorporate massive data volumes in the analysis.
❑Or determine upfront which Big data is relevant.

19-01-2026 MBA(IT) 26
Privacy and Security
❖It is another most important challenge with Big Data. This challenge includes sensitive,
conceptual, technical as well as legal significance.
❖Most of the organizations are unable to maintain regular checks due to large amounts of data
generation. However, it should be necessary to perform security checks and observation in real
time because it is most beneficial.
❖There is some information of a person which when combined with external large data may lead
to some facts of a person which may be secretive and he might not want the owner to know this
information about that person.
❖Some of the organization collects information of the people in order to add value to their
business. This is done by making insights into their lives that they’re unaware of.

19-01-2026 MBA(IT) 27
Challenges in the era of AI?
❑In the AI era, goal of Big Data management is not just storage and analytics, but also
consumption and generation
❑The biggest source of new volume is now AI itself, lot of synthetic volume is generated on
internet by AI
❑Multimodality - Modern databases must hold Video, Audio, Code, and Images that are linked.
You can now search a video library by describing the action in text (e.g., "Show me the clip
where the car crashes")
❑Data is now stored as vectors which is a list of numbers representing meaning
❑Dirty data, earlier meant typos and missing values; not it means fake data and hallucinations
❑Check this using AI tool – How do companies solve/avoid veracity problem of AI tools? How
to stop AI from lying?

19-01-2026 MBA(IT) 28
Discussion
Discuss how Big Data can be applied to Healthcare industry – types of data, challenges, types of
healthcare problems, preventive, diagnostic, prescriptive. How does AI help here?

19-01-2026 MBA(IT) 29
Thank you!

19-01-2026 MBA(IT) 30