Data is present in homogeneous sources as well as in heterogeneous sources.

Slide 46 make notes and last 4 slides

 Homogeneous sources → data coming from similar systems (e.g., multiple databases with the same schema).
 Heterogeneous sources → data coming from different types of systems or formats (e.g., CSV files, APIs, and databases).
Structured data

 Data which is in organized form(e.g., rows and columns) and can be easily used by a computer program.
 Relationships exist between entities of data, such as classes and their objects.
 Data stored in databases is an example of structured data.

Semi-structured data

 Data which does not conform to a data model but has some structure.

 It is not in a form which can be used easily by a computer program.

 For example XML, HTML etc.

Unstructured data

 Data which does not conform to a data model or is not in a form which can be used easily by a computer program.

 About 80%-90% data of an organization is in this format .

 For example memos, chat rooms, PowerPoint presentations, images, videos, letters etc,.
Structured Data

 Most of the structured data is held in RDBMS.

 An RDBMS conforms to the relational data model wherein the data is stored in rows/columns.

 The number of rows/records/tuples in a relation is called the cardinality of a relation and the number of columns are
referred as the degree of a relation.
Sources of Structured Data

On-Line Transaction Processing (OLTP)

Ease of Working with Structured Data

1. Insert/Update/Delete:
DML (Data Manipulation Language) operations make it easy to add, change, or remove data, and help in storing,
accessing, and analyzing it.

2. Security
Encryption and tokenization protect data throughout its lifecycle. Only authorized users can access or view sensitive
information, ensuring safety and compliance.
3. Index:
An index is a data structure that makes data retrieval faster (mainly for SELECT queries). It uses extra storage and write
operations, but the speed improvement is usually worth it.
 4. Scalability:
Traditional RDBMS systems can be scaled up by upgrading the database server’s hardware — such as increasing storage,
memory, or processing power.
5. Transaction processing: RDBMS has support for Atomicity, Consistency, Isolation, and Durability (ACID) properties of
transaction.

 Atomicity: A transaction is atomic, meaning it is executed as a single indivisible unit — either all operations
succeed, or none are applied.
 Consistency: The database moves from one consistent state to another consistent state. In other words, if the
same piece of information is stored at two or more places, they are in complete agreement.
 Isolation: Transactions are executed in isolation, ensuring that concurrent transactions do not interfere with each
other and the system behaves as if each transaction is running alone.
 Durability: All changes made to the database during a transaction are permanent and that accounts for the
durability of the transaction.

Semi-structured Data

Semi-structured data is also referred to as self-describing structure.

Features of Semi-Structured Data

1. It doesn’t follow strict data models like relational databases or tables.
2. Uses tags to define and separate different elements.
3. Tags also create hierarchies of records and fields. There’s no strict separation between data and schema.
4. Entities in the same group don’t need to have the same attributes, and if they do, the order of attributes doesn’t matter.
Sources of Semi-Structured Data

XML: Extensible Markup Language (XML) is widely used in web services, especially with SOAP-based services.

JSON: JavaScript Object Notation (JSON) is used to exchange data between servers and web applications. It’s popular with REST-
based web services. Databases like MongoDB and Couchbase store data in JSON format.

Unstructured Data
Unstructured data does not conform to any pre-defined data model.
The structure is quite unpredictable.
Sources of Unstructured Data
Web Pages, Images, Free-Form Text, Audios, Videos, Body of Email, Text Messages, Chats, Social Media data, and Word
Document.
Issues with Terminology mean that the way we label data isn’t always clear or accurate:

1. Sometimes data has a structure even if it’s not officially defined.

2. Data might have some structure, but if that structure doesn’t help in processing, it’s still called “unstructured.”
3. Data can have hidden or unexpected structure that we don’t know about in advance.

Dealing with Unstructured Data

The following techniques are used to find patterns in or interpret unstructured data:
[Link] Mining:
Data mining deals with large datasets and uses techniques from AI, machine learning, statistics, and databases to find patterns
and relationships. It is the analysis step in the “knowledge discovery in databases” process.
Popular Algorithms:
1. Association Rule Mining (Market Basket Analysis): Finds relationships between items. For example, if someone buys
bread, they are likely to also buy eggs or cheese.
2. Regression Analysis: Predicts the relationship between variables. The variable to predict is called the dependent variable,
and the predictors are independent variables.
3. Collaborative Filtering: Predicts a user’s preferences based on the preferences of group of users

2. Text Analytics / Text Mining:

Text data is mostly unstructured and hard to process. Text mining extracts useful information and patterns from text using
statistical and machine learning methods. Tasks include text categorization, clustering, sentiment analysis, and
concept/entity extraction.
3. Natural Language Processing (NLP):
NLP helps computers understand and process human language.
4. Noisy Text Analytics:
This extracts structured or semi-structured information from messy text like chats, blogs, emails, or messages. The text
may have spelling mistakes, abbreviations, acronyms, missing punctuation, or filler words like “um” or “uh.”
5. Manual Tagging with Metadata:
Adding metadata manually to unstructured data to give it meaning and context.
6. Part-of-Speech (POS) Tagging:
Labeling each word in a sentence with its part of speech, like noun, verb, or adjective.
7. Unstructured Information Management Architecture (UIMA):
An open-source platform by IBM for analyzing text and unstructured data in real time to find hidden meaning and
relationships.
Characteristics of Data
Data has three main characteristics:
1. Composition: Describes the structure of data — where it comes from, its type, granularity, and whether it’s static or real-
time.
2. Condition: Refers to the state of data — can it be used directly, or does it need cleaning and processing?
3. Context: Explains the source and purpose of the data — where and why it was generated.
Big Data
Big Data is high-volume, high-velocity, and high-variety information assets that demand cost effective, innovative forms of
information processing for enhanced insight and decision making.
VOLUME
 Amount of data generated
 Online & offline transactions

 In kilobytes or terabytes
 Saved in records, tables, files
VELOCITY
 Speed of generating data
 Generated in real-time

 Online and offline data

 In Streams, batch or bits
VARIETY
 Structured & unstructured
 Online images & videos

 Human generated - texts

 Machine generated - readings
 Variety includes structured, semi structured, unstructured
Challenges with Big Data
1. Capture: Data is growing extremely fast. It’s challenging to decide what data to use, what to ignore, and how to extract
useful insights from the rest.
2. Cloud and Virtualization: Cloud computing helps manage big data cost-effectively, but it raises questions about whether
to host data inside or outside the enterprise.
3. Data Retention: Deciding how long to keep data is tricky. Some data is useful long-term, while other data becomes
irrelevant quickly.
4. Skill Shortage: There aren’t enough skilled professionals in data science to implement big data solutions effectively.
5. Handling Big Data: Challenges include capturing, storing, preparing, searching, analyzing, transferring, securing, and
visualizing data. Big data is too large, fast-moving, and often doesn’t fit traditional database structures, so it must be
processed quickly.
6. Data Visualization: Visualizing big data is becoming a separate field, but there is a shortage of business visualization
experts.
Sources of Big Data
1. Internal Data Sources (within the organization):
 Data Storage: File systems, SQL databases (Oracle, MySQL, PostgreSQL, etc.), NoSQL databases (MongoDB, Cassandra,
etc.).
 Archives: Scanned documents, patient records, student admissions and assessments, etc.
2. External Data Sources (outside the organization):

 Public Web: Wikipedia, weather data, census data. etc.

3. Both Internal & External Sources:
 Sensor Data: Car sensors, smart meters, office devices, appliances, etc.
 Machine Logs: Event logs, application logs, business process logs,etc.
 Social Media & Documents: Twitter, blogs, Facebook, LinkedIn, YouTube, Instagram, PDFs, CSVs, Word, Excel, PPT, etc.

 Business Applications: ERP, CRM, HR systems, Google Docs, etc.

 Media: Audio, video, images, podcasts, etc.

Why Big Data

A Typical Data Warehouse Environment

 Day-to-day business data comes from ERP systems, CRM, legacy systems, and third-party applications.

 Data from these sources may have different formats and come from the same or different locations.
 The data is cleaned, transformed, and standardized using ETL (Extraction, Transformation, Loading).
 The processed data is loaded into the data warehouse or data marts.
 Business intelligence (BI) and analytics tools are then used to support decision-making.
A Typical Hadoop Environment

 Data comes from many sources: web logs, images, audio, video, social media, documents, PDFs, etc.
 Hadoop handles both internal data (inside the company) and external data (outside the company) by storing it in the
Hadoop Distributed File System (HDFS).
 If needed, this data can be sent back to operational systems, data warehouses, data marts, or an Operational Data Store
(ODS) for further processing and analysis.
What is Big Data Analytics?
1. Technology-Driven: Uses tools like IBM, Tableau, SAS, and WPS to process and analyze large datasets.
2. Business Insights: Helps understand customers, improve services, and make better business decisions.

3. Competitive Advantage: Provides insights that allow faster and smarter decision-making than competitors.
4. Collaboration: Brings together IT teams, business users, and data scientists.
5. Handling Large Data: Works with data that exceeds traditional storage and processing limits.
6. Move Code to Data: Instead of moving huge datasets, small programs are run where the data resides for efficient
processing.
Classification of Analytics
There are two main ways to classify analytics:
1. By Type: Basic, Operationalized, Advanced, and Monetized.
2. By Generation: Analytics 1.0, 2.0, and 3.0.
First School of Thought (By Type):

1. Basic Analytics: Simple reporting and visualization of historical data to gain basic insights.
2. Operationalized Analytics: Analytics integrated into everyday business processes.
3. Advanced Analytics: Uses predictive and prescriptive models to forecast the future.
4. Monetized Analytics: Analytics directly used to generate business revenue.
Importance of Big Data Analytics

Different approaches to data analysis lead to different outcomes:

1. Reactive — Business Intelligence (BI):
Analyzes past data to help businesses make faster, better decisions. Results are shown through dashboards, reports,
alerts, etc.
2. Reactive — Big Data Analytics:
Works on very large datasets but still analyzes static (already collected) data.
3. Proactive — Analytics:
Uses predictive models and data mining to make future decisions but relies on traditional databases with limited capacity.
4. Proactive — Big Data Analytics:
Processes massive amounts of data (terabytes to exabytes) quickly to find insights and solve complex problems in real
time.
Terminologies used in Big data Environments
In-Memory Analytics
 Normally, getting data from hard drives is slow.
 To make it faster, companies often pre-compute and store summaries or reports. But if you need new or different data,
you have to do this process again.
 In-memory analytics solves this by storing data directly in RAM (main memory) instead of on a hard drive.

 This makes data access much faster, gives quick insights, and reduces the need for complex IT setup.
In-Database Processing
 Also called in-database analytics.
 Normally, data from company systems (like sales or inventory) is cleaned and stored in a data warehouse. Then, it’s
exported to other tools for analysis.
 With in-database processing, the database itself does the analysis, so you don’t need to move the data elsewhere.
 This saves time and increases efficiency.

 Many big database companies now offer this feature

Symmetric Multiprocessor System (SMP)
 In SMP, multiple processors share one main memory.
 All processors can access the same input/output devices and are managed by one operating system.
 Each processor also has its own cache memory for faster work.

 They are connected through a common system bus, making them tightly linked.
Massively Parallel Processing (MPP)
 Many processors work together to run different parts of a program at the same time.
 Each processor has its own memory and operating system.
 They communicate through messages to coordinate tasks.
 MPP is harder to program since the work must be divided properly among processors.

 Unlike SMP, MPP processors don’t share memory or OS — they work independently but in parallel.
Difference Between Parallel and Distributed Systems
Parallel Systems
 All processors are tightly connected and work together to process queries.
 The user doesn’t know which processor is doing the work — everything looks like one system.

 Processors share the same memory or communicate using messages to coordinate tasks.
Distributed Database Systems
 They are loosely connected systems made up of multiple machines.
 Each machine can run its own programs and serve its own users.
 Data is spread across different machines, so answering a query may need data from many systems.
Shared Nothing Architecture
 There are three main types of architectures:
1. Shared Memory – all processors share the same memory.
2. Shared Disk – all processors share the same disks but have their own memory.
3. Shared Nothing – each processor has its own memory and disk, and nothing is shared.

 In Shared Nothing, every processor works independently, which makes the system highly scalable and efficient.
Advantages of Shared Nothing Architecture
1. Fault Isolation:
If one node fails, the problem stays within that node and doesn’t affect others.
2. Scalability:
Since no memory or disk is shared, new nodes can be easily added without slowing down the system. This makes it easy
to scale up as data grows.
CAP Theorem (Brewer’s Theorem)
The CAP theorem says that in a distributed system, you can’t have all three of these features at the same time:
1. Consistency: Every read gives the most recent data.

2. Availability: The system always responds, even if some parts fail.

3. Partition Tolerance: The system keeps working even when there are network issues or communication breaks between
nodes.
You can only choose two out of three — not all at once.
NoSQL (Not Only SQL)
 The term NoSQL was created in 1998 by Carlo Strozzi for a simple database that didn’t use SQL.
 Popular because they can scale easily and handle all types of data — structured, semi-structured, and unstructured.
 NoSQL databases are:

1. Open-source
2. Non-relational (don’t use tables like SQL)
3. Distributed (data spread across servers)
4. Schema-less (no fixed structure)
5. Cluster-friendly (works well with many connected machines)

6. Designed for modern web apps

Where is it Used?
 Common in big data and real-time web apps
 Used to store log data, social media data, and unstructured data that SQL databases can’t handle easily.
 Key Features of NoSQL

1. Non-relational: Data is stored as key-value pairs, documents, columns, or graphs, not in traditional tables.
2. Distributed: Data is spread across multiple nodes using low-cost hardware.
3. No full ACID support: They don’t fully follow ACID rules (Atomicity, Consistency, Isolation, Durability). Instead, they follow
CAP theorem, often prioritizing availability and partition tolerance over strict consistency.
4. Schema-less: NoSQL databases don’t require a fixed schema, allowing flexible storage of different types of data.

Types of NoSQL Databases

 Key-value or the big hash table.

 Schema-less



Why Use NoSQL?

1. Scale-out architecture: Can grow by adding more servers, unlike traditional monolithic databases.
2. Handles all data types: Can store structured, semi-structured, and unstructured data.
3. Dynamic schema: Data can be added without a fixed schema, enabling faster development and easier updates.
4. Auto-sharding: Data is automatically distributed across servers, balancing load and handling server failures smoothly.
5. Replication: Supports data copying across servers for high availability, fault tolerance, and disaster recovery.

Advantages of NoSQL

1. Easy to scale:
o Can grow quickly by adding more servers (cluster scale).
o Handles huge numbers of reads and writes (performance scale).
o Can store billions of documents (data scale).
2. Flexible schema:
o No need for a fixed structure; different records can have different fields (e.g., MongoDB).
3. Cost-effective and easy to implement:
o Offers scalability, high availability, and fault tolerance at lower operational costs.
4. Flexible consistency:
o Follows CAP theorem; often sacrifices strict consistency for higher availability and partition tolerance.
5. Data replication and partitioning:
o Sharding: Data is automatically spread across multiple servers, balancing load and allowing server changes
without downtime.
o Replication: Copies of data are stored across servers and data centers for high availability and fault tolerance
NoSQL databases don’t use standard SQL, but databases like MongoDB and Cassandra have their own powerful query languages
to work with the data.
NewSQL
 Combines NoSQL’s scalability with SQL databases’ ACID guarantees for online transaction processing (OLTP).
 Supports the relational model and uses SQL as the main interface.
 Built on shared-nothing architecture, allowing scalable and efficient performance.

Hadoop
 Open-source framework by Apache, written in Java.
 Built to support Nutch, a text search engine, using MapReduce and Google File System ideas.
Features of Hadoop
1. Handles large amounts of structured, semi-structured, and unstructured data using cheap, common hardware.
2. Uses a shared-nothing architecture.

3. Replicates data across multiple machines, so if one fails, others can continue processing.
4. Designed for high throughput, not fast response — best for batch processing large datasets.
5. Works alongside OLTP and OLAP, but does not replace relational databases.
6. Not suitable for tasks that can’t be parallelized or have data dependencies.
7. Not ideal for processing many small files; best for large files and datasets.

Key Advantages of Hadoop

1. Stores data in native format: HDFS is schema-less, storing raw data as is; structure is applied only when processing.
2. Scalable: Can handle huge datasets across hundreds of cheap servers working in parallel.
3. Cost-effective: Scale-out architecture reduces storage and processing costs per terabyte.
4. Fault-tolerant: Data is replicated across nodes, so if one fails, another copy is available.
 5. Flexible: Works with structured, semi-structured, and unstructured data for tasks like log analysis, social media insights,
recommendations, and marketing analysis.
6. Fast processing: Uses “move code to data” approach for quicker computation compared to traditional systems.
Versions of Hadoop

Hadoop 1.0 :It has two main parts:

1. Data Storage framework (HDFS):

o Stores files in their original format without a fixed schema.
o Provides flexibility and easy access for business use.
2. Data Processing framework (MapReduce):
o Uses Map and Reduce functions to process data.

o Mappers create intermediate key-value data.

o Reducers process this data to produce the final output.
o Highly distributed, parallel, fault-tolerant, and scalable.
Limitations of Hadoop 1.0
1. Requires programming skills: Users needed expertise in MapReduce and languages like Java.
2. Batch processing only: Good for tasks like log analysis or large-scale data mining, but not suitable for real-time processing.
3. Tightly coupled with MapReduce: Existing tools had to rewrite functionality in MapReduce or move data in/out of HDFS,
causing inefficiency.
Hadoop 2.0

 HDFS is still used for data storage.

 Introduces YARN (Yet Another Resource Negotiator) for resource management.
 YARN allows any application that can be split into parallel tasks to run efficiently.
 Uses an ApplicationMaster to run any type of application, not just MapReduce.
 Supports both batch processing and real-time processing
HDFS (Hadoop Distributed File System)
 Distributed storage system ofHadoop with streaming access and file permissions.
 Based on Google File System (GFS).

 Can scale a cluster from one node to thousands of nodes.

 Handles large datasets on cheap hardware.
 Fault-tolerant: stores files across multiple machines with redundant copies for recovery in case of failure.
HBase
 Database built on HDFS for fast, random data access.
 First non-batch component in the Hadoop ecosystem.
 Low latency compared to HDFS.
 NoSQL, non-relational, column-oriented database.

 Tables can have thousands of columns and multiple rows.

 Each row has column families, each family has columns, and each column can store multiple key-values.
 Based on Google BigTable.
 Used by Facebook, Twitter, Yahoo, etc.
Hadoop Data Ingestion Components
1. Sqoop (SQL to Hadoop)
o Purpose: Move data between RDBMS (MySQL, Oracle, DB2) and Hadoop (HDFS, Hive, HBase).
o Functions:
 Import data from RDBMS to Hadoop.

 Export data from Hadoop to RDBMS.

o Features:
 Connector-based architecture for easy integration with external databases.
 Can populate Hive and HBase tables.
 Integrates with Oozie to schedule automated tasks.

2. Flume
o Purpose: Collect and aggregate logs from multiple machines into Hadoop.
o Features:
 Developed for high-volume event data ingestion.
 Default storage is HDFS, but can also write to HBase or Solr.
Hadoop Data Processing Components
1. MapReduce
o A programming model for distributed and parallel processing of large datasets.
o Works with data stored in HDFS.
o Two phases:

 Map phase: Converts input data into key-value pairs.

 Reduce phase: Combines and aggregates the data into smaller results.
o Final output is stored back in HDFS.
2. Spark
o Supports in-memory computing, making it 10–100x faster than disk-based MapReduce.

o Can process huge datasets even if they don’t fit in memory (falls back to disk).
o Can run with Hadoop (on YARN) or independently.
o Works with Scala, Python, Java, R.
o Key libraries:
 Spark SQL: Query structured data using SQL.

 Spark Streaming: Real-time data processing.

 MLlib: Machine learning operations.
 GraphX: Graph-based computations.
o Often used with Hadoop: Hadoop handles storage and batch processing, Spark handles fast, real-time analytics.
Hadoop Data Analysis Components
1. Pig
o High-level scripting language for Hadoop, alternative to MapReduce.
o Two parts:
 Pig Latin: SQL-like language. Scripts are converted into MapReduce jobs to process data in HDFS. You can
load data, filter, sort, group, join, and save results back to HDFS. Good for ETL (Extract, Transform, Load)
and analyzing large datasets.
 Pig Runtime: Environment where Pig scripts run.

2. Hive
o Data warehouse software built on Hadoop.
o Performs summarization, querying, and analysis of data.
o Uses HiveQL (SQL-like language). Queries are converted into MapReduce jobs and executed on Hadoop.
From slide 114 to 117
Hadoop Distributions
 Hadoop is open-source and anyone can download it.
 The main components of Hadoop are:
1. Hadoop Common – Core utilities and libraries.

2. Hadoop Distributed File System (HDFS) – Stores data across multiple machines.
3. Hadoop YARN – Manages resources and job scheduling.
4. Hadoop MapReduce – Processes large datasets in parallel.
Integrated Hadoop Systems by Top Vendors
These are Hadoop solutions offered by leading companies:
 EMC Greenplum
 Oracle Big Data Appliance

 Microsoft Big Data Solution

 IBM InfoSphere
 HP Big Data Solutions
They provide ready-to-use Hadoop platforms with vendor support.
Cloud-Based Hadoop Solutions
 Amazon Web Services (AWS): Offers a complete set of cloud services to manage big data, focusing on cost reduction,
scalability, and faster innovation.
 Google Cloud Storage Connector for Hadoop: Allows running MapReduce jobs directly on data in Google Cloud Storage
without copying it to HDFS. This reduces cost, improves performance, and increases reliability by avoiding a single point of
failure.