📚 Parallel Processing & Big Data 🌐

Brief Overview
This note covers Parallel Processing and was created
from a 44-page PDF. It gathers core concepts on
distributed computing, Hadoop and MapReduce,
Spark, stream processing, and data‑warehousing
fundamentals—all in a concise, study‑ready format.
Key Points
Overview of Hadoop’s file system and
MapReduce workflow
MapReduce and Spark implementations,
including Word‑Count examples
Streaming data models, windowing, and
algebraic operations
OLAP concepts: star/snowflake schemas, roll‑up,
drill‑down, and data cubes
Practical insights for building and optimizing
distributed data pipelines
📊 Parallel Processing in MapReduce
Definition: MapReduce executes the map() and
reduce() functions on many machines in parallel, each
handling a subset of the input data.
Map phase
Input partitions (files or file fragments) →
Map tasks.
Each task runs map() on its partition,
emitting (key, value) pairs.
Output is locally sorted and partitioned by
reduce key.
Separate intermediate files are created for
each future reduce task.
Shuffle & Sort
Reduce tasks fetch the intermediate files
from all map nodes.
Files are merged and globally sorted so
that all occurrences of a given key are
together.
Reduce phase
Each Reduce task processes a distinct set
of keys.
The reduce() function receives a key and
an iterator over all its values.
 Parallelism is illustrated by Figure 10.8 (input
partitions → map nodes → reduce nodes).
🗄️ Distributed File Systems
Definition: A distributed file system stores data
across multiple machines, providing parallel I/O and
fault tolerance.
HDFS (Hadoop Distributed File System)
Files are split into blocks and replicated
(default 3×) across nodes.
Guarantees availability even if some
machines fail.
Storage adapters allow MapReduce to read
from / write to other big‑data stores (e.g.,
HBase, MongoDB, Cassandra, Amazon Dynamo).
🛠️ Hadoop MapReduce Overview
API Requirements
Types must be declared for input and output
keys/values of both map() and reduce().
Programmers implement map() and reduce() as
methods of classes extending Mapper and
 Reducer.
Input format classes (e.g., TextInputFormat)
define how raw files are broken into records
(lines → records).
Mapper & Reducer Classes (Java)
Component Purpose Typical Type
Example
Mapper key Byte offset of a LongWritable
line
Mapper value Content of the Text
line
Mapper output Word extracted Text
key from the line
Mapper output Count (initially 1) IntWritable
value
Reducer input Same as mapper Text
key output key
Reducer input Iterable of Iterable
values counts
Reducer output Word Text
key
 Reducer output Total count IntWritable
value
Combiner Function
Definition: A combiner runs after the map phase on
each map node, performing a local reduction to lower
network traffic.
In the word‑count example, the combiner can
sum local occurrences of each word, emitting a
single (word, local‑count) pair per map node.
Reduces the volume of data shuffled to reducers.
Controlling Parallelism
Hadoop lets the programmer set the number of
map and reduce tasks.
Jobs can consist of multiple MapReduce steps;
each step reads the previous step’s output from
the distributed file system.
💻 Hadoop Word Count Example
public class WordCount {
public static class Map extends Mapper {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable key, Text value, Context context)

 throws IOException, InterruptedException {
String line = [Link]();
StringTokenizer tokenizer = new StringTokenizer(line);
while ([Link]()) {
[Link]([Link]());
[Link](word, one);
}
}
}

public static class Reduce extends Reducer {

public void reduce(Text key, Iterable values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += [Link]();
}
[Link](key, new IntWritable(sum));
}
}

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link](job, new Path(args[0]));
[Link](job, new Path(args[1]));
[Link](true);
}
}

Map: tokenizes each line, emits (word, 1).

Reduce: sums all values for a given word.
Optional Combiner can be set to the same class
as Reduce to perform local aggregation.
📐 Relational Operations on MapReduce
Relational Operation MapReduce
Implementation
Selection (σ) Single map() that filters
rows; optional reduce()
that passes through
unchanged.
Group‑by / Aggregation map() emits (groupKey,
(γ) value); reduce()
aggregates the list of
values for each key.
Join (equijoin r ⋈ s) map() emits (joinKey,
(tag, record)) for both
relations; reduce() groups
by joinKey, separates
tags, and emits
cross‑product of matching
tuples.
Complex queries can be decomposed into a
series of MapReduce steps, but this often leads
to verbose and hard‑to‑maintain code compared
with SQL.
🚀 Beyond MapReduce: Algebraic Operations
Motivation
Direct algebraic operators (e.g., join, outer‑join,
semi‑join) make programmer intent clearer and
can be executed more efficiently than a custom
map‑reduce composition.
Modern data‑processing systems expose these
operators to simplify pipelines and enable
advanced analytics (e.g., machine‑learning
models as operators).
Apache Tez (brief)
Low‑level API for building DAGs of algebraic
operators.
Used internally by Hive‑on‑Tez to compile SQL
queries into executable tasks.
Apache Spark Overview
Definition: Spark works with Resilient Distributed
Datasets (RDDs)—immutable, partitioned collections
that survive node failures.
Core Concepts
Transformations (lazy) – e.g., map, flatMap,
reduceByKey; define a new RDD without
immediate execution.
 Actions (eager) – e.g., saveAsTextFile, collect;
trigger evaluation of the transformation lineage.
Lazy Evaluation enables query optimizers to
rewrite the operation graph before execution.
The lineage of transformations forms a DAG;
nodes may have multiple downstream
consumers.
RDD vs DataSet
Feature RDD DataSet
Type safety Generic Encoded schema
Java/Scala (better
objects optimization)
API level Low‑level, Higher‑level,
functional SQL‑like
Suitable for Unstructured/text Structured data
data (Parquet, ORC,
Avro)
Spark Word Count Example
import [Link];
import scala.Tuple2;
import [Link];
import [Link];
import [Link];

public class WordCount {

 public static void main(String[] args) throws Exception {
SparkSession spark = [Link]()
.appName("WordCount")
.getOrCreate();

JavaRDD lines = [Link]()

.textFile(args[0])
.javaRDD();

JavaRDD words = [Link](s ->

[Link]([Link](" ")).iterator());

JavaPairRDD ones = [Link](s ->

new Tuple2<>(s, 1));

JavaPairRDD counts = [Link]((i1, i2) -> i1 + i2);

[Link]("outputDir");
[Link]();
}
}

flatMap expands each line into multiple word

records.
mapToPair creates (word, 1) pairs.
reduceByKey groups by word and sums the
counts (a parallel group‑by/aggregate).
Key Takeaways for Parallel Execution
Partitioning: RDDs (or intermediate files in
Hadoop) are split across machines; operations
act locally on each partition.
Repartitioning: Functions like reduceByKey may
shuffle data so that all records for a key end up
on the same node.
 DAG Optimization: Because Spark builds a lazy
DAG, the engine can reorder or combine
operations to minimize data movement.
These notes provide a concise yet complete reference
for MapReduce parallelism, Hadoop’s implementation
details, relational mapping, and the transition to
algebraic frameworks such as Apache Spark.
📊 Spark Dataset Operations 🚀
Spark Dataset is a typed distributed collection that
provides the benefits of both RDDs (strong typing)
and DataFrames (optimizations via Catalyst).
The Row type gives column‑access by name, while
custom classes let the compiler know each field’s
type.
Reading Parquet with a schema
Parquet files store column names and
types, so Spark can automatically infer a
schema.
Example reads two relations – instructor
and department – from Parquet sources.
Dataset instructor = [Link]().parquet("...");
Dataset department = [Link]().parquet("...");
[Link]([Link]("salary").gt(100000))
.join(department,
[Link]("dept name")
.equalTo([Link]("dept name")))
.groupBy([Link]("building"))
.agg(count([Link]("ID")));

Pipeline meaning
1. Selection – keep only instructors with
salary > 100000.
2. Join – match on dept name.
3. Group‑by – group the joined rows by the
department’s building.
4. Aggregation – compute of instructor
count

IDs per building.

Using a custom bean class
When a POJO (e.g., Instructor) matches the
Parquet schema, Spark can map columns directly
to getters/setters, avoiding runtime
name‑lookup.
Dataset instructor = [Link]()
.parquet("...")
.as([Link]([Link]));

Advantages
 Compile‑time type safety.
Faster attribute access (getSalary() instead
of col("salary")).
More concise code when the schema is
known.
Encoders translate between JVM objects and Spark’s
internal binary format, enabling efficient serialization
for the custom class.

📈 Streaming Data Concepts ⏱️

Streaming data is an unbounded flow of tuples that
arrive continuously, often with strict latency
requirements.
Typical Application Domains
📉 Stock market – trade tuples streamed in real
time; traders and regulators need sub‑second to
microsecond latency.
🛒 E‑commerce – purchase and search events
form streams; used for campaign monitoring,
demand spikes, fraud detection.
🔧 Sensors – periodic readings from devices;
fast anomaly detection and fault isolation in
 vehicles, factories, buildings.
🖧 Network monitoring – packet‑level or
aggregated flow tuples; real‑time detection of
failures or DDoS attacks.
💬 Social media – posts/tweets streamed to
followers; ranking, advertisement targeting,
sentiment alerts.
Parallel processing is essential in all these scenarios
because streams can reach very high volumes.

🔍 Querying Streaming Data 💡

Unlike data‑at‑rest, streams are unbounded; a query
must produce results without waiting for the entire
stream.
Core Strategies
Strategy How it Typical Drawbacks
works use‑case
Continuous Treat each Real‑time Flood of
queries incoming dashboards intermediate
tuple as an that need results;
insert; every heavy for
SQL‑style change.
 queries run high‑rate
continuously, streams.
emitting
updates.
Stream Extend Aggregates Requires
query SQL/relational over fixed explicit
languages algebra with intervals window
window (e.g., hourly definition;
operators; sales). handling
windows out‑of‑order
turn a timestamps
portion of can be
the stream complex.
into a
temporary
relation.
Algebraic User‑defined Custom Requires
operators functions analytics, programming
on streams (UDFs) machine‑learningeffort;
process pipelines. debugging
each tuple; can be
stateful harder.
operators
can
 maintain
aggregates.
Pattern Define Event‑driven May need
matching patterns alerts (e.g., sophisticated
(CEP) and actions; fraud pattern
when a detection). languages;
tuple performance
sequence depends on
matches a pattern
pattern, the complexity.
action fires.

🪟 Stream Extensions to SQL 🕒

Stream‑oriented SQL adds window functions that
are not covered by classic relational windows.
Common Window Types
Tumbling window – non‑overlapping, fixed‑size
intervals (e.g., one‑hour buckets).
Hopping window – fixed size but windows shift
by a step, causing overlaps (e.g., 1‑hour window
every 20 minutes).
Sliding window – a window centered on each
incoming tuple; size defined by time or row
 count.
Session window – groups activity of the same
user separated by inactivity gaps; optionally
limited by a maximum duration.
Azure Stream Analytics Example (SQL)
SELECT item,
[Link] AS window_end,
SUM(amount) AS total_amount
FROM order
TIMESTAMP BY datetime
GROUP BY itemid,
TUMBLINGWINDOW(hour, 1)

[Link] provides the end‑time of

each window.
The query treats order as a stream, while the
result is a relation (one row per hour per item).
Join Semantics for Streams
Stream ↔ Relation – result is a stream;
timestamp of the output equals the stream
tuple’s timestamp.
Stream ↔ Stream – allowed only when the join
condition bounds the time difference (e.g.,
“within 1 hour”) to avoid unbounded state.
⚙️ Algebraic Operations on Streams 🔄
 Processing streams algebraically involves routing
tuples through a directed acyclic graph (DAG) of
operators.
DAG vs. Publish‑Subscribe Routing
Aspect DAG (e.g., Publish‑Subscribe
Apache Storm) (e.g., Apache
Kafka)
Nodes Spouts (data Topics act as
sources) andlogical channels;
bolts producers
(operators). publish,
consumers
subscribe.
Edges Fixed Dynamic; any
connections operator can
defined in a subscribe to any
topology file. topic.
State handling Each bolt may Topics retain
maintain local messages for a
state; messages configurable
are passed to retention period,
downstream enabling replay
bolts. after failures.
 Flexibility Adding/removing Operators can
operators be
requires added/removed
topology restart. at runtime by
(un)subscribing
to topics.
Tuple flow: A tuple emitted by a spout travels
along all outgoing edges, reaching every
downstream bolt (or every subscriber of the
topic).
Fault tolerance: Both models provide
mechanisms (checkpointing in Storm, log‑based
replay in Kafka) to recover from node failures
without losing data.
Apache Spark Structured Streaming adopts the
discretized stream model: the incoming stream is split
into time‑based windows, each window is treated as
a static dataset, and standard algebraic operators
(e.g., groupBy, join) are applied.

📚 Summary of Big Data Foundations (Section 10.7)

Modern applications often process
non‑relational data at scales far beyond a single
 traditional system.
The Internet of Things (IoT) connects countless
sensors and devices, generating continuous
streams of data.
A diverse set of query languages has emerged
to handle varied data types and massive
volume/velocity.
Scaling requires parallel storage (e.g.,
distributed file systems) and parallel processing
(e.g., MapReduce).
Distributed file systems expose a familiar
file‑system interface while spreading files across
many machines.
Key‑value stores (often called NoSQL) provide
fast look‑ups by key and limited query
capabilities.
Parallel/distributed databases retain the
relational interface but store data across multiple
nodes and execute queries in parallel.
📖 Key Terminology
Term Definition
Volume >blockquote>The sheer
amount of data
 generated, stored, and
processed.
Velocity >blockquote>The speed
at which data is created
and must be processed
(e.g., streaming).
Conversion >blockquote>The process
of transforming data from
one format or schema to
another.
Internet of Things (IoT) >blockquote>Network of
sensors and embedded
devices that continuously
emit data.
Distributed File System >blockquote>A system
that stores files across
many machines while
presenting a single
namespace.
NameNode / DataNode >blockquote>In HDFS,
the NameNode manages
metadata; DataNodes
store the actual blocks.
 Sharding >blockquote>Dividing a
large dataset into smaller,
more manageable pieces
(shards).
Partitioning Attribute >blockquote>Column
used to determine how
records are distributed
across shards.
Key‑Value Storage >blockquote>Stores
System records as (key, value)
pairs; retrieval is based on
the key.
Document Store >blockquote>A NoSQL
database that stores
semi‑structured
documents (e.g., JSON).
Shard Key >blockquote>The
attribute whose value
decides the shard
placement of a record.
Parallel Databases >blockquote>Databases
that distribute data and
query processing across
multiple nodes.
 Reduce Key >blockquote>Key on
which the reduce()
function aggregates
values in MapReduce.
Shuffle Step >blockquote>Phase
where map outputs are
transferred to the
appropriate reducers.
Streaming Data >blockquote>Unbounded,
continuously arriving
tuples that must be
processed in near‑real
time.
Data‑at‑Rest >blockquote>Static data
that is stored and
queried, as opposed to
streaming data.
Window (on streams) >blockquote>Finite
subset of a stream (e.g.,
time‑based or
count‑based) treated as a
relation.
Continuous Queries >blockquote>Queries that
run perpetually, emitting
 results as new data arrive.
Punctuations >blockquote>Markers
that indicate no more
data will arrive for a given
time interval, allowing
windows to be closed.
Lambda Architecture >blockquote>Hybrid
design combining batch
processing with real‑time
stream processing.
Tumbling / Hopping / >blockquote>Different
Sliding / Session windowing strategies that
Window control how stream data
are grouped temporally.
Publish‑Subscribe >blockquote>Messaging
Systems platforms where
producers publish to
topics and consumers
subscribe to them.
Discretized Streams >blockquote>Model that
treats a continuous
stream as a sequence of
static micro‑batches.
 Superstep >blockquote>In
bulk‑synchronous
processing, a phase
where all nodes perform
local computation before
synchronizing.

🛠️ Prominent Big‑Data Tools

Apache Hadoop – Open‑source implementation
of MapReduce and HDFS.
Apache Spark – In‑memory cluster computing;
supports batch and streaming (Structured
Streaming).
Apache Tez – DAG‑based engine for algebraic
operators.
Apache Hive – SQL‑like interface on top of
Hadoop, Tez, or Spark.
Apache Impala – Low‑latency SQL engine for
Hadoop.
Apache HBase / Cassandra / MongoDB / Riak
– Distributed key‑value / document stores.
Google BigTable / Amazon DynamoDB –
Hosted, scale‑out key‑value services.
Google Spanner / CockroachDB – Globally
consistent, parallel relational databases.
 Amazon EMR / Azure HDInsight – Managed
cloud services for Hadoop, Spark, Hive, etc.
Apache Kafka – Publish‑subscribe platform for
high‑throughput, fault‑tolerant streaming.
Apache Flink – Stream‑processing engine with
native support for event‑time semantics.
Apache Giraph / Neo4j – Graph‑processing
platforms (Pregel model for Giraph).
📚 Further Reading
Davoudian et al. (2018) – Survey of NoSQL data
stores.
Hadoop documentation – HDFS and MapReduce
details.
Spark homepage – Architecture and
programming guide.
Kafka website – Streaming data platform
concepts.
Flink website – Stream processing fundamentals.
Malewicz et al. (2010) – Pregel bulk‑synchronous
graph processing.
Valiant (1990) – Bulk‑synchronous parallel model.
📊 Chapter 11 – Data Analytics
Definition: Data analytics is the systematic
processing of historic and current data to discover
patterns, build predictive models, and support
decision making.
Decision‑support relies on large‑scale analytics
to maximize business value (e.g., targeted
advertising, inventory planning).
ETL pipeline (Extract, Transform, Load) gathers
data from heterogeneous sources into a unified
data warehouse.
Aggregates & dashboards present concise
summaries; OLAP systems enable near‑real‑time
multidimensional queries.
Statistical analysis tools (R, SAS, SPSS) provide
deeper insight; R integrates with Spark for
parallel execution.
Predictive modeling (e.g., decision trees) uses
past data to forecast outcomes such as loan
default risk.
Machine learning & data mining uncover
hidden patterns and drive automated
predictions.
Business Intelligence (BI) and Decision Support
focus on reporting and aggregation; they differ
 from online transaction processing (OLTP),
which handles small, fast updates.
🏢 11.2 Data Warehousing
Definition: A data warehouse is a centralized
repository that stores integrated data from multiple
operational sources under a unified schema,
preserving historical snapshots for analytical
querying.
Provides a single consolidated interface for
decision‑support workloads, keeping OLTP
systems free from heavy analytic queries.
Supports both relational and non‑relational
sources; schema unification may involve
extensive transformation.
11.2.1 Components of a Data Warehouse
Data Gathering
Source‑driven (push) vs. destination‑driven
(pull) extraction.
Freshness & Replication
Synchronous replication is costly; most
warehouses work with slightly stale data
(e.g., “yesterday’s” data).
 Schema Integration
Reconciling differing source schemas;
often results in a materialized view of
source data.
Data Cleansing & Fuzzy Lookup
Correct typographical errors, standardize
addresses, resolve inconsistent codes.
Deduplication (Merge‑Purge) & Householding
Remove duplicate records; group related
records (e.g., multiple residents at one
address).
Transformation & Enrichment
Unit conversion, joining disparate sources,
applying user‑defined functions.
Update Propagation (View Maintenance)
Synchronize changes from source systems
to the warehouse; analogous to the
view‑maintenance problem.
Summarization
Store aggregated facts (e.g., total sales per
product) instead of raw transactional rows
to reduce storage and speed queries.
11.2.2 Multidimensional Data & Warehouse Schemas
Fact Tables – Record events (e.g., each sale);
contain measure attributes (numeric,
 aggregable) and dimension attributes (foreign
keys).
Dimension Tables – Store descriptive
information (e.g., product, customer, date).
Schema Type Structure
Star Schema Single central fact table
linked directly to multiple
dimension tables.
Snowflake Schema Dimension tables further
normalized into
sub‑dimensions, forming a
hierarchy.
Illustration: Figure 11.2 (star schema) shows a fact
table sales with foreign keys item_id, store_id,
customer_id, date pointing to dimension tables
item_info, store, customer, date_info.
Multidimensional data enable OLAP operations
such as roll‑up, drill‑down, slice, and dice.
11.2.3 Database Support for Data Warehouses
Parallel / Distributed Databases provide the
relational interface while distributing storage and
query processing across many nodes.
 NoSQL key‑value stores (e.g., HBase,
Cassandra) can serve as the underlying storage
layer for massive fact tables, offering high write
throughput.
Hybrid ELT approach (Extract‑Load‑Transform)
leverages the warehouse’s own parallel
processing capabilities (e.g., MapReduce, Spark)
for data transformation, reducing data
movement.
📈 Tools & Platforms for Analytics (Recap)
Apache Hadoop – Batch processing, HDFS
storage.
Apache Spark – In‑memory analytics, supports
both batch and streaming (Structured Streaming).
Apache Hive / Impala – SQL front‑ends for
Hadoop/Spark.
Apache Kafka – Real‑time publish‑subscribe for
streaming pipelines.
Apache Flink – Stream processing with precise
event‑time handling.
Google Cloud Dataflow / Amazon Kinesis –
Managed streaming services.
Neo4j / Apache Giraph – Graph analytics
platforms.
The notes above capture the essential concepts,
terminology, and system components introduced in the
Big‑Data and Data‑Analytics sections, ready for
inclusion in a comprehensive study guide.
🏦 Transaction‑Processing vs. Data‑Warehouse
Systems
Definition: Transaction‑processing databases handle
many small, often update‑heavy queries, while
data‑warehouse databases serve far fewer queries
that scan large data volumes.
Concurrency control
Transaction systems need locking or
multiversion concurrency control (MVCC)
to keep data consistent when reads and
writes overlap.
Data warehouses rarely perform updates
after insertion → no concurrency‑control
overhead (no lock management, no
versioning).
Workload characteristics
Transaction DB: many short queries, high
mix of reads + writes.
 Warehouse DB: few long queries, heavy
scans, inserts only; deletes only to reclaim
space.
Storage layout
Transaction systems traditionally use
row‑oriented storage.
Warehouse systems favor
column‑oriented storage for better
read‑only scan performance.
📊 Storage Layouts: Row‑Oriented vs.
Column‑Oriented
Row‑oriented storage: All attributes of a tuple are
stored together sequentially in a file.
Column‑oriented storage: Each attribute is stored in
its own file; values from successive tuples appear
consecutively.
Aspect Row‑oriented Column‑oriented
Read pattern Suited for Ideal when only
retrieving whole a subset of
tuples attributes is
needed
 I/O for Reads Reads only
selective unnecessary required
queries attributes columns
(wasted
bandwidth)
Cache usage Irrelevant Cache fills with
columns may relevant data
occupy cache only
Compression Mixed‑type data Homogeneous
→ lower column data →
compression higher
ratios compression
Write pattern Single I/O writes Multiple I/Os to
whole tuple write a single
tuple (one per
column)
Typical use OLTP (high OLAP /
case update rate) data‑warehouse
(large scans,
aggregations)
Benefit 1: Queries that touch few columns avoid
loading the rest, improving I/O and cache
efficiency.
 Benefit 2: Uniform column types enable effective
compression, reducing disk space and read time.
Drawback: Fetching a single tuple requires a
separate read for each column, making column
stores unsuitable for high‑frequency
point‑lookups.
Commercial systems that adopt columnar storage for
warehousing include Teradata, Sybase IQ, Amazon
Redshift, and columnar extensions in Oracle, SAP
HANA, SQL Server, and IBM DB2.
🗂️ Data Lakes
Definition: A data lake is a repository that stores raw
data in its original format—structured,
semi‑structured, or unstructured—without enforcing a
common schema up front.
Key traits
Accepts diverse file types (e.g., logs,
JSON, Avro, Parquet).
Defers schema definition → flexible but
requires more effort at query time.
Typical tools
 Apache Hadoop and Apache Spark
provide unified APIs for querying both
structured and unstructured assets.
Contrast with warehouses
Warehouses require upfront ETL to
conform to a unified schema.
Lakes prioritize low ingestion cost;
warehouses prioritize query performance
and schema consistency.
📈 Online Analytical Processing (OLAP)
Definition: OLAP enables interactive analysis of
multidimensional data, supporting operations such as
aggregation, slicing, dicing, and drill‑down.
Purpose: Discover patterns by grouping data
along interesting dimensions (e.g., sales by
product, time, customer segment).
Core concepts
Fact tables contain measurable values
(e.g., quantity).
Dimension tables describe context (e.g.,
item_name, color, clothes_size).
📊 Aggregation on Multidimensional Data
Consider a sales relation:
sales(item_name, color, clothes_size, quantity)

Measure attribute: quantity (numeric,

aggregate‑able).
Dimension attributes: item_name, color,
clothes_size.
Cross‑Tabulation (Pivot‑Table)
Definition: A cross‑tab displays aggregated values
for two dimensions, with optional row/column totals.
Example dimensions: rows = item_name, columns
= color; aggregation = SUM(quantity) over all
clothes_size.
Provides quick insight (e.g., total shirts sold in
pastel colors).
🧊 Data Cubes
Definition: A data cube extends cross‑tabs to n
dimensions, forming an n‑dimensional array where
each cell holds an aggregated measure.
 For sales, a 3‑dimensional cube spans item_name
× color × clothes_size.
Cell identification: Tuple of dimension values
(e.g., (shirt, dark, medium)).
Summary cells: Use the special value all to
indicate aggregation over an entire dimension
(equivalent to row/column totals in a cross‑tab).
Size Considerations
Number of cells = product of cardinalities of all
dimensions.
Example: 4 items × 3 colors × 3 sizes = 36 cells;
with all summaries, the total grows to 80 cells.
🔀 OLAP Operations
Operation Description Typical effect
Pivot Choose different Changes the 2‑D
dimensions for view of the cube
rows and
columns
Slice Fix a single Produces a 2‑D
dimension value sub‑cube
(e.g.,
clothes_size =
large)
 Dice Fix values for Yields a smaller
two or more sub‑cube
dimensions (e.g.,
color = dark and
clothes_size =
medium)
Roll‑up Aggregate to a Reduces detail,
coarser creates summary
granularity (e.g., rows/columns
sum over all
sizes)
Drill‑down Reverse of Increases detail,
roll‑up; move to often by adding
finer granularity dimensions
(requires original
data)
Pivoting is the interactive selection of
dimensions that appear as rows vs. columns.
Slicing/Dicing adds explicit dimension values
atop the cross‑tab, often labelled as all (no
restriction) or a specific value.
Roll‑up and drill‑down exploit dimension
hierarchies (see next section) to navigate
between aggregated and detailed views.
🌐 Dimension Hierarchies
Definition: A hierarchy orders attribute values from
fine to coarse granularity, enabling multi‑level
aggregation.
Time hierarchy example: hour → day → month →
quarter → year.
Location hierarchy example: city → state →
country → region.
Product hierarchy example: item_name →
category (e.g., skirt, dress → womenswear).
Hierarchy levels appear above the cross‑tab when a
higher‑level aggregation is shown (e.g., “all” for
clothes_size).
Analysts can roll up from hour to day or drill
down from year to month.
Hierarchies facilitate flexible reporting without
redefining the underlying data model.