0% found this document useful (0 votes)
3 views22 pages

Big Data Analytics Question Bank

The document is a question bank for a course on Big Data and Analytics, covering multiple modules with questions categorized by Bloom's taxonomy levels. It includes questions on topics such as Big Data definitions, characteristics, analytics types, Hadoop architecture, NoSQL databases, and Hive. Each question specifies the Bloom's level and marks, indicating the complexity and assessment weight of each question.

Uploaded by

nischith.17.r
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
3 views22 pages

Big Data Analytics Question Bank

The document is a question bank for a course on Big Data and Analytics, covering multiple modules with questions categorized by Bloom's taxonomy levels. It includes questions on topics such as Big Data definitions, characteristics, analytics types, Hadoop architecture, NoSQL databases, and Hive. Each question specifies the Bloom's level and marks, indicating the complexity and assessment weight of each question.

Uploaded by

nischith.17.r
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Question Bank: Big Data and Analytics[22CD751]

MODULE 1 and few questions from MODULE 2


SI. NO. Question Bloom's Level Mark
s

1 Define digital data and explain its types with examples. Level 1: 5
Remember

2 List different types of data sources used in analytics. Level 1: 3


Remember

3 Explain the term Big Data with a suitable definition. Level 1: 3


Remember

4 Differentiate between Big Data and Analytics. Level 2: 5


Understand

5 Describe the characteristics (5Vs) of Big Data with Level 2: 6


examples. Understand

6 Give domain specific examples where Big Data is Level 2: 4


applied. Understand

7 Compare Traditional Business Intelligence and Big Data. Level 3: 6


Apply

8 Illustrate a typical data warehouse architecture. Level 3: 5


Apply

9 Illustrate a Hadoop environment with neat diagram. Level 3: 6


Apply

10 What is Big Data Analytics? Give its significance. Level 1: 4


Remember

11 Classify Analytics into types and explain each. Level 2: 6


Understand

12 Explain the importance of Big Data Analytics in business Level 2: 5


decision making. Understand
13 Analyze technologies used in Big Data environments. Level 4: 6
Analyze

14 Compare few top analytical tools used in industry. Level 4: 6


Analyze

15 Explain NoSQL and its advantages over RDBMS. Level 2: 5


Understand

16 Summarize Hadoop components and its ecosystem. Level 2: 6


Understand

17 Design a use case where Big Data Analytics can solve a Level 5: 8
real-world problem. Evaluate

18 Evaluate the challenges faced in Big Data Level 5: 7


implementation. Evaluate

19 Propose a Big Data solution for healthcare or finance Level 6: 8


sector. Create

20 Formulate a data pipeline combining warehouse and Level 6: 8


Hadoop for analytics. Create

21 Differentiate between structured, semi-structured, and Level 2: 5


unstructured data with examples. Understand

22 Explain different types of data sources such as IoT, social Level 2: 5


media, and enterprise systems. Understand

23 Describe the 5Vs of Big Data with real-world examples. Level 2: 6


Understand

24 Illustrate the Big Data Life Cycle with an example use Level 3: 6
case. Apply

25 Explain the stages of Big Data Analytics: Collection, Level 2: 6


Cleaning, Analysis with tools. Understand

26 Compare Data Warehouse vs Hadoop environment with Level 4: 6


proper justification. Analyze

27 Discuss the challenges of Big Data Analytics. Level 4: 6


Analyze

28 Explain types of NoSQL databases with suitable Level 2: 5


examples. Understand
29 Analyze how Spark is used in real-time analytics with Level 4: 7
industry examples. Analyze

30 Design a Big Data architecture for a smart city project. Level 6: 8


Create

31 How would you differentiate OLAP and OLTP systems in terms Level 3: 8
of their data processing roles and use cases? provide examples of Apply
applications that typically use each type?

MODULE 2

Bloom’s
[Link] Question Marks
Level

Explain the reasons behind the emergence of Hadoop as a


1 Understand 8
solution for processing big data.

Compare RDBMS and Hadoop with respect to scalability, data


2 Analyze 8
handling, and fault tolerance.

Explain the architecture of HDFS with NameNode and DataNode


3 Understand 8
roles.

Analyze the Hadoop Distributed File System (HDFS) and discuss


4 its key features, explaining how each feature contributes to Analyze 8
efficient big data storage and processing.

Describe the history and evolution of Hadoop. Highlight its major


5 Remember 8
milestones.

Explain the role of Hadoop YARN in managing resources and


6 Understand 8
applications.

Describe the core components of Hadoop ecosystem and explain


7 Understand 8
their functions.

Explain the working of MapReduce programming with an


8 Understand 8
example.

Differentiate between Mapper, Reducer, Combiner, and


9 Analyze 8
Partitioner in Hadoop MapReduce.
Bloom’s
[Link] Question Marks
Level

Describe the process of sorting and searching in MapReduce


10 Understand 8
with a suitable example.

Explain how compression improves the efficiency of Hadoop


11 Understand 8
processing.

Analyze a real-world use case where Hadoop provides a better


12 Analyze 8
solution than traditional RDBMS.

Explain the process of data replication in HDFS and discuss its


13 Understand 8
importance in fault tolerance.

Discuss how Hadoop handles large-scale data analytics across


14 Analyze 8
distributed systems.

Explain the role of NameNode, Secondary NameNode, and


15 Understand 8
DataNode in HDFS.

Describe the process of resource scheduling and job execution in


16 Understand 8
Hadoop YARN.

Compare MapReduce and traditional batch processing


17 Analyze 8
techniques, highlighting the advantages of MapReduce.

Explain the concept of a Partitioner in MapReduce and give an


18 Apply 8
example scenario where custom partitioning is useful.

Discuss the importance of fault tolerance and data reliability in


19 Analyze 8
HDFS with examples.

Explain the end-to-end process of running a MapReduce job on


20 Understand 8
Hadoop, from input to output.

MODULE 3

SI.
Question Bloom’s Level Marks
No

Define the CAP Theorem and explain its three components Level 1:
1 10
with an example. Remember
SI.
Question Bloom’s Level Marks
No

Classify the types of NoSQL databases and mention two Level 2:


2 10
use-cases for each type. Understand

Compare the terms used in RDBMS and MongoDB (e.g., Level 2:


3 8
table vs collection, row vs document). Understand

Illustrate CRUD operations in MongoDB with syntax and


4 Level 3: Apply 6
examples.

Demonstrate how to find documents using search criteria


5 Level 3: Apply 5
(e.g., AND/OR, range queries) in MongoDB.

Explain how null values are handled in MongoDB with Level 2:


6 4
examples. Understand

Write and explain queries using count(), limit(), sort(), and


7 Level 3: Apply 6
skip() in MongoDB.

Describe the use of aggregate functions in MongoDB with


8 Level 4: Analyze 7
appropriate examples.

Analyze how the CAP theorem influences MongoDB’s


9 Level 4: Analyze 6
design choices.

Compare MongoDB with other NoSQL databases in terms of


10 Level 4: Analyze 6
data model and consistency.

Design a MongoDB collection structure for an e-commerce Level 5:


11 8
application including CRUD queries. Evaluate

Formulate an aggregation pipeline to group and summarize


12 Level 6: Create 8
data in MongoDB.

SI.
Question Bloom’s Level Marks
No

Write MongoDB commands to insert a single document and


13 Apply 5
multiple documents into a collection named Students.
SI.
Question Bloom’s Level Marks
No

Demonstrate how to update one document and update many


14 Apply 6
documents with appropriate MongoDB syntax.

How do you delete one document and delete many Understand /


15 5
documents in MongoDB? Give examples. Apply

Write a query to retrieve only the first 5 documents from a


16 Apply 4
collection using limit(). Explain the syntax.

Show how to sort documents in ascending and descending


17 Apply 5
order using sort(). Use a price or age field as an example.

Explain the purpose of the skip() function. Write a query to skip Understand /
18 5
the first 10 documents and return the next 5. Apply

Combine sort() and limit() in a single query to display the top 3


19 Analyze 6
highest paid employees from the Employee collection.

Write a CRUD-based scenario: Insert student records, update


20 one record, delete one record, and display the remaining data Create 8
using sort() and limit().

Write MongoDB queries to: i) Insert three new documents into a


Student collection. ii) Retrieve students with marks > 75. iii) Update
21 Apply 10
the department of a given student. iv) Delete students with marks <
35.

Using a sample collection, demonstrate the commands for Create,


Understand /
22 Read, Update, and Delete in MongoDB. Explain how each command 10
Apply
affects the data.

With a Student collection, perform: i) insertOne() ii) insertMany() iii)


23 updateOne() iv) updateMany(). Provide appropriate examples with Apply 10
syntax.

For an Employee collection, write a query to: i) Retrieve documents


where the Skills array contains both “Communication” and
24 Analyze 10
“Recruitment”. ii) Sort the results by Dept in ascending order iii) Limit
the result to 5 documents. Explain each part.
Explain the CAP theorem in the context of NoSQL databases like
25 MongoDB. How do consistency, availability, and partition tolerance Understand 10
influence design choices?

Compare and contrast key-value, document, column-family, and Understand /


26 10
graph-based NoSQL databases with examples. Analyze

MODULE 4

SI Bloom’s CO
Question Marks
No Level Mapping

Describe the key features and purpose of Hive in Big Data


1 L1 8 CO1
processing.

2 Explain the architecture of Hive and its main components. L2 8 CO1

List and describe the various data types supported by Hive.

Hive supports several data types that can be applied to store


structured and semi-structured data efficiently:

1. Numeric Data Types

●​ TINYINT, SMALLINT, INT, BIGINT – used for storing


integer values of different ranges.

●​ FLOAT, DOUBLE, DECIMAL – used for decimal and


floating-point numbers.​
Example: INT for age, DECIMAL(10,2) for salary.
3 L1 8 CO1
2. String Data Types

●​ STRING – general-purpose text.

●​ CHAR(n) – fixed-length string.

●​ VARCHAR(n) – variable-length string with a limit.​


Example: STRING for customer name, CHAR(2) for
state codes.

3. Date and Time Types

●​ DATE – stores date values.


SI Bloom’s CO
Question Marks
No Level Mapping

●​ TIMESTAMP – stores date and time.​


Example: DATE for joining date, TIMESTAMP for
order timestamp.

4. Complex Data Types

●​ ARRAY<type> – ordered collection of elements.

●​ MAP<key_type, value_type> – key-value pairs.

●​ STRUCT – collection of named fields.​


Example: ARRAY<STRING> for multiple phone
numbers, MAP<STRING, INT> for product quantity
mapping, STRUCT<name:STRING, age:INT> for
storing nested customer info.

5. Usage Application

●​ Hive data types help ensure data integrity, efficient


storage, and query optimization.

●​ By applying the correct type to each column, large


datasets can be processed faster and more accurately
in Hive queries.

4 Describe the different Hive file formats and their uses. L2 8 CO1

Explain the role of Hive Query Language (HQL) in querying


Big Data.

Hive Query Language (HQL) is a SQL-like language designed


to query, manage, and analyze large datasets stored in
Hadoop. Applying HQL allows users to perform data
5 operations without writing complex MapReduce programs. L2 8 CO2

1. Role of HQL in Big Data Analytics

●​ Data Retrieval: Enables querying large datasets


stored in HDFS using familiar SQL syntax (SELECT,
WHERE, GROUP BY, ORDER BY).
SI Bloom’s CO
Question Marks
No Level Mapping

●​ Data Manipulation: Supports operations like


INSERT, UPDATE, DELETE, and LOAD DATA to manage
tables and partitions.

●​ Data Aggregation: Provides aggregation functions


(COUNT, SUM, AVG, MAX, MIN) to summarize large
datasets efficiently.

●​ Data Transformation: Supports joins, subqueries,


and views for transforming raw data into meaningful
insights.

2. Advantages

●​ Simplifies Big Data querying without needing


low-level MapReduce coding.

●​ Integrates with other tools like Spark for advanced


analytics.

●​ Supports structured, semi-structured, and partitioned


data efficiently.

3. Example

-- Count total orders per customer

SELECT customer_id, COUNT(*) AS total_orders

FROM orders

GROUP BY customer_id

ORDER BY total_orders DESC;

●​ This HQL query efficiently aggregates millions of


order records to provide insights on customer
activity.

4. Application

●​ HQL is widely applied in reporting, analytics, and


ETL pipelines in Big Data ecosystems, enabling quick
insights from huge datasets.
SI Bloom’s CO
Question Marks
No Level Mapping

Demonstrate the use of RC File implementation in Hive for


6 L3 8 CO2
data storage.

Apply a User Defined Function (UDF) in Hive on a sample


7 dataset and explain its effect. L3 8 CO2

Compare Hive and Pig in terms of purpose, execution, and


data handling.

In the Hadoop ecosystem, Apache Hive and Apache Pig are


two high-level tools built on top of MapReduce to simplify
Big Data processing.

●​ Hive is primarily used for data analysis and querying


large structured datasets.

●​ Pig is designed for data transformation and


manipulation using a simple scripting approach.

Both tools convert user commands into MapReduce jobs


internally but differ in purpose, language, and usage pattern.

8 L4 8 CO3
SI Bloom’s CO
Question Marks
No Level Mapping

Analyze the given E-commerce Sales dataset and identify


9 suitable Hive data types for each attribute.[minimum 10 L4 8 CO3
attributes]

10 Explain the concept and benefits of Pig in Big Data analytics. L1 8 CO1

11 Describe the anatomy of Pig and its main components. L2 8 CO1

Explain the philosophy behind Pig and its advantages over


12 L2 8 CO2
traditional MapReduce programming.

Demonstrate the use of relational operators in Pig with an


13 L3 8 CO2
example dataset.

Apply eval functions in Pig to transform data in a given


14 L3 8 CO2
dataset.

Analyze complex data types in Pig and provide suitable use


cases.

1. Complex Data Types in Pig

Pig supports several complex data types to handle nested or


structured data:

1.​ Tuple

o​ Definition: An ordered set of fields, which


can hold multiple values of different data
types.
15 L4 8 CO3
o​ Example: (name, age, city) → ('John', 25,
'Bangalore')

o​ Use Case: Representing a single record with


multiple attributes.

2.​ Bag

o​ Definition: An unordered collection of tuples;


allows duplicates.

o​ Example: {(‘John’, 25), (‘Alice’, 30)}


SI Bloom’s CO
Question Marks
No Level Mapping

o​ Use Case: Storing multiple records of a


dataset, such as all orders by a customer.

3.​ Map

o​ Definition: Key-value pairs; keys are strings,


values can be any data type.

o​ Example: [ ‘product’: 'Laptop', ‘price’: 50000 ]

o​ Use Case: Storing attributes of a record with


dynamic or variable fields.

2. Analysis

●​ Complex data types allow hierarchical and


semi-structured data to be stored and processed
efficiently.

●​ They enable Pig to handle nested JSON or XML-like


data, which is common in Big Data environments.

●​ Using tuples, bags, and maps reduces the need to


flatten data before processing.

3. Practical Use Cases

●​ E-commerce analytics: Use bags to store all items


purchased in a transaction.

●​ Web logs: Use maps to store key-value pairs of user


actions (page, timestamp, duration).

●​ Social media data: Tuples to represent user profiles


with multiple attributes (name, location, interests).

4. Summary

●​ Complex data types in Pig provide flexibility,


efficiency, and scalability for Big Data processing.

●​ Proper use of these types simplifies data


transformation, aggregation, and analysis on large
datasets.
SI Bloom’s CO
Question Marks
No Level Mapping

16 Explain Piggy Bank and its use in extending Pig functionality. L2 8 CO2

Compare the execution modes of Pig and discuss when each


17 L4 8 CO3
mode is preferred.

Design a solution using Hive to query a large dataset


18 L6 8 CO4
efficiently.

Develop a Pig script to process a dataset and extract


19 L6 8 CO4
meaningful insights.

Propose a combined Hive and Pig workflow to handle a Big


20 L6 8 CO4
Data analytics task.

MODULE 5

SI Bloom’s CO
Question Marks
No Level Mapping

1 Describe the key features and components of Apache Spark. L1 8 CO1

Explain the concept of a unified Spark platform and its


2 L2 8 CO1
advantages.

Identify the main users of Spark and their roles in Big Data
3 L1 8 CO1
analytics.

Describe the different storage layers in Spark and their


purposes.

Apache Spark has multiple storage layers that allow


efficient handling of Big Data across memory and disk.
4 These layers optimize computation, storage, and retrieval in L2 8 CO2
distributed environments.

1. Memory (RAM)

●​ Description: Spark primarily stores data in memory


(RAM) for fast access.
SI Bloom’s CO
Question Marks
No Level Mapping

●​ Purpose: Enables low-latency computations and


supports iterative algorithms (e.g., machine
learning, graph processing).

●​ Example: Caching intermediate RDDs in memory


speeds up repeated transformations.

2. Disk Storage

●​ Description: Data that does not fit in memory is


stored on disk.

●​ Purpose: Provides reliability and fault tolerance;


acts as a backup when memory is insufficient.

●​ Example: Large datasets in HDFS can be processed


without exhausting memory resources.

3. External Storage Systems

●​ Description: Spark can integrate with external


storage like HDFS, S3, Hive, or JDBC-compliant
databases.

●​ Purpose: Allows persistent storage and access to


distributed datasets across clusters.

●​ Example: Reading from an S3 bucket or writing


processed data to Hive tables.

4. Caching and Persistence Layers

●​ Description: Spark allows persisting RDDs or


DataFrames in memory, disk, or both with different
storage levels:

o​ MEMORY_ONLY, MEMORY_AND_DISK,
DISK_ONLY, etc.

●​ Purpose: Optimizes performance for iterative


computations and repeated data access.

●​ Example: Persisting a dataset used in multiple ML


algorithm iterations.
SI Bloom’s CO
Question Marks
No Level Mapping

Demonstrate how to create RDDs from existing datasets in


5 L3 8 CO2
Spark.

6 Apply common RDD transformations on a sample dataset. L3 8 CO2

7 Apply RDD actions to extract insights from a sample dataset. L3 8 CO2

8 L2 8 CO2

Analyze the differences between RDD transformations and


9 L4 10 CO3
actions.

Compare the performance of Spark RDDs with Hadoop


10 L4 10 CO3
MapReduce for large datasets.

Evaluate the benefits of persisting RDDs for iterative


algorithms.

In Apache Spark, RDDs (Resilient Distributed Datasets)


can be persisted (cached) in memory or disk, which is
highly beneficial for iterative algorithms such as machine
learning, graph processing, and iterative data
transformations.

1. Faster Computation

●​ Persisting RDDs avoids recomputing the same


11 dataset across multiple iterations. L5 8 CO4
●​ Iterative algorithms, like gradient descent, often
reuse the same dataset multiple times.

●​ Example: MLlib’s K-Means clustering requires


repeated access to feature data. Persisting the RDD
reduces computation time.

2. Reduced I/O Overhead

●​ Without persistence, Spark recomputes RDDs from


the source or parent transformations, causing
additional disk and network I/O.
SI Bloom’s CO
Question Marks
No Level Mapping

●​ Persisting RDDs minimizes repeated data


shuffling across the cluster.

3. Fault Tolerance

●​ Persisted RDDs maintain lineage information,


allowing Spark to recompute only missing partitions
in case of node failure.

●​ This ensures reliability without fully recomputing


all iterations.

4. Flexible Storage Levels

●​ Spark provides multiple persistence levels:

o​ MEMORY_ONLY – fastest, keeps data in RAM

o​ MEMORY_AND_DISK – falls back to disk if


memory is insufficient

o​ DISK_ONLY – stores on disk for large datasets

●​ Users can choose based on data size and cluster


resources.

Design a Spark program using RDDs to perform a specific


12 L6 8 CO4
data analysis task.

Explain the basics of a Data Lakehouse and its role in


13 L2 8 CO1
modern data architectures.

Analyze different data loading strategies in a Data


Lakehouse.

In a Data Lakehouse, selecting the right data loading


strategy is critical to ensure efficient storage, fast
14 L4 10 CO3
querying, and high-quality analytics. Data can be
structured, semi-structured, or unstructured, requiring
different approaches.
SI Bloom’s CO
Question Marks
No Level Mapping

1. Batch Loading

●​ Description: Load large volumes of data


periodically (daily, weekly).

●​ Use Case: Historical transactional data, periodic ETL


jobs.

●​ Advantages:

o​ Simple to implement

o​ Suitable for large, static datasets

●​ Disadvantages:

o​ Not real-time

o​ Higher latency for analytics

2. Streaming/Real-Time Loading

●​ Description: Data is ingested continuously as it


arrives using tools like Kafka, Spark Streaming, or
Flink.

●​ Use Case: Clickstream data, IoT sensor data,


real-time logs.

●​ Advantages:

o​ Low-latency analytics

o​ Supports near real-time dashboards and


alerts

●​ Disadvantages:

o​ More complex infrastructure

o​ Higher processing overhead


SI Bloom’s CO
Question Marks
No Level Mapping

3. Incremental Loading

●​ Description: Only new or changed data is loaded


into the Lakehouse.

●​ Use Case: Customer or product master tables


updated daily.

●​ Advantages:

o​ Reduces data movement and storage


requirements

o​ Faster than full batch reloads

●​ Disadvantages:

o​ Requires tracking of changes (CDC or


timestamps)

4. Hybrid Loading

●​ Description: Combines batch and streaming


approaches.

●​ Use Case: Large historical data is loaded via batch,


new data is ingested via streaming.

●​ Advantages:

o​ Optimizes performance and freshness

o​ Flexible for diverse workloads

Evaluate best practices for performance optimization in a


Data Lakehouse.

A Data Lakehouse combines the benefits of Data Lakes and


15 Data Warehouses, allowing analytics on structured, L5 8 CO4
semi-structured, and unstructured data. Optimizing
performance is crucial for fast, reliable queries and efficient
data processing.
SI Bloom’s CO
Question Marks
No Level Mapping

1. Data Partitioning

●​ Description: Dividing large tables into smaller


partitions based on key columns (e.g., date, region).

●​ Benefit: Reduces query scan size and improves read


performance.

●​ Example: Partitioning sales data by month for faster


aggregation queries.

2. Data Clustering/Sorting

●​ Description: Organizing data within partitions


based on frequently queried columns.

●​ Benefit: Speeds up filtering and range queries.

●​ Example: Clustering logs by user ID for faster


retrieval of user activity.

3. Efficient File Formats

●​ Description: Use columnar formats like Parquet


or ORC instead of row-based formats.

●​ Benefit: Reduces I/O, enables predicate pushdown,


and improves compression.

●​ Example: Storing transactional tables in ORC format


for analytics.

4. Caching Frequently Accessed Data

●​ Description: Store hot data in memory or faster


storage layers.

●​ Benefit: Reduces latency for repeated queries and


analytical workloads.

●​ Example: Caching a product catalog in Spark for


repeated analytics.

5. Data Compaction
SI Bloom’s CO
Question Marks
No Level Mapping

●​ Description: Combine small files into larger ones to


reduce metadata overhead.

●​ Benefit: Improves query planning and reduces


storage system load.

●​ Example: Merging multiple daily log files into


monthly files.

6. Query Optimization

●​ Description: Use query engines (Spark SQL, Presto)


with cost-based optimizers and predicate
pushdown.

●​ Benefit: Efficient execution plans reduce CPU,


memory, and I/O usage.

7. Monitoring and Tuning

●​ Description: Regularly monitor query performance


and tune cluster resources.

●​ Benefit: Ensures consistent performance under


varying workloads.

Anlyze the advantages and limitations of using RDDs


16 L4 8 CO2
compared to DataFrames and Datasets in Spark.

Analyze Spark’s storage layers for efficiency and scalability


17 L4 10 CO3
in a Big Data project.

Explain the architecture and key characteristics of Resilient


Distributed Datasets (RDDs) in Spark. Discuss how
immutability and fault tolerance are implemented.

1. Architecture of RDDs
18 RDDs are organized as a distributed collection of L2 8 CO4
partitions, where each partition is processed on different
nodes of a cluster. The architecture includes:

●​ Driver Program: Creates SparkContext and defines


RDD operations.
SI Bloom’s CO
Question Marks
No Level Mapping

●​ Cluster Manager: Allocates resources (YARN, Mesos,


Kubernetes, or Standalone).

●​ Executors: Execute tasks on data partitions.

●​ Partitions: Logical chunks of data distributed across


the cluster.

Spark applies lazy evaluation, meaning transformations are


not executed until an action is called.

2. Key Characteristics of RDDs

✅ Distributed​
Data is automatically split across multiple nodes, enabling
parallel computation.

✅ Immutable​
Once created, an RDD cannot be modified. Any change
results in the creation of a new RDD.

✅ Lazy Evaluation​
Transformations (e.g., map, filter) build a lineage graph
rather than executing immediately.

✅ In-Memory Computation​
RDDs can be cached in memory for faster iterative
processing.

✅ Typed Data​
RDDs can store any kind of data, like strings, objects,
numbers, etc.

3. Immutability in RDDs

Immutability ensures better consistency and parallel


execution:

●​ When a transformation is applied, a new RDD is


generated instead of altering the original.
SI Bloom’s CO
Question Marks
No Level Mapping

●​ Immutability avoids conflicts in distributed


execution.

●​ It makes RDD lineage tracking easier for fault


recovery.

Example:​
If rdd1 undergoes map(), Spark creates rdd2 instead of
changing rdd1.

4. Fault Tolerance in RDDs

RDDs are fault tolerant due to lineage:

●​ Spark tracks the sequence of transformations used


to create an RDD.

●​ If a partition fails, Spark recomputes only the lost


partition using the lineage graph, not the entire
dataset.

●​ No need for data replication like Hadoop.

For additional safety, developers may cache or persist


RDDs in memory or disk.

Example storage levels:

●​ MEMORY_ONLY

●​ MEMORY_AND_DISK

●​ DISK_ONLY

19 L6 8 CO4

20 L5 8 CO4

You might also like