Big Data Analytics - Complete Questions and

Answers

1. Justify that Spark and Python provide powerful Big Data

Analysis tools

Answer (Simple & Exam-Ready):

Apache Spark and Python together form a very powerful Big Data analysis platform because
they combine high speed processing, easy programming, and rich data analytics libraries.

1. High Speed In-Memory Processing (Spark)

Spark stores data in RAM instead of disk.

This makes it up to 100 times faster than Hadoop for data analysis.
Intermediate results are also stored in memory for quick access.

2. Support for All Types of Data

Spark can process:

Structured data
Semi-structured data
Unstructured data

Data can come from HDFS, Cassandra, Hive, S3, JSON, Parquet, etc.

3. Powerful Spark SQL for Analytics

Spark SQL allows:

SQL queries on big data

ETL operations
Ad-hoc querying

Supports HiveQL, JDBC, ODBC for database connectivity.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

4. Strong Python Integration (PySpark)

Python easily connects with Spark using PySpark.

Python is:
Easy to write and understand
Very expressive
Used with powerful libraries like:
NumPy
SciPy
Pandas
User Defined Functions (UDFs)

5. Advanced Analytics & Machine Learning

Spark provides:

MLlib for machine learning

GraphX for graph analytics
Spark Streaming for real-time data

Python helps in building ML models easily using these tools.

6. Interactive and Real-Time Analysis

Spark provides interactive shell in Python and Scala.

Supports real-time streaming data analysis.

✅ Conclusion:

Spark provides speed, scalability and big data processing power, while Python provides
simplicity and rich analytics libraries. Together, Spark + Python form a powerful, fast, and
flexible Big Data Analysis toolset.

2. Discuss the method of creation of RDDs using Transform

and Action commands

Answer (Simple & Exam-Ready):

generated by "Markdown to PDF Fast Converter" 👉 [Link]
In Apache Spark, an RDD (Resilient Distributed Dataset) is the core data structure used for
distributed data processing. RDDs are created and processed using two types of operations:

1. Transformations
2. Actions

1. Creation of RDD using Transformations

Transformation is an operation that creates a new RDD from an existing RDD. It is lazy,
meaning it does not execute immediately.

✅ Common Transformation Operations:

map() – applies a function to each element

filter() – selects required elements

flatMap() – splits and flattens data

distinct() – removes duplicate values

groupByKey() – groups data by key

reduceByKey() – reduces values by key

✅ Example:

RDD1 = [Link](x → x*2)

This creates a new RDD (RDD1) but does not execute immediately.

✅ Important Points:

Transformations do not produce output directly.

They only define how data should be processed.
Execution happens only after an action is called.
RDDs created by transformations are fault-tolerant and partitioned for parallel
processing.

2. Creation of RDD using Actions

Actions are operations that trigger the actual execution of transformations and produce
output.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

✅ Common Action Operations:

collect() – returns all elements to the driver

count() – returns number of elements

reduce() – reduces elements using a function

first() – returns first element

take(n) – returns first n elements

saveAsTextFile() – saves RDD to storage

✅ Example:

[Link]()

This executes all previous transformations and returns the result.

✅ Important Points:

Actions trigger real computation.

Results are either:
Returned to the driver, or
Written to external storage.
After an action, Spark executes the full DAG (Directed Acyclic Graph) of transformations.

3. Method of RDD Creation using Transform + Action (Step-wise)

1. Data is loaded into Spark from HDFS, S3, JSON, etc.

2. Initial RDD is created.
3. Transformations are applied to generate new RDDs.
4. Action command is called.
5. Spark executes all transformations and produces output.

✅ Conclusion:

Transformations are used to create new RDDs lazily. Actions are used to execute
transformations and get results. Together, Transform + Action form the complete RDD
processing workflow in Spark.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

3. Explain the concept of DataFrames (SchemaRDD) in Spark.
How are DataFrames created from Parquet, JSON, Hive, or
RDD sources? Explain Text Mining in detail.

A. Concept of DataFrames (SchemaRDD) in Spark

DataFrame is a distributed collection of data organized into named columns, similar to a

table in a database. It is also called SchemaRDD because it has a schema (structure).

✅ Key Features of DataFrames:

Data is stored in row and column format.

Supports operations like:
filter()

groupBy()

join()

aggregation
DataFrames are optimized using Spark SQL engine.
A DataFrame can be converted into an RDD, and an RDD can also be converted into a
DataFrame.

B. Creation of DataFrames from Different Sources

DataFrames can be created from multiple data sources such as Parquet, JSON, Hive, external
databases, and RDDs.

1. Creating DataFrame from Parquet Files

Parquet is a column-oriented storage format.

Spark can directly read Parquet files and create a DataFrame.
It is fast for long analytical queries.
Metadata is stored with the file for easy access.

2. Creating DataFrame from JSON Files

JSON is a semi-structured data format.

generated by "Markdown to PDF Fast Converter" 👉 [Link]
 Spark reads JSON and automatically infers schema.
Each JSON record becomes a row in the DataFrame.

3. Creating DataFrame from Hive Tables

Spark SQL supports Hive integration using HiveContext.

Spark can directly read Hive tables as DataFrames.
Supports:
HiveQL queries
UDFs
SerDe (Serialization/Deserialization)

4. Creating DataFrame from RDD

An RDD of Row objects can be converted into a DataFrame.

Requires defining a schema manually.
Used when data is already present in RDD format.

C. Explanation of Text Mining (In Detail)

✅ Definition of Text Mining

Text Mining is the process of extracting useful information, knowledge, patterns, and
relationships from unstructured textual data such as documents, emails, websites, and
messages.

D. Areas and Applications of Text Mining

1. Natural Language Processing (NLP) – Understanding and processing human language.

2. Information Retrieval (IR) – Searching and retrieving required documents.
3. Information Extraction (IE) – Extracting structured information from unstructured text.
4. Document Clustering – Grouping similar documents into clusters.
5. Document Classification – Assigning documents to predefined categories.
6. Summarization – Generating short versions of long documents.
7. Categorization and Filtering – Organizing documents into semantic groups.
generated by "Markdown to PDF Fast Converter" 👉 [Link]
E. Text Mining Process (Phases)

Phase 1: Text Preprocessing

Noise removal
Tokenization
POS Tagging
Word sense disambiguation
Parsing

Phase 2: Feature Generation

Bag of Words (BoW)

Stemming / Lemmatization
Stop word removal
TF–IDF weighting

Phase 3: Feature Selection

Dimensionality reduction:
PCA
LDA

Phase 4: Classification / Clustering

Supervised: Naïve Bayes, SVM, Decision Tree

Unsupervised: K-Means

Phase 5: Analysis

Visualization
Interpretation of results

✅ Conclusion:

DataFrames provide a structured, fast, and efficient way to process Big Data in Spark. They
can be created from Parquet, JSON, Hive tables, and RDDs. Text Mining is the complete
process of extracting meaningful knowledge from unstructured text using preprocessing,
feature extraction, and machine learning techniques.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

5. Summarize various link analysis metrics used for analytics.
How is PageRank iterated and computed using relative
authority of in-linking pages? How does ranking algorithm
compute topic-sensitive PageRank?

A. Link Analysis Metrics Used for Web Analytics (Summary)

Link analysis metrics are used to measure importance, influence, and connectivity of web
pages or nodes in a graph.

✅ Major Link Analysis Metrics:

1. In-Degree

Number of incoming links to a page.

Indicates popularity or visibility.

2. Out-Degree

Number of outgoing links from a page.

Shows how many pages it references.

3. Closeness Centrality

Measures how close a page is to all other pages.

Uses shortest path distance.

4. Betweenness Centrality

Shows how often a page lies on shortest paths between other pages.
Indicates control over information flow.

5. PageRank

Measures authority of a page based on quality of in-links.

More important pages pass more rank.

6. Hubs and Authorities

Hub: Page that links to many authority pages.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

 Authority: Page that receives many links from hubs.

7. Communities, Triangle Count, Clustering Coefficient, K-Neighbourhood

Used for community detection and network analysis.

B. Computation of PageRank Using Relative Authority of In-Linking Pages

✅ Basic Idea:

A page gets higher rank if it is linked by important pages. Rank passed from a parent page is:

Proportional to parent's authority

Inversely proportional to its number of out-links

✅ PageRank Formula (Relative Authority Method):

If a page v has parent pages u ∈ PA(v), then:

R(v) = Σ [R(u) / ch(u)] × nc

Where:

R(v) = PageRank of page v

R(u) = Rank (authority) of parent page u
ch(u) = Number of out-links (children) of u
PA(v) = Set of parent pages linking to v
nc = Normalization constant (sum of all ranks = 1)

✅ Iteration Process of PageRank:

1. Assign initial equal rank to all pages.

2. Distribute rank from each page equally to its out-links.
3. Recalculate ranks using the formula.
4. Repeat the process (iterations).
5. Ranks converge to a stable value after few iterations.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

C. Topic-Sensitive PageRank Computation

✅ Meaning:

Topic-Sensitive PageRank gives different PageRank values for different topics. A page may
be important for one topic but not for another.

✅ How Topic-Sensitive PageRank Works:

1. Topics are defined

Each topic has a bag of related words.

1. Surfing Probability is Modified

User is more likely to visit pages related to that topic.

1. Topic-Based Weights are Assigned

Pages containing topic words get higher probability weight.

1. Topic-Specific Rank Rₜₛ(v) is Computed

Same PageRank formula is used,

But with topic-biased probabilities.

1. Final Rank Depends on Topic

One page can have high rank for "Education" topic and
Low rank for "Sports" topic.

✅ Final Conclusion (Exam-Ready):

Link analysis metrics like In-degree, Betweenness, Closeness, PageRank, Hubs and
Authorities help in measuring importance and influence of web pages.

PageRank using relative authority computes rank based on:

Parent's importance
Number of out-links
Iterative rank distribution until convergence.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

Topic-Sensitive PageRank modifies ranking by adding topic-based surfing probabilities, so
that ranks become topic-dependent instead of global.

6. Discuss five phases and steps in the phases during Text

Analytics

Text Analytics (Text Mining) is the process of extracting useful information and knowledge
from unstructured text data. It is carried out in five main phases, and each phase has specific
steps.

Phase 1: Text Preprocessing

This phase cleans and prepares raw text for analysis.

Steps:

Noise removal – Removing unwanted symbols, HTML tags, errors.

Tokenization – Splitting text into words or tokens.
Parts-of-Speech (POS) tagging – Identifying nouns, verbs, adjectives, etc.
Word sense disambiguation – Choosing the correct meaning of a word.
Parsing – Creating grammatical structure of sentences.

✅ Purpose: Convert raw text into cleaned and structured text.

Phase 2: Feature Generation

This phase converts text into numerical form so that machines can process it.

Steps:

Bag of Words (BoW) – Represents documents as a collection of words.

Stemming / Lemmatization – Converting words to root form.
Stop word removal – Removing common words like is, the, in.
TF–IDF weighting
TF (Term Frequency) – How often a word appears in a document.
IDF (Inverse Document Frequency) – How unique the word is across all documents.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

✅ Purpose: Convert text into meaningful numerical features.

Phase 3: Feature Selection

This phase selects only important features and removes unnecessary ones.

Steps:

Dimensionality Reduction using:

PCA (Principal Component Analysis)
LDA (Linear Discriminant Analysis)
Feature evaluation using:
Mutual Information
Chi-square test

✅ Purpose: Improve accuracy and reduce computation time.

Phase 4: Classification / Clustering

This phase applies machine learning algorithms to analyze the text.

Steps:

Supervised Learning (Labeled Data):

Naïve Bayes
Support Vector Machine (SVM)
Decision Tree
Unsupervised Learning (Unlabeled Data):
K-Means clustering

✅ Purpose: Group or classify documents automatically.

Phase 5: Analysis

This phase evaluates and interprets the final results.

Steps:

Evaluation of results
generated by "Markdown to PDF Fast Converter" 👉 [Link]
 Visualization
Graphs
Word clouds
Topic maps
Interpretation of relationships and patterns

✅ Purpose: Extract knowledge and support decision making.

✅ Final Exam-Ready Conclusion:

The five phases of Text Analytics are:

1. Text Preprocessing – Clean and prepare data

2. Feature Generation – Convert text into numeric features
3. Feature Selection – Select important features
4. Classification / Clustering – Apply ML algorithms
5. Analysis – Evaluate and visualize results

Together, these phases convert raw unstructured text into useful knowledge for decision
making.

7. List the steps in Document Clustering method. How are

clusters used for fast information retrieval or filtering?
(Example: Student Grade Cards)

A. Steps in the Document Clustering Method

Document Clustering groups similar text documents into clusters automatically using
unsupervised learning. It is widely used for fast retrieval and filtering of information.

✅ Step 1: Data Collection

Collect all text documents.

Example: Student grade cards of all departments.

✅ Step 2: Text Preprocessing

generated by "Markdown to PDF Fast Converter" 👉 [Link]

 Noise removal
Tokenization
Stop word removal
Stemming / Lemmatization
This cleans the text data.

✅ Step 3: Feature Generation

Convert text into numerical vectors using:

Bag of Words (BoW)
TF–IDF

✅ Step 4: Feature Selection / Reduction

Remove irrelevant features.

Use PCA or LDA to reduce dimensions.

✅ Step 5: Apply Clustering Algorithm

Use K-Means clustering or similar algorithm.

Documents with similar features fall into the same cluster.

✅ Step 6: Cluster Evaluation & Labeling

Analyze the formed clusters.

Assign meaningful labels if required.

B. Use of Clusters for Fast Information Retrieval & Filtering (Student Grade
Cards Example)

🎓 Example: Student Grade Cards of a University

Assume we have thousands of student grade cards from different streams.

✅ Step 1: Clustering the Grade Cards

Grade cards are grouped into clusters such as:

Cluster 1: High performers (Distinction)

Cluster 2: First Class students
Cluster 3: Second Class students

generated by "Markdown to PDF Fast Converter" 👉 [Link]

 Cluster 4: Failed / Backlog students

✅ Fast Information Retrieval

If the admin wants:

"All Distinction students" → System directly searches only Cluster 1, instead of all
records.

✅ This reduces search time greatly.

✅ Filtering Using Clusters

Examples of filtering:

Filter students eligible for scholarships → Use High performer cluster

Filter students needing remedial classes → Use Fail / Low-grade cluster
Filter department-wise performance → Use subject-based clusters

✅ Benefits in Grade Card System

Very fast searching

Efficient filtering
Easy performance analysis
Better decision making
Reduced computational cost

✅ Final Exam-Ready Conclusion:

Document clustering steps include: data collection, preprocessing, feature generation,

feature selection, clustering, and evaluation.

Clusters help in fast retrieval and filtering by limiting the search to only the relevant group.

In the student grade card example, clustering helps the university quickly identify top
performers, average students, and weak students, making academic analysis faster and
more efficient.

generated by "Markdown to PDF Fast Converter" 👉 [Link]

8. Explain Text Mining in detail. Describe the phases of the
text mining process: Pre-processing, Feature Generation,
Classification/Clustering, and Analysis

A. What is Text Mining? (Detailed Explanation)

Text Mining (or Text Analytics) is the process of extracting useful information, patterns,
knowledge, and relationships from large amounts of unstructured text data, such as emails,
documents, web pages, social media posts, and reports.

✅ Key Points about Text Mining

Works on unstructured and semi-structured data.

Combines:
Natural Language Processing (NLP)
Machine Learning (ML)
Data Mining
Statistics
Helps in:
Document classification
Clustering
Information retrieval
Summarization
Sentiment analysis
Knowledge discovery

Text mining is widely used in search engines, recommendation systems, social media
analysis, business intelligence, and decision support systems.

B. Phases of the Text Mining Process (With Steps)

The Text Mining process consists of five main phases. Below is the detailed explanation of
the phases asked in the question.

1. Pre-processing Phase
generated by "Markdown to PDF Fast Converter" 👉 [Link]
This phase cleans and prepares raw text for further analysis.

✅ Steps in Pre-processing

Noise Removal – Removes HTML tags, special characters, errors.

Tokenization – Splits text into words or tokens.
POS Tagging – Identifies nouns, verbs, adjectives, etc.
Word Sense Disambiguation – Selects the correct meaning of ambiguous words.
Parsing – Builds the grammatical structure of sentences.

✅ Purpose: Convert raw unstructured text into clean and structured text.

2. Feature Generation Phase

This phase converts text into numerical features that machines can process.

✅ Steps in Feature Generation

Bag of Words (BoW) – Represents text as a set of word counts.

Stemming / Lemmatization – Converts words to their root form.
Stop Word Removal – Removes common words like is, the, in.
TF–IDF Weighting
TF (Term Frequency): Frequency of a term in a document.
IDF (Inverse Document Frequency): How unique a term is across all documents.

✅ Purpose: Transform text into meaningful numeric vectors.

3. Feature Selection Phase

This phase selects only the most important features and removes unnecessary ones.

✅ Steps

Dimensionality Reduction using:

PCA (Principal Component Analysis)
LDA (Linear Discriminant Analysis)
Term Evaluation using:
Mutual Information
Chi-Square Test
generated by "Markdown to PDF Fast Converter" 👉 [Link]
✅ Purpose: Reduce complexity and improve accuracy.

4. Classification / Clustering Phase

This phase applies machine learning algorithms to the features.

✅ (a) Classification – Supervised Learning

Used when labeled data is available.

Naïve Bayes
Support Vector Machine (SVM)
Decision Tree

✅ (b) Clustering – Unsupervised Learning

Used when labels are not available.

K-Means clustering

✅ Purpose: Automatically classify or group similar documents.

5. Analysis Phase

This is the final interpretation and visualization phase.

✅ Steps in Analysis

Evaluation of results
Visualization using:
Graphs
Word clouds
Topic maps
Interpretation of discovered knowledge and relationships

✅ Purpose: Convert processed data into useful knowledge for decision making.

✅ Final Exam-Ready Conclusion

generated by "Markdown to PDF Fast Converter" 👉 [Link]

Text Mining is the process of extracting useful knowledge from unstructured text data using
NLP, ML, and data mining techniques.

The main phases are:

1. Pre-processing – Cleaning and preparing text

2. Feature Generation – Converting text into numerical form
3. Feature Selection – Selecting important features
4. Classification / Clustering – Applying ML algorithms
5. Analysis – Evaluating and visualizing results

These phases together transform raw text into meaningful information for fast decision
making.

9. Explain Web Mining Taxonomy: Web Content Mining, Web

Structure Mining, and Web Usage Mining with Applications

Web Mining is the process of extracting useful knowledge from web documents and web
services. It is broadly classified into three types based on the data source:

1. Web Content Mining

2. Web Structure Mining
3. Web Usage Mining

1. Web Content Mining

✅ Meaning:

Web Content Mining is the process of extracting useful information from the contents of web
pages such as:

Text
Images
Audio
Video
Structured records (tables, lists)

It can be done by:

generated by "Markdown to PDF Fast Converter" 👉 [Link]
 Direct mining of web documents
Mining through search engines

✅ Applications of Web Content Mining:

Classifying web documents into categories

Identifying topics of web pages
Finding similar web pages
Search engine relevance improvement
Recommendation systems
Filtering and query-based search

2. Web Structure Mining

✅ Meaning:

Web Structure Mining is the process of discovering structure information from the web using
hyperlinks. It treats the web as a graph:

Web pages → Nodes

Hyperlinks → Edges

It studies:

In-links and out-links

Hubs and authorities
PageRank

✅ Applications of Web Structure Mining:

Ranking of web pages (PageRank)

Finding authoritative web pages
Community detection
Improving search engine results
Identifying important websites

3. Web Usage Mining

✅ Meaning:
generated by "Markdown to PDF Fast Converter" 👉 [Link]
Web Usage Mining is the application of data mining techniques to discover user behavior
patterns from web usage data such as:

Web server logs

Application server logs
Application-level logs

✅ Phases of Web Usage Mining:

1. Pre-processing
2. Pattern Discovery
3. Pattern Analysis

✅ Applications of Web Usage Mining:

Understanding user navigation behavior

Personalized website content
E-commerce product recommendations
Website traffic analysis
Improving website performance
Targeted online advertising

✅ Final Exam-Ready Conclusion

Web Content Mining extracts useful information from the actual contents of web pages.

Web Structure Mining analyzes the link structure of the web using graph theory.

Web Usage Mining discovers user access patterns from web logs.

Together, these three form the Web Mining Taxonomy used for search engines,
recommendation systems, business intelligence, and web analytics.

generated by "Markdown to PDF Fast Converter" 👉 [Link]