BIG DATA ANALYTICS

MODULE - 5

Dr. Prasanna Lakshmi G S,

Dept. of I S E,
SVIT.
Module 5
• Apache Spark is a fast and general-purpose computing engine used for big data
analytics.
• It can process data up to 100 times faster than traditional systems(Mapreduce,
RDBMS) and works with HDFS (Hadoop Distributed File System) and other
compatible data sources.
• Spark uses a simple and expressive programming model (RDD, dataframe, dataset)
that allows programmers to write less code for complex tasks.
• It is useful for analytics, ETL (Extract, Transform, Load), machine learning, stream
processing, and graph computations.
• Spark runs on Windows, Linux, and Mac systems. It needs Java and supports
multiple languages like Python, Scala, and R.
• Spark version 2.3.1 requires:
Java 8+
Python 2.7+/3.4+
R 3.1+
Scala 2.11.x
Spark Architecture 1) Data Storage
• Spark stores data in HDFS or any Hadoop-
compatible data sources like:
HBase, Cassandra, Ceph, or Amazon S3
(Simple Storage Service).
2) Standard API
• APIs are available for building
applications using: Scala, Java, Python, R
• These APIs make it easy for developers to
write programs.
3) Resource Management
• Spark can be run on:
 A standalone server, or
 Cluster managers like YARN or Mesos.
• These manage the allocation of resources
like CPU, memory, etc.
 S3 (Simple Storage
Apache Mesos –
Ceph – Storage system Service) – Cloud
Cluster manager
Storage Service
• A reliable, open • A cluster manager • A cloud storage
source storage that divides and service from Amazon
system to store huge manages system where large data
amounts of data (in resources (like CPU files can be stored
petabytes or more). and RAM) for running and accessed by
• It supports HDFS- big data tasks, spark.
Compatible storage including spark and
and allows efficient hadoop jobs.
access and
distribution.
Main Features of
1. High-Speed Processing
• Spark can handle complex data very fast.
• Its in-memory engine can be up to 100 times faster than Hadoop because it stores

Spark data in RAM instead of reading/writing from disk every time.

2. Uses Both RAM and Disk
Spark stores temporary results in memory (RAM) first. If memory is full, it stores on the
disk — this is smart memory usage.
3. Supports Object Store
You can upload data from storage systems like Amazon S3 and use it directly in Spark. It
treats the Object Store as a source of input.
4. High Performance from Cache
If data is already in memory (cached), Spark runs much faster because it avoids reading
from disk again.
5. RDD Support (Resilient Distributed Datasets)
• RDD is Spark’s core data structure.
• It stores data in chunks across different computers for parallel processing.
• You can think of it like a table split into different parts.
• RDD’s are fault tolerant.
6. Handles All Types of Data
• structured, semi-structured, and unstructured data from various sources.
7. Supports many new functions in addition to map and reduce functions.
8. Spark waits before actually running a task — it delays execution until needed to
optimize performance.
9. Spark has simple and consistent APIs in Scala, Java, and Python, making it easy for
developers to write code. Spark supports Scala, Java, Python, Clojure, and R, so many
developers can use it.
10. Spark provides an interactive tool (shell) for exploring data quickly using Scala or
Python. It's helpful for testing code and learning APIs.
Spark Software stack
Spark Stack Components • It includes Classification, regression,
1. Spark SQL clustering, collaborative filtering,
dimensionality reduction, optimization.
• Runs SQL queries on spark data.
• It is used in recommender systems and
• Supports JDBC, and ODBC.
classification tasks.
• Works with dataframes, hive, cassandra,
5. Spark GraphX
parquet, etc.
• API for graph processing.
• Allows UDF (User defined functions).
• Extends RDD with Resilient Distributed
2. Spark Streaming Property.
• For real-time streaming data.
• Has Operators: subgraph, joinVertices,
• Uses Dstream (series of RDD’s) aggregateMessages.
• Processes data in micro-batches. • Has many graph algorithms.
3. SparkR 6. Spark Arrow
• Ligtweight front-end for spark from R. • For columnar in-memory analytics.
• Uses SparkR API via RDD class. • Supports vectorised UDFs (V-UDFs)
• Run jobs from R shell on cluster. • Used in high-performance Python UDFs for
4. Spark Mllib SerDe (Serialization/Deserialization) pipelines.
• Machine Learning library in spark.
Introduction to data analysis with spark
• “Data Analysis is the process of carefully using statistical and/or logical methods
to describe, summarize, and assess data.”
Steps of Data Analysis: repeated information.
1. Data Storage: • Editing and organizing data for
Collect and store data from analysis.
different sources. Storage can • Validating and transforming
be in systems like HDFS, data.
Cassandra, Hive, or S3. 3. ETL (Extract, Transform, Load):
2. Data Pre-processing: Extract data from various sources,
Clean and prepare the data. This transform it into the required
includes: format, and load it into the system
• Removing incorrect or extreme for analysis.
values.
• Filtering out irrelevant or
4. Applications of Analyzed Data: production of a high-demand model and improving
marketing in certain regions.
Once the data is analyzed using statistical or logical
methods, the results can be used in different ways to These analytics support:
support decision-making. These applications help • Business Processes (BP):
businesses understand the past, predict the future, and The regular operations and activities of a business
decide what actions to take next. (like sales, production, etc.).
The main types of data analysis applications are: • Business Process Automation (BPA):
• Descriptive Analytics – What happened? Using technology to automate routine tasks (like
– This type of analysis looks at past data to explain what auto-generating sales reports).
has already occurred. • Business Intelligence (BI):
– Example: A car company checks last year's sales data to Using data to help make smart business decisions.
see which model sold the most. • Decision Modeling:
• Predictive Analytics – What is likely to happen? Creating models or plans to help managers choose
– This uses past and current data to predict future trends. the best actions.
– Example: Based on current demand and past trends, • Knowledge Discovery:
the company predicts that a certain car model will be in
Finding new patterns, trends, or useful information
high demand next year.
hidden in large sets of data.
• Prescriptive Analytics – What actions should be
taken?
– This suggests possible actions to achieve desired Data Sources → Data Storage → Preprocessing & ETL →
outcomes. Querying & Analysis → Reporting & Applications
– Example: The analysis recommends increasing the
Spark SQL
• Spark SQL is a module (part) of Apache Spark that allows you to work with structured data using SQL
(Structured Query Language).
• It uses special tools like:
1. Catalyst Optimizer – makes SQL queries run faster.
2. Tungsten Engine – improves execution performance.
• Spark SQL will allow to :
 Run SQL queries just like in databases (e.g., SELECT, WHERE, GROUP BY)
 Work with big data stored in files (like JSON, CSV, Parquet) or databases (like Hive, Cassandra, MySQL)
 Use DataFrames, which are like tables in memory
• What Spark SQL Does
 It processes structured data such as tables from databases.
 It provides flexible APIs that allow support for many types of data sources like JSON, Hive, Cassandra,
and Parquet.
 It performs ETL operations, which means it can extract data from different sources, transform it into the
required format, and then load it for further processing and analysis. After this, users can run custom
SQL queries on the data.
Features of Spark SQL
• Spark SQL supports many programming languages including SparkR, PySpark, Python, and
Java. These can be used for writing code to analyze data.
• It allows applications to connect to databases using JDBC and ODBC APIs . This means
external programs like Java-based applications or Microsoft Excel can connect to Spark SQL to
fetch or analyze data.
• Spark SQL can read data from different file formats such as Hive tables, JSON files, and
Parquet files. It can transform this data into a suitable format and then allow users to run
ad hoc queries. (An ad hoc query is a one-time query designed for a specific purpose. For
example, a SQL query to select records based on a certain ID.)
• It includes support for Hive, allowing it to use Hive tables and databases. It also supports
User Defined Functions (UDFs) .
• Spark SQL supports query languages like HiveQL and Cassandra CQL . This means users can
run queries on data stored in Hive and Cassandra using their native query languages.
• It supports Spark Streaming for processing both OLTP and structured streaming data .
OLTP stands for Online Transaction Processing, which is useful for real-time applications.
1. RDD’s, HDFS, and S3
• RDD (Resilient Distributed Dataset): Spark’s internal data structure to store and process
data in memory.
• HDFS : A distributed file system to store big files (used in big data systems).
• Amazon S3: cloud storage by amazon.
2. ETL Process
• Extract: Pull data from source (e.g., Cassandra, JSON, Parquet).
• Transform: Clean or format the data (e.g., change date format, remove duplicates).
• Load: Store the cleaned data back in the required format or structure.
3. JSON, Parquet, Hive:
These are the data formats used to store structured or semistructured data.
• JSON – It’s a plain text format, easy to read. It is good for web data or logs.
• Parquet – a colum-oriented binary format – faster and can be compressed. It is good for
analytics and big data processing. Spark reads it faster than JSON.
• Hive – a dataware house system on top of HDFS. Data is stored in tables, similar to SQL
databases.
4. SparkSQL / HiveQL / CassandraCQL (Query Processing Layer)
This block processes the queries on the data. Assume data as library, and this
• SparkSQL: SQL like queries on Spark dataframes. layer as a librarian, who knows how
• HiveQL : Hive specific SQL commands. to find any book (data) using your
• CassandraCQL: used to query cassandra databases. search query (SQL).
5. SparkR, PySpark, Spark Streaming, OLAP:
These are tools used to access and process data using Spark.
• PySpark: Use Python to write Spark code.
• SparkR: Use R language for Spark.
• Spark Streaming: Process real-time data (like live Twitter feed).
• OLAP (Online Analytical Processing): For deep analytics, such as slicing and dicing data for business
insights.
6. Spark-Cassandra Connector-Java :
• This allows Spark to directly connect and fetch data from CassandraDB, a NoSQL database.
7. CassandraDB:
• A high-performance NoSQL database, suitable for handling huge volumes of fast-changing data.
8. Applications:
• General category – this includes any program or platform that wants to use the results from Spark SQL.
Examples:
• Business dashboards
• Reporting tools
• Machine learning pipelines
9. Applications in Java / Excel :
Applications like:
• Java-based web apps
• Microsoft Excel sheets
• These need a way to connect with Spark SQL to fetch data.
10. JDBC/ODBC driver
• JDBC (Java Database Connectivity) – used by Java applications.
• ODBC (Open Database Connectivity) – used by tools like Excel.
These drivers are like translators that let Excel or Java apps send SQL queries to Spark.
11. JDBC/ODBC Server
• These are the servers handle incoming queries from JDBC/ODBC drivers.
12. Hive Server (Thrift Server)
The hive server (Thrift server) is a bridge between: external applications (Like excel, Java apps) and
SparkSQL or hive for querying big data.
It receives SQL queries from external tools using ODBC or JDBC drivers, sends those queries to
sparkSQL or hive, and then returns the results.
Many external tools (like Excel or BI tools) do not directly understand Spark or Hive. They
communicate using ODBC or JDBC, which are standard database connectors.
So, Hive Server (Thrift) acts like a translator or middleman:
• It accepts requests in JDBC/ODBC format.
• It translates them to Spark SQL/HiveQL.
• It sends the results back to the original application.
Example workflow of the block diagram
Step Action
1 Excel connects to Spark via ODBC.

2 User uses Excel's Query Wizard or Power Query to select a table or filter data.

Excel generates the SQL query in the background, such as:

3 SELECT * FROM sales WHERE month = 'April'

4 The ODBC driver sends this query to the Thrift Server.

5 SparkSQL executes it and sends back the data.

6 Excel displays the result in a worksheet or PivotTable.

Using HiveQL for Spark SQL
Spark SQL lets you run SQL queries on large data using Apache Spark.
It works with data stored in different formats like Hive, JSON, Parquet, etc.
1. SQLContext
• It’s used to run SQL queries.
• It doesn’t need Hive to run.
• Good for simple SQL queries and working with JSON, Parquet, etc.
2. HiveContext
• It’s an extended version of SQLContext.
• It connects with Hive Server.
• Allows you to run HiveQL commands.
• Gives access to Hive features like tables, UDFs, and Hive's metadata.
• Use it when you want to access data in Hive
Use of Aggregation and Statistical Functions
Aggregation Functions in HiveQL Statistical Functions in HiveQL
Function Role of it Example Function Role of it Example

Standard How spread out

count(*) Counts all rows Total number of records stdev()
deviation the data is
Counts non-null values in Count of values in "age" Same as above,
count(expr) a column column Sample standard
sampleStdev() but for sample
deviation
data
sum(col) Adds up values Total sales
Variance of How much data
variance()
Sum of unique sales values varies
sum(DISTINCT col) Adds unique values only
amounts
Like above, but
sampleVariance() Sample variance
on sample
avg(col) Average of a column Average marks

avg(DISTINCT col) Average of unique values Avg of unique scores

min(col) Minimum value Lowest price

max(col) Maximum value Highest rating

Using Python Advanced Features with Spark SQL
Python is a popular programming language known for being:
• Easy to read and write
• High-level (you don’t worry about system-level things)
• Object-Oriented (you can use classes and objects)
• Interactive (can run code line by line)
It supports:
• Basic data types: like numbers, strings, booleans
• Data containers: like lists, dictionaries ({}), sets, and tuples
• Functions and classes: you can write reusable code and create objects
Python comes with many built-in libraries for different tasks:
Purpose Examples
Regular expressions re module
Unit testing unittest
Web work cgi, http, urllib
Image handling PIL, imageio
Email smtplib, email
Threading threading
sqlite3, mysql-
Databases
Python + SparkSQL = PySpark
• PySpark is the python API for apache spark, means spark programs can be written in python.
• PySpark is more powerful because
 Processing of big data can be done using Spark SQL.
 Use pythons simple syntax and powerful libraries.
 Work easily with dataframes, hive tables, JSON, etc.
Python Libraries for Analysis
• NumPy and SciPy are python libraries, where NumPy – Numerical Python, and SciPy –
Scientific Python. They help in dooing math, statistics, and scientific calculations quickly.
• They are free and open-source, and widely used in data analysis and machine learning.
• Other useful python libraries for data analysis are as follows:
About spark and pythons UDF’s
• UDF – User Defined Function, which is written to process
each line of data.
• Problem with old UDF’s :- Spark had to convert data
between Python and JVM (Java Virtual Machine). This
conversion (called SerDe) was slow (Serialization and NumPy Features
Deserialization). Feature Functionality Example

• Spark 2.3, Introduced Vectorized UDFs (VUDFs) using N-dimensional

arrays
Stores data in 1D, 2D, 3D...
formats
array([[1,2],[3,4]])

Apache Arrow. This allows data to move faster between

Math operations Do math on arrays [Link](arr)
Spark and Python — improving performance. Now, Python
UDFs can work with entire columns (vectors) instead of one Linear Algebra
Matrix multiplication,
inversion
[Link](A, B)

row at a time.
Random Numbers Create random data [Link](3,2)

NumPy Fourier
Transforms
Signal processing [Link]()

NumPy is like a super-fast calculator for handling arrays (tables Array indexing Select part of the array arr[0:2]
of numbers). Apply operations across
Broadcasting arr + 10
shapes

Integration with
Use C for speed boost [Link]
C/C++
• NumPy can do many array operations:
 Creating arrays: [Link]([1,2,3])
 Sorting: [Link](arr)
 Reshaping: [Link](2,3)
 Math functions: [Link](x), [Link](x), [Link](x)
 Statistics: [Link](arr), [Link](arr)
• NumPy allows:
 Import/export of data (save/load)
 Interfacing with databases
 Reading and writing from files
SciPy
• SciPy is a Python library Feature Explanation
built on top of NumPy. Built on NumPy Uses NumPy's arrays and functions as a base
• It adds more advanced Reads MATLAB files Can open .mat files used in MATLAB
scientific functions for Includes tools for math operations like integration and
Special functions
optimization
working with numbers,
Can calculate distances between points (e.g., for
equations, and data. Distance functions
clustering or geometry)
• SciPy = NumPy + Extra Mesh grids Helps create 2D/3D grids for simulations or plotting
scientific tools
Shape tools Resize and reshape arrays
Polynomial functions Work with polynomials (like equations in x)
Real and imaginary
Can handle complex numbers like 3 + 4j
support
Data type casting Change data types (e.g., int → float) easily
Matrix support Create and work with matrices (used in math, ML, etc.)
Can be used along with Spark for advanced matrix
Works with Spark
operations
Pandas
• Pandas is a Python library used for data analysis and manipulation.
• The name “Pandas” come from:
a) Pan = Panel data (used in economics)
b) da = Data
• Python has 3 main data structures:

Structure Explanation Example

Series 1D labeled data (like a column) [Link]([1, 2, 3])
DataFrame 2D table (like Excel) [Link]({'A': [1, 2], 'B': [3, 4]})
Panel (older) 3D container of DataFrames Used for panel data, but now less common

• A Panel is like a collection of DataFrames.

• About Axis in panel :
Axis Meaning
Items (axis 0) The DataFrame labels (like 'Sheet1', 'Sheet2')
Major Axis (axis 1) Rows in each DataFrame
Minor Axis (axis 2) Columns in each DataFrame
Other features of pandas are:
1. SQL-like Operations
You can use:
• SELECT: df[['column1']]
• WHERE: df[df['age'] > 30]
• GROUPBY: [Link]('region').sum()
• JOIN: [Link](df1, df2, on='id')
2. GroupBy (Split-Apply-Combine)
• Break data into groups → process each group → combine result.
[Link]('category').sum()
3. Size Mutability
• You can add or delete columns in DataFrames anytime.
df['new_col'] = df['old_col'] * 2 # Add column
del df['old_col'] # Delete column
4. Merging and Concatenation
• You can merge DataFrames like SQL joins, or combine them vertically/horizontally.
[Link]([df1, df2]) # Stack vertically
5. Slicing and Dicing
You can select rows, columns, or blocks of data:
[Link]['2023'] # Row
df['sales'] # Column
[Link][0:2, 0:2] # Specific block
6. Pandas package includes Rpy interface for R functions plus additional functions.
7. Panda ecosystem has statics, machine learning, integrated development environment
(IDE), API and several out of core features.
User Defined Functions
• A UDF is a custom function you write yourself to perform a specific task on your data.
• It helps when the built-in Spark functions are not enough.
• UDFs can be used inside SQL queries or DataFrame operations.
Disadvantages of older UDF
• Standard UDFs take one row at a time → This is slow.
• Every time Python and Spark exchange data, there is an overhead due to:
- SerDe = Serialization and Deserialization (converting data between Python and
Spark JVM).
• Earlier: Most UDFs were written in Java/Scala, and Python had to call them, which
added complexity.
Solution for above Disadvantage
• PySpark allows you to write UDFs directly in [Link] can register them and call them
inside SQL or apply them on DataFrames.
Why use UDF’s
• Reusable logic
• Shorter code
• Useful when built-in spark SQL
functions cant solve your problem.
• You can apply them to different
products or datasets (eg., cars,
toys, electronics)
Vectorized UDF (VUDF)
• Vectorized UDFs are faster and smarter
versions of UDFs.
• They use Pandas Series instead of single
rows — meaning batch/column-wise
operations.
• They avoid unnecessary overhead by using
Apache Arrow (a high-performance library
for memory sharing between Python and
JVM).
Benefits of VUDF:
• Faster than regular UDF
• Works like Pandas (easy to write and read)
• Uses columnar memory, which is faster for Big Data

What is Apache Arrow?

• It’s a tool that helps Spark and Python exchange data quickly.
• Used in VUDFs to reduce conversion time (serialization/deserialization).
Grouped Vectorized UDFs (GVUDFs)
• GVUDFs work on groups of data — for example, "group by showroom ID, then calculate
total yearly sales."
• Uses Pandas' split-apply-combine logic.

Split-Apply-Combine Steps:
• Split the DataFrame into groups (e.g., group by showroom ID)
• Apply a custom function on each group
• Combine the results back into one DataFrame
Comparison of VUDF (Vectorized UDF) and GVUDF (Grouped
Vectorized UDF)
Term Description
A function that processes one column (Series) of a
VUDF (Vectorized UDF)
DataFrame at a time.
A function that processes groups of rows
GVUDF (Grouped Vectorized UDF)
(DataFrames) together, based on a groupBy.

Type Input
VUDF [Link] (one column)
GVUDF [Link] (entire group of rows)

Use When You Need Use

To apply a custom function to a single column (like
VUDF
salary → bonus)
To apply a function on each group (like per showroom,
GVUDF
per year, etc.)
UDF Type Function Decorator
VUDF @pandas_udf(return_type)
@pandas_udf(schema,
GVUDF
functionType="grouped_map")

UDF Type Output

VUDF [Link] (column)
[Link] (multiple columns and
GVUDF
rows)

UDF Type Example

Calculate 10% bonus of each employee’s
VUDF
salary
Compute total sales per showroom using
GVUDF
groupBy on showroom_id