Data Engineering

Programs Offered

ne
i
Post Graduate Programmes (PG)

l
• Master of Business Administration
• Master of Computer Applications

n
• Master of Commerce (Financial Management / Financial
Technology)

O
• Master of Arts (Journalism and Mass Communication)
•
•
Master of Arts (Economics)
Master of Arts (Public Policy and Governance) Data Engineering
•
•
•
•
Master of Social Work
Master of Arts (English)
Master of Science (Information Technology) (ODL)
Master of Science (Environmental Science) (ODL)

i t y
Diploma Programmes
• Post Graduate Diploma (Management)

r s
e
• Post Graduate Diploma (Logistics)
• Post Graduate Diploma (Machine Learning and Artificial

•
Intelligence)
Post Graduate Diploma (Data Science)

i v
Undergraduate Programmes (UG)
•
•
•
•
Bachelor of Business Administration
Bachelor of Computer Applications
Bachelor of Commerce
Bachelor of Arts (Journalism and Mass Communication)
Un
•

•
•
English / Sociology)
Bachelor of Social Work y
Bachelor of Arts (General / Political Science / Economics /

it
Bachelor of Science (Information Technology) (ODL)

A m
c ) DIRECTORATE OF Product code

(
DISTANCE & ONLINE EDUCATION
Amity Helpline: 1800-102-3434 (Toll-free), 0120-4614200
AMITY

For Distance Learning Programmes: dladmissions@[Link] | [Link]/addoe

DIRECTORATE OF
For Online Learning programmes: elearning@[Link] | [Link] DISTANCE & ONLINE EDUCATION
(c
)A
m
ity
U
ni
ver
Data Engineering

si
ty
O
nl
in
e
 e
in
© Amity University Press

All Rights Reserved

nl
No parts of this publication may be reproduced, stored in a retrieval system or transmitted
in any form or by any means, electronic, mechanical, photocopying, recording or otherwise
without the prior permission of the publisher.

O
SLM & Learning Resources Committee

ty
Chairman : Prof. Abhinash Kumar

Members :

si
Dr. Divya Bansal
Dr. Coral J Barboza
Dr. Monica Rose

er
Dr. Winnie Sharma

Member Secretary : Ms. Rita Naskar

v
ni
U
ity
m
)A
(c

Published by Amity University Press for exclusive use of Amity Directorate of Distance and Online Education,
Amity University, Noida-201313
 Contents

e
Page No.

in
Module - I Data Engineering Basics for Everyone 01
1.1 The Modern Data Ecosystem
1.1.1 Overview of the Modern Data Ecosystem

nl
1.1.2 Key Players in the Data Ecosystem
1.1.3 Introduction to Kubernetes Objects
1.2 Data Analysis and Data Analyst Role

O
1.2.1 Defining Data Analysis
1.2.2 Data Analytics vs. Data Analysis
1.2.3 Responsibilities of a Data Analyst

ty
1.2.4 A Day in the Life of a Data Analyst
1.3 Data Sources and Types
1.3.1 Understanding Different Types of Data

si
1.3.2 Sources of Data Using Service Bindings
1.4 Data Storage and Processing er
1.4.1 Overview of RDBMS and NoSQL Databases
1.4.2 Introduction to Data Marts and Data Lakes
1.4.3 ETL and Data Pipelines
v
1.5 Big Data and Processing Tools
1.5.1 Foundations of Big Data
ni

1.5.2 Overview of Big Data Processing Tools

1.6 Gathering and Importing Data
U

1.6.1 Identifying Data for Analysis

1.6.2 Data Sources and Collection Methods
1.7 Data Wrangling and Cleaning
1.7.1 Introduction to Data Wrangling
ity

1.7.2 Tools for Data Wrangling

1.7.3 Data Cleaning Techniques
1.8 Statistical Analysis and Data Mining
m

1.8.1 Overview of Statistical Analysis

1.8.2 Introduction to Data Mining
1.9 Communicating and Sharing Data Analysis Findings
)A

1.9.1 Introduction to Data Visualisation

1.9.2 Communicating and Sharing Data Analysis Results
1.10 Communicating and Sharing Data Analysis Results
1.10.1 Overview of Career Paths in Data Analysis
(c

1.10.2 The Many Paths to Data Analysis

 Module -II Relational Database Basics 57
2.1 Relational Database Concepts

e
2.1.1 Introduction to Relational Databases

in
2.1.2 Understanding Tables, Rows and Columns
2.1.3 Primary Keys, Foreign Keys and Relationships
2.2 Using Relational Databases

nl
2.2.1 Data Manipulation with SQL
2.2.2 Querying Data with SELECT, INSERT, UPDATE and DELETE
2.2.3 Filtering and Sorting Data

O
2.3 MySQL and PostgreSQL
2.3.1 Overview of MySQL and PostgreSQL
2.3.2 Installation and Setup

ty
2.3.3 Working with Databases in MySQL and PostgreSQL
2.4 Database Design Project
2.4.1 Designing a Relational Database Schema

si
2.4.2 Entity-Relationship (ER) Diagrams
2.4.3 Creating and Implementing the Database
er
Module - III: SQL for Data science 100
3.1 Introduction to Databases and SQL
3.1.1 Overview of Databases and Database Management Systems (DBMS)
v
3.1.2 Introduction to SQL and Its Importance in Data Manipulation
3.1.3 Understanding the SELECT Statement
ni

3.1.4 Using COUNT, DISTINCT and LIMIT

3.2 Data Manipulation with SQL
U

3.2.1 INSERT, UPDATE and DELETE Statements

3.2.2 Working with String Patterns and Ranges in SQL
3.2.3 Sorting Result Sets with ORDER BY
ity

3.2.4 Grouping Result Sets with GROUP BY

3.3 Relational Database Concepts and Database Instance on Cloud
3.3.1 Understanding Relational Database Concepts
3.3.2 Creating a Database Instance on Cloud (e.g., IBM Db2)
m

3.3.3 Getting Started with the Db2 Console

3.3.4 Types of SQL Statements: DDL vs. DML
3.4 Table Operations in SQL
)A

3.4.1 CREATE TABLE Statement

3.4.2 ALTER, DROP, and TRUNCATE Tables
3.5 Built-in Database Functions and Date/Time Functions
3.5.1 Using Built-in Database Functions (e.g., SUM, AVG, MAX and MIN)
(c

3.5.2 Date and Time Built-in Functions in SQL

3.6 Accessing Databases Using Python and DB-API
 3.6.1 Introduction to Python’s DB-API
3.6.2 Connecting to a Database Using ibm_DB-API

e
3.6.3 Creating Tables and Loading Data in Python

in
3.6.4 Querying Data from Databases with Python

Module - IV: Big Data, Hadoop, and Spark Basics 162

4.1 Big Data and Introduction to Hadoop Ecosystem

nl
4.1.1 Impact of Big Data and Scaling Challenges
4.1.2 Tools of Big Data and Beyond the Hype

O
4.1.3 Introduction to the Hadoop Ecosystem
4.1.4 Understanding Hadoop and MapReduce
4.2 Components of the Hadoop Ecosystem
4.2.1 Introducing HDFS, Hive, HBase and Spark

ty
4.2.2 Other Modules in the Hadoop Ecosystem
4.2.3 Working with HDFS and HBase

si
4.3 Introduction to Apache Spark
4.3.1 Why Use Apache Spark
4.3.2 Functional Programming Basics
er
4.3.3 Parallel Programming with Resilient Distributed DataSets (RDDs)
4.3.4 Scale-out and Data Parallelism in Apache Spark
4.4 Spark DataFrames and SparkSQL
v
4.4.1 Introduction to DataFrames and DataSets
4.4.2 Catalyst and Tungsten Optimisations
ni

4.4.3 ETL with DataFrames

4.4.4 Real-World Usage of SparkSQL
U

4.5 Apache Spark Development and Runtime Environment

4.5.1 Apache Spark Architecture Overview
4.5.2 Apache Spark Cluster Modes
4.5.3 Applications of Running Apache Spark
ity

4.5.4 Using Apache Spark on IBM Cloud

4.6 Monitoring and Tuning Apache Spark
4.6.1 The Apache Spark User Interface
m

4.6.2 Monitoring Jobs and Debugging Parallel Jobs

4.6.3 Managing Memory and Processor Resources in Spark
4.6.4 Setting Apache Spark Configuration and Running on Kubernetes
)A

Module - V: Apache Spark for Data Engineering and Machine Learning 222
5.1 Spark for Data Engineering
5.1.1 Introduction to Apache Spark
5.1.2 Working with Spark DataFrames and DataSets
(c

5.1.3 SparkSQL for Data Analysis and Manipulation

5.1.4 ETL with Spark and Data Wrangling
 5.2 Spark ML for Machine Learning
5.2.1 Introduction to Spark MLlib

e
5.2.2 Machine Learning with Spark

in
5.2.3 Building and Training ML Models with Spark
5.2.4 Evaluating Model Performance in Spark
5.3 Project

nl
5.3.1 Implementation of Data Engineering and Machine Learning Project Using Spark
5.3.2 Applying Spark for Data Processing and Analysis
5.3.3 Utilising Spark ML for Building Predictive Models

O
ty
si
v er
ni
U
ity
m
)A
(c
 Data Engineering 1

Module - I: Data Engineering Basics for Everyone

Notes

e
Learning Objectives

in
At the end of this module, you will be able to:
●● Discuss overview of the modern data ecosystem

nl
●● Identify key players in the data ecosystem
●● Recognise introduction to Kubernetes objects
●● Analyse defining data analysis

O
●● Identify data analytics vs. data analysis
●● Recognise responsibilities of a data analyst
●● Identify a day in the life of a data analyst

ty
●● Analyse understanding different types of data
●● Analyse sources of data using service bindings
●● Learn overview of RDBMS and NoSQL databases

si
●● Analyse introduction to data marts and data lakes
●● Recognise ETL and data pipelines er
●● Analyse foundations of big data and big data processing tools
●● Analyse identifying data for analysis
●● Recognise data sources and collection methods
v
●● Analyse introduction and tools of data wrangling
●● Identify data cleaning techniques
ni

●● Analyse overview of statistical analysis

●● Learn introduction to data mining and data visualisation
U

●● Identify overview of career paths in data analysis

●● Identify the many paths to data analysis

Introduction
ity

A data ecosystem is a group of business infrastructure and apps that are used to
collect and analyse data. It helps businesses learn more about their customers and come
up with better strategies for marketing, selling and running their business.
m

1.1 The Modern Data Ecosystem

A modern big data environment, which is sometimes called a “technology stack,” is
made up of three main parts: a responsive data design, scalable delivery and AI-driven
)A

intelligent data management. Data Ecosystem is the name for the computer languages,
packages, methods, cloud-computing services and general technology that an
organisation uses to collect, store, analyse and use data. No two groups use the same
information in the same way. Due to this, each company has its own data environment.
In some situations, these ecosystems may cross, such as when data is pulled or scraped
(c

from a public source or when third-party providers (like cloud storage providers) are
used.

Amity Directorate of Distance & Online Education

 2 Data Engineering

1.1.1 Overview of the Modern Data Ecosystem

Notes

e
The idea of the data ecosystem can be understood by looking at the key steps in the
life cycle of a data project: sensing, collecting, organising, analysing and storing.

in
Components of a Data Ecosystem
1. Sensing

nl
Sensing is the process of finding sources of information for your job. It means
figuring out how good the data is so you can figure out if it’s useful. This review involves
asking questions like:

O
™™ Are the numbers correct?”
™™ Is the information current and fresh?
™™ Is all the information there?

ty
™™ Are the facts correct? Can you believe it?
Data can come from internal sources like databases, files, customer relationship
management (CRM) systems and other tools. It can also come from outside sources, like
websites or companies that gather info from other sources.

si
Key pieces of the data ecosystem leveraged in this stage include:
™™ Internal data sources: databases, files and other tools that are owned by your
company.
er
™™ External data sources: Databases, files, websites and other data sources that
come from outside your organisation
v
™™ Software: Made-to-order software that only exists for data sensing
™™ Algorithms are a set of steps or rules that make it easy to check data for
ni

correctness and completeness before using it.

2. Collection
Once a possible source of data has been found, data must be gathered. Data
U

gathering can be done by hand or with the help of machines. Still, it is usually not
possible to collect large amounts of data by hand. Because of this, data scientists use
computer languages to make software that automates the process of gathering data.
ity

For example, you can write code to “scrape” useful information from a website.
This kind of code is called a “web scraper.” You can also create and code an API, which
stands for “application programming interface,” to get specific information directly from a
database or connect with a web application.
m

Key pieces of the data ecosystem leveraged in this stage include:

™™ Various programming languages: These include R, Python, SQL and JavaScript
™™ Code packages and libraries: Existing code that’s been written and tested and
)A

allows data scientists to generate programs more quickly and efficiently

™™ APIs: Software programs designed to interact with other applications and
extract data

3. Wrangling
(c

Data wrangling is a set of processes meant to turn raw data into a more usable
shape. Depending on the quality of the data, this could mean combining various datasets,

Amity Directorate of Distance & Online Education

 Data Engineering 3

finding and filling data gaps, removing data that isn’t needed or is wrong, “cleaning” and
Notes

e
organising data for future analysis.
Data wrangling, like data gathering, can be done directly or automatically. Manual

in
methods can work well if a sample is small enough. Most big data projects need
automation because the amount of data is too big and there is too much of it.
Key pieces of the data ecosystem leveraged in this stage include:

nl
™™ Algorithms: Algorithms are a set of steps or rules that must be followed in order
to solve a problem. In this case, the problem is figuring out how to evaluate and
change data.

O
™™ Various programming languages: These include R, Python, SQL and
JavaScript. You can write algorithms in these languages.
™™ Data wrangling tools: For different parts of the data wrangling process, you

ty
can buy or find free data wrangling tools. There are many examples, such as
OpenRefine, DataWrangler and CSVKit.

4. Analysis

si
After raw data has been looked over and made easier to use, it can be analysed.
This research can be diagnostic, descriptive, predictive, or prescriptive, depending on the
problem you want to solve with your data project. Even though each type of research
er
is different, they all use the same steps and tools. Usually, you start your research with
some kind of automation, especially when your collection is very big. After the automatic
processes are done, data scientists use their skills to get more information.
v
Important parts of the data ecosystem that are used at this stage are:
™™ Algorithms: These are a set of steps or rules that must be followed to answer a
ni

problem. In this case, the problem is figuring out how to analyse different pieces
of data.
™™ Statistical models: These are math models that are used to study and
U

understand data.
™™ Data visualization tools: These include programmes like Tableau, Microsoft BI
and Google Charts, which can turn data into graphs. You might also be able to
ity

use other features of data visualisation tools.

5. Storage
During every stage of the data life cycle, data must be saved in a way that is both
safe and easy to reach. Your organization’s data control rules will tell you what kind of
m

storage means to use.

Important parts of the data ecosystem that are used at this stage are:
)A

™™ Cloud-based storage solutions: These let a company keep data off-site and
access it from anywhere in the world.
™™ On-site servers: These give businesses a better idea of how their data is kept
and used.
™™ Other storing media: These include hard drives, USB drives, CD-ROMs and
(c

floppy discs.

Amity Directorate of Distance & Online Education

 4 Data Engineering

1.1.2 Key Players in the Data Ecosystem

Notes

e
1. Data Engineer
A person who is responsible for building and managing the data architecture is called

in
a data engineer. He or she is responsible for making the data accessible for use in the
operations and analysis of the company. An online transaction process, also known
as the real database, is referred to as OLTP. A data engineer is required to distinguish

nl
between OLTP and the maintenance of a data warehouse or OLAP, which is essentially
a duplicate of the database used for analysis. Any queries that need to be executed have
to be implemented on an OLAP system rather than an OLTP system. It was necessary

O
to exert the effort required to acquire the data from a variety of sources. After this, they
will be tasked with organising and reorganising the data. It is necessary to construct data
repositories in order to store the data.

ty
Skills:
™™ Solid understanding of the design and architecture of computer systems.
™™ Extensive familiarity with both relational database management systems and

si
non-relational database management systems.
™™ Capability in computer programming.

2. Data Analyst
er
It is not possible to simply gain insights by looking into a massive dataset. When
data is represented visually, rather than numerically, it is much easier to understand. The
primary responsibility of a data analyst is to translate complex numerical information into
v
understandable language and to create meaningful visualisations. They clean the data
so that insights may be derived from it, uncover correlations, discover trends and apply
ni

statistical models. They are the ones who provide the responses to the queries on the
organization’s historical performance. They illustrate, in the most basic terms, how the
company has performed in the past.
U

Skills:
™™ Proficient in the use of spreadsheets as well as well-known business
intelligence applications like tableau and power bi.
ity

™™ An analytical frame of mind, since their observations serve as the basis for
judgements to be made in the future.

3. Data Scientist
m

For the purpose of producing meaningful insights, data scientists oversee and
synthesise the work produced by data engineers and data analysts. They do this by
employing machine learning or deep learning in order to generate accurate predictive
models. They are the ones who are responsible for answering the questions that will
)A

determine the development of the company in the years to come. A professional data
scientist should always be ready with a valuable proposal, drawn from the analysis of
previous data, to help shape the more successful expansion of a company in the future.

Skills:
(c

™™ A solid foundation in areas like as mathematics, statistics, programming and

both machine learning and deep learning.

Amity Directorate of Distance & Online Education

 Data Engineering 5

™™ In addition to this, they are required to have a solid understanding of the

Notes

e
relevant topic.

4. Business Intelligence Analyst

in
BI Analysts acknowledge the work done by data scientists and data analysts in
order to examine the many alterations that may be made to their company and to take
appropriate steps as a result of their findings. Their primary concentration is on the

nl
market forces and other outside influences that have an effect on their company. Offer a
variety of options for their company by monitoring data on the many distinct operations of
their organisation.

O
Skills:
™™ Capability in business management
™™ Knowledge of both the technological industry and of business finances and

ty
sales is required.
™™ It is important to put the appropriate steps into action in light of the insights and
suggestions.

si
1.1.3 Introduction to Kubernetes Objects
An open-source container management platform called Kubernetes manages the
er
release, scaling, descaling and load sharing of containers. It is also known as a tool for
container orchestration. It was created by Google and given to the CNCF (Cloud Native
Computing Foundation) and is developed in the Golang programming language. A single
functional unit that is simple to deploy and maintain may be created using Kubernetes by
v
combining ‘n’ containers. It functions flawlessly with all private, public and on-premises
cloud service providers.
ni

Kubernetes is an open-source tool for managing a cluster of Docker containers.

Along with automating the release and growth of containers, it also heals by automatically
restarting failed containers and rescheduling them when their hosts die. This feature
U

makes the programme more available.

Features of Kubernetes:
1. Automated Scheduling – Kubernetes has a powerful scheduler that makes it easy to
ity

start containers on cluster machines. It makes the best use of resources.

2. Self-Healing Capabilities: It can reschedule, replace and start up the dead devices.
3. Rollouts and rollbacks can be done automatically. It allows rollouts and rollbacks to
get the containerized programme to the state you want.
m

4. Horizontal Scaling and Load Balancing: Kubernetes can make a service bigger or
smaller depending on what is needed.
5. Resource Utilisation: Kubernetes monitors and optimises how resources are used,
)A

making sure containers use their resources well.

6. Support for multiple clouds and mixed clouds: Kubernetes can be set up on different
cloud systems and run containerized apps across multiple clouds.
7. Extensibility– It’s easy to add more to Kubernetes. You can add custom tools and
(c

controls to it.
8. Community Support- Kubernetes has a big and busy community that adds new
features and fixes bugs often.
Amity Directorate of Distance & Online Education
 6 Data Engineering

What are Kubernetes Objects?

Notes

e
Kubernetes objects are parts of a Kubernetes cluster that keep track of what is
supposed to happen. Administrators make objects that describe how they want the

in
cluster to be and Kubernetes uses these objects to keep the cluster in the desired
state automatically. Kubernetes objects can also be seen as class instances, which is
another way to understand them. Each object that is made points to a class that has
already been made. This class tells the API server how to handle system resources and

nl
talk to specific parts.

1.2 Data Analysis and Data Analyst Role

O
Introduction

1.2.1 Defining Data Analysis

ty
Data analysis is the science of looking at data to draw conclusions that can be used
to make choices or learn more about different topics. It means putting data through
processes. This process is used to come to specific conclusions that will help us reach

si
our goals, such as operations that can’t be planned ahead of time because they might
show specific problems. er
Types of Data Analysis
There are different kinds of data research and each has its own goal and way of
doing things. Let’s talk about some of the most important kinds:
v
Descriptive Analysis
ni

With descriptive analysis, the important parts of a record are summed up and
described. To explain the data, you have to figure out measures of central tendency
and spread. The detailed analysis gives a full picture of the data and shows how it is put
together and what its qualities are.
U

Inferential Analysis
Statistical models and tests are used in inferential analysis to draw conclusions
ity

about group parameters like the mean or proportion. Models and hypothesis testing
are used in this unit of research to make predictions and come to conclusions about the
population.

Predictive Analysis
m

With predictive analysis, you can figure out what will happen or what will happen
based on past data and other important information. Statistical models and machine
learning algorithms are used to find trends in the data and make guesses about what will
)A

happen in the future.

Prescriptive Analysis
Proactive analysis is a way to decide what to do about a problem or situation. It uses
mathematical modelling, optimisation tools and other data-driven methods to figure out
(c

what to do. It uses math models, facts and business rules to figure out what the best
move or choice is.

Amity Directorate of Distance & Online Education

 Data Engineering 7

Text Analysis
Notes

e
Text analysis is the process of getting useful information from text data that is
not organised in a certain way. It uses different methods, such as natural language

in
processing (NLP), text mining, sentiment analysis and subject modelling, to find trends
and insights in text data.

1.2.2 Data Analytics vs. Data Analysis

nl
Data Analytics
Analytics is a way to turn raw facts and numbers into specific actions. This is

O
done by evaluating and interpreting the raw data in the context of fixing organisational
problems and making decisions. Analytics is the process of finding and talking about
important trends in data. Analytics is especially useful in areas with a lot of recorded
information. To measure success, analytics uses statistics, computer code and operation

ty
lookup at the same time. Data visualisation is often used to talk about insights in
analytics. The goal of Data Analytics is to get information that can be used to make better
business decisions and get better business results.

si
Data Analysis
It is the process of looking at, changing, cleaning and modelling raw facts and
er
numbers in order to get useful information and come to good decisions.

[Link]. Data Analytics Data Analysis

1. It is characterised as a conventional or It is characterised as a specialised type of
v
generic type of analytics. analytics.
2. It consists of several phases, including In order to process data, raw data must first be
ni

the acquisition of data and the inspection defined in a meaningful way, followed by data
of business data. cleansing and conversion to obtain meaningful
information from raw data.
3. It facilitates decision making through the analyses the data with an emphasis on business
U

analysis of enterprise data. data insights.

4. It employs diverse data processing tools, It analyses data using various tools, including
including Tableau, Python and Excel, Rapid Miner, Open Refine, Node XL, KNIME, etc.
among others.
ity

5. On this, descriptive analysis cannot be On this, a descriptive analysis can be conducted.

performed.
6. It is not concerned with inferential It facilitates inference analysis.
analysis.
m

1.2.3 Responsibilities of a Data Analyst

)A

Data Analyst Roles and Responsibilities

Data organisation for sales numbers, market research, logistics, languages and
other behaviour is the job of a data analyst. To guarantee the accuracy and integrity of
the data, they make use of their technological skills. Data is then studied, created and
displayed to help people, companies and organisations make better decisions.
(c

●● Using automated techniques for primary and secondary sources’ data extraction
●● Removing damaged data, fixing code flaws and dealing with related problems

Amity Directorate of Distance & Online Education

 8 Data Engineering

●● Building and managing databases and information systems and reformatting data.
Notes

e
●● Executing analysis to assess the accuracy and importance of data
●● Analyse reports and performance indicators to filter data in order to find and fix code

in
bugs.
●● It may be helpful for diagnosis and prognosis to use statistical methods to find,
examine and understand patterns and trends in complicated data sets.

nl
●● Assigning numerical numbers to crucial business processes in order to assess and
contrast the performance of businesses over time.
●● Examining local, regional, national and global trends that affect the business and the

O
sector.
●● Creating management reports that use relevant data to highlight trends, patterns
and forecasts
●● Developing data governance methods while collaborating with programmers,

ty
engineers and management executives to find ways to improve processes.
●● Creating final analysis reports for stakeholders to help them understand the data-
analysis procedures and make important decisions based on numerous data points

si
and trends.
EDA, commonly referred to as the Exploratory Data Analysis Project, is a crucial
component of the job description for a data analyst. Such assignments for data analysts
er
need the analyst to look at the data to find and detect trends. The next stage for data
analysts is to use data modelling approaches to summarise the main aspects of data
analysis.
v
Data Analyst Skills Required
ni

You should possess the following eight talents as a data analyst:

●● Data cleansing and preparation
●● Data analysis and investigation
U

●● Knowledge of statistics
●● Data analysis and visualisation
●● Dashboards and reports
ity

●● Expression and writing

●● Expertise within the field
●● Solution-oriented
A background in mathematics, statistics, computer science, information
m

management, or economics can provide a solid foundation for a data analyst’s career.

1.2.4 A Day in the Life of a Data Analyst

)A

The primary responsibility of a data analyst is to extract actionable insights from

data and communicate them to key business decision-makers in order to assist them in
achieving their objectives. The data procedure can be divided into four straightforward
steps:
1. First, a data analyst requires a query, which is typically driven by business requirements.
(c

For instance, what characteristics distinguish consumers who subscribe to a service

from those who do not?

Amity Directorate of Distance & Online Education

 Data Engineering 9

2. Next, a data analyst is tasked with locating the necessary data to answer the query
Notes

e
at hand. This data may exist in multiple locations or be carefully organised in a single
database.

in
3. The data analyst will then use tools and techniques such as SQL, Tableau and Python
to identify patterns and derive conclusions from the data.
4. Finally, these insights should be communicated to stakeholders so that business-

nl
improving actions can be implemented!

A Typical Day in the Life of a Data Analyst

It is common for a data analyst to be working on multiple data analytics initiatives

O
simultaneously. In this capacity, you will perform the following duties for each project:
●● Meetings with stakeholders, data engineers and other team members to gain a
deeper understanding of the research query.

ty
●● Identifying, navigating and analysing a database
●● Working on a dashboard and coding with Python to illustrate data patterns
discovered

si
●● Constructing a Tableau dashboard to visually communicate your findings
●● Presenting your findings and suggestions to relevant parties
●● Data analysts perform tasks such as:
●●
er
Assist large retailers in analysing their supply chain and determining methods to
increase revenue
●● Assist non-profits in maximising the results of their donation campaigns
v
●● Assist healthcare providers in enhancing patient outcomes and reducing care costs
Data analysts accomplish this by collecting hints (also known as data) to discover
ni

and interpret patterns and trends within a business. Using data modelling and
visualisation tools, analysts then present their findings to business executives and
stakeholders. Through the dissemination of these insights, analysts assist businesses in
U

informing decision-making processes, optimising business procedures and anticipating

future trends.
A data analyst’s typical day fluctuates depending on the industry they work in and
the type of data being collected. Regardless of industry or data, a data analyst can
ity

anticipate to spend the majority of their workday conducting a series of tasks in order to
successfully reach their conclusion. These duties consist of:
●● Recognising or diagnosing an issue to be examined
●● Collecting pertinent information from multiple sources
m

●● Clean the data and prepare it for analysis

●● Examining the data in search of insights, patterns, or trends
)A

●● Interpreting the findings of the analysis through the use of reports and displays
●● Collaborating with relevant parties and presenting crucial findings

1.3 Data Sources and Types

(c

Introduction
Two categories of data sources can be distinguished: statistical and non-statistical.
Statistical sources refer to data collected for official purposes, including censuses and
Amity Directorate of Distance & Online Education
 10 Data Engineering

surveys administered by the government. The acquisition of data for other administrative
Notes

e
purposes or for the private sector is referred to as non-statistical sources.

What are the different sources of data?

in
The following are the two data sources:
1) Internal resources

nl
Internal sources are those that use information gathered from the organization’s own
reports and papers. An organisation may, for instance, release an annual report that
includes information on its profit and loss, overall sales, loans and salaries.

O
2) External factors
External sources are sources of information that are used to collect data. As an
illustration, the Karnataka Transport Corporation would be regarded as an external
data source if a tour operator obtained information on tourism in Karnataka from them.

ty
Types of Data
A) Primary data

si
™™ Primary data are facts that an investigator personally gathers.
™™ It is being collected for the first time.
™™ It is distinctive and more reliable.
er
™™ An example of primary data is the population census that the Indian
government conducts every 10 years.
B) Secondary data
v
™™ Information from a secondary source is referred to as secondary data.
ni

™™ Rather than being an original compilation, it is taken from previously published

or unpublished sources.
™™ Examples of secondary data include an individual’s address found in a phone
book or a company’s phone number found on Just Dial.
U

1.3.1 Understanding Different Types of Data

“Data is the new crude oil.” Today, data permeates every industry. Whether you
ity

are a data scientist, marketer, businessperson, data analyst, researcher, or a member

of any other profession, you must experiment with unstructured or structured data. This
information is so vital to our operations that it must be handled and stored without error.
When interacting with these data, it is essential to understand the data types in order to
obtain the desired results. There are two forms of data: qualitative and quantitative, which
m

are further divided into the following categories: The data is categorised into four groups:
™™ Nominal data.
™™ Ordinal data.
)A

™™ Discrete data.
™™ Continuous data.
Therefore, there are four different types of data: nominal, ordinal, discrete and
continuous. The majority of businesses use data insights to create and implement
(c

campaigns, design strategies, introduce new products and services and conduct
experiments. Today, at least 2.5 quintillion bytes of data are produced every day,
according to a report.
Amity Directorate of Distance & Online Education
 Data Engineering 11

Types of Data
Notes

e
Qualitative or Categorical Data
Qualitative or Categorical Data is information that cannot be measured or

in
counted numerically. These data categories are organised by category, not by number.
Consequently, it is also referred to as Categorical Data. These data may include audio,
visuals, symbols, or text. Gender, whether male, female, or other, is qualitative data.

nl
Qualitative data reveals people’s perceptions. This information assists market
researchers in comprehending the preferences of consumers, allowing them to formulate
ideas and strategies accordingly.

O
Other instances of qualitative data include:
™™ What language do you converse in?
™™ Favourite vacation locale

ty
™™ Opinion regarding something (agreement, disagreement, or neutrality)
™™ Colours
The Qualitative data are further classified into two parts:

si
Nominal Data
Nominal Data is used to designate variables with no numerical or ordinal value.
er
The colour of hair can be regarded as nominal data because no two colours can be
compared. “Nominal” is derived from the Latin word “nomen,” which means “name.” With
nominal data, we cannot perform numerical duties or categorise the data in any particular
v
order. There is no meaningful order to these data; their values are distributed across
distinct categories.
ni

Examples of Nominal Data:

™™ Hair colour (blonde, auburn, brown, black, etc.)
™™ Relationship status (single, widowed, married).
U

™™ Nationality (Indian, German, United States).

™™ Sex (Male, Female and Others)
™™ Eye Colour (such as Black or Brown)
ity

Ordinal Data
Ordinal data contain a built-in ordering whereby a number is shown in a certain
order dependent on where it falls on a scale. Although we cannot conduct mathematical
operations on this data, they are utilised for observational reasons such as gauging
m

consumer happiness, contentment, etc.

Qualitative data with positioned values in relation to one another are called
ordinal data. These kinds of data might be seen as falling “in-between” qualitative and
)A

quantitative data. Ordinal data cannot be used for statistical analysis since it only shows
sequences. In contrast to nominal data, which lacks order, ordinal data do have an order.
Examples of Ordinal Data:
™™ When businesses request feedback, experience, or satisfaction on a 1 to 10
(c

scale
™™ Examination letter grades (A, B, C, D, etc.)

Amity Directorate of Distance & Online Education

 12 Data Engineering

™™ Positioning of contestants (First, Second, Third, etc.)

Notes

e
™™ Economic Condition (High, Moderate and Low)
™™ Level of Education (Higher, Secondary, Primary)

in
Quantitative Data
Quantitative information may be expressed as numerical values, making it possible
to count and analyse statistical data. These kinds of data are also known as numerical

nl
data. It provides answers to questions like “how much,” “how many,” and “how often.”
Examples of quantitative data include the cost of a phone, the memory capacity of a
computer, a person’s height or weight, etc.

O
Statistics may be used to modify quantitative data. Numerous graphs and charts,
such as bar graphs, histograms, scatter plots, boxplots, pie charts and line graphs,
among others, can be used to display these data.

ty
Examples of Quantitative Data:
™™ A person or thing’s height or weight
™™ Environmental Temperature

si
™™ Grades and Scores, such as 59, 80, 60, etc.
™™ Time er
Additionally, the quantitative data are separated into two different groups:

Discrete Data
Discrete refers to something that is different or separate. The values in the discrete
v
data are all integers or whole numbers. The total number of students in a class is an
illustration of discrete data. There is no way to convert these integers to decimal or
ni

fractional values. The discrete data cannot be divided since they are countable and have
limited values. Most of these statistics are shown as frequency tables, number lines, or
bar graphs.
U

Here are some instances of discrete data:

™™ The total number of students who are physically present in a certain classroom.
™™ The cost of a cell phone
ity

™™ The total number of employees at a company

™™ The total number of participants in a certain competition
™™ The duration of a week;

Continuous Data
m

The format used to express continuous data is fractional numbers. It may be

the Android phone’s version, a person’s height, the size of an object, or something
completely else. Continuous data are used to describe information that may be
)A

categorised in more specific ways. The continuous variable can be given any value as
long as it falls within the specified range.
The main difference between discrete data and continuous data, which only includes
partial or average values, is that discrete data only contains whole numbers or integers.
(c

The fractional values required to record a variety of data types, including temperature,
height, breadth, time and speed, among others, are nevertheless retained by continuous
data.

Amity Directorate of Distance & Online Education

 Data Engineering 13

Here are a few instances of continuous data:

Notes

e
™™ A person’s stature or height
™™ A vehicle’s speed of travel

in
™™ “Time-taken” to finish the assignment
™™ The Wi-Fi Frequency
™™ The cost per market share.

nl
1.3.2 Sources of Data Using Service Bindings
Connecting and synchronising two data sources is known as data binding. When

O
data binding is employed, a change to one element in a data set automatically changes
the bound data set as well.
Data entry, reporting, text box components and connecting an application’s user

ty
interface (UI) to the data it displays are just a few uses for data binding. Additionally, it
enables online users to alter data representation in a web page’s parts without the
requirement for sophisticated scripting or programming.

si
The data and data objects of different logic functions can be joined together through
data binding. Extensible Markup Language (XML) and User Interface (UI) are examples
of data types having separate languages that may be coupled.
er
Every change made to one piece of data instantly updates the other set. In binding
syntax, the data source is the data provider and the other data set is the data consumer.
The binding creates the link between the data source and consumer, enabling the
connection between data from a visual element and the data source.
v
The requirement to manipulate the Document Object Model (DOM) is removed via
data binding. DOM is an Extensible Markup Language (XML) and Hypertext Markup
ni

Language (HTML) application programming interface (API).

What kinds of data binding are there?

U

The data flow of the following forms of data binding often serves as a definition for
each:
●● One-way binding is a data binding type that is rather simple. The data consumer
ity

data set immediately updates when changes are made to the data provider, but not
the other way around.
●● Two-way binding happens when alterations to either the data consumer or the data
provider trigger the automatic updating of the other.
●● One-way-to-source binding refers to the opposite of one-way binding. Changes to
m

the data consumer automatically update the data provider, but not the other way
around.
●● One-time binding occurs when modifications to the data provider do not result in an
)A

automatic update to the data consumer. When only a static snapshot of the data is
needed and the data is static, this technique is advantageous.
Data binding might be simple or difficult. Microsoft defines simple data binding as
the ability to attach to a single data element. A complicated data binding is created by
(c

bonding together several components.

How to use data binding

Amity Directorate of Distance & Online Education

 14 Data Engineering

Data source objects (DSOs), which supply data to the pages being viewed, make up
Notes

e
the data binding architecture. Additionally, DSOs offer information to agents that ensure
data synchronisation between DSOs and consumers as well as data consumers who
display DSO information.

in
The [Link] framework offers both basic and sophisticated data binding
techniques. Setting up basic data binding will allow the data value from the data source to
be tied to a single UI control.

nl
By connecting the attributes of target objects and data sources, such as Common
Language Runtime, Language Integrated Query and XML objects, Windows Presentation
Foundation [Link] uses data binding. The utilisation of available data templates may also

O
be used to control how data is presented.
Users can declaratively tie UI components to data sources using data binding
frameworks. Additionally, these libraries include classes and techniques for detecting

ty
data changes. Collections, domains and objects become more obvious as a result.

Data binding examples

The usage of data binding is demonstrated by the examples below:

si
™™ Reporting. Reports that display data from a data source on a computer screen
or printer are frequently put together using binding.
er
™™ Data entry. Another popular technique for inputting big amounts of data and
maintaining its synchronisation and updating with a data source is data binding.
™™ Lookup databases. Larger data presentations frequently use lookup tables as a
component. Using controls and data binding to display and alter data.
v
™™ ● Master-detail formats. This is a paradigm
ni

r a communication protocol used when one device or process controls another.

Two data tables may be joined together in these forms.

1.4 Data Storage and Processing

U

Introduction

What is data processing?

ity

Data processing is the process of gathering data and turning it into information that
can be used. In order to avoid having a negative effect on the finished result, or data
output, data processing, which is normally carried out by a data scientist or team of data
scientists, must be done appropriately. Unprocessed data is the starting point for data
processing, which converts information into a more readable format (graphs, papers,
m

etc.) and gives it the shape and context needed for computer interpretation and use by
personnel throughout a business.
)A

Six Stages of Data Processing

1. Data collection
The first step in data processing is data collecting. Data is taken out of sources like
data warehouses and reservoirs. In order to gather data (and later utilise that data
as information), it is essential that the sources of that data be trustworthy and well-
(c

constructed.
2. Data preparation
Amity Directorate of Distance & Online Education
 Data Engineering 15

Data preparation comes next after data collection. Unprocessed data are cleaned
Notes

e
up and structured during the data preparation step, commonly referred to as “pre-
processing,” in order to be used in the following stage of data processing. Unprocessed
data is carefully checked for mistakes throughout preparation. The goal of this stage is

in
to get rid of bad data (redundant, inaccurate, or incomplete data) and start producing
high-quality data for the best possible business intelligence.
3. Data input

nl
The information is then inputted and converted into a language that the final destination
can understand (perhaps a CRM like Salesforce or a data repository like Redshift).
Data entry is the first stage in turning raw data into information that can be used.

O
4. Processing
The data entered into the computer during the previous step is processed for
interpretation at this stage. Machine learning algorithms are used to process the

ty
data, though the actual process may differ slightly depending on the data’s source
(data lakes, social networks, connected devices, etc.) and intended use (looking
at advertising patterns, using connected devices to diagnose illnesses, figuring out
customer needs, etc.).

si
5. Data output/interpretation
The output/interpretation step is where non-data scientists may finally access the
er
data. It is typically delivered in simple text, videos, photos, diagrams and other
easily readable formats. Self-service data analytics initiatives may now be started by
organisation or institution members.
v
6. Data storage
Data storage is the last stage of data processing. Data is preserved for use in the
ni

future after processing. The majority of the knowledge will be helpful in the future,
even though some of it may be used right away. Effective data storage is furthermore
necessary for compliance with data protection legislation like the GDPR. Members of
the company may quickly and readily access data as needed when it is properly kept.
U

1.4.1 Overview of RDBMS and NoSQL Databases

●● RDBMS
ity

RDBMS, or relational database management systems, is an acronym. In essence, it

is a programme that makes it possible to create, delete and modify relational databases.
A relational database is a type of database that organises tabular data into rows and
columns for storing and retrieving. It is a subset of DBMS that E.F. Codd developed in the
1970s. Relational DBMS concepts are the foundation for all of the major DBMS, including
m

SQL, My-SQL and ORACLE.

By emulating the following traits, relational database management systems protect
the integrity of data:
)A

™™ Entity Integrity: A database table’s records cannot include duplicates of any

other records.
™™ Referential Integrity: Records of tables may only be destroyed if they are not
referenced by other tables. If not, it could lead to data that are inconsistent.
(c

™™ User-defined Integrity: The users determine the rules for access and
confidentiality.

Amity Directorate of Distance & Online Education

 16 Data Engineering

™™ Domain integrity: The database tables’ columns are constrained by specified

Notes

e
default values, data types, or ranges.

What is table/Relation?

in
All of the data in relational databases is in the form of relations. Data is stored in
tables in the RDBMS database. Data is stored in rows and columns in a table, which
is a collection of connected data elements. Each table represents a real-world object

nl
for which data was gathered, such as a person, place, or event. The collection of data
grouped into relational tables forms the logical perspective of the database.

Properties of a Relation:

O
™™ A unique name is used in the database to identify each relation.
™™ There are no redundant tuples in the relation.
™™ The order of the tuples in a relation is random.

ty
™™ Each relation cell holds precisely one value, making each attribute in a relation
atomic.
The simplest type of RDBMS data storage is a table.

si
●● NoSQL Database
Large volumes of unstructured and semi-structured data may be managed and
er
stored using the NoSQL database management system (DBMS). NoSQL databases
use flexible data models that may adapt to changes in data structures and expand
horizontally to meet growing data volumes, in contrast to traditional relational databases
that store data in tables with predetermined schemas. NoSQL initially meant “non-SQL”
v
or “non-relational” databases, but the phrase has subsequently changed to signify “not
only SQL,” since NoSQL databases have expanded to include a broad range of database
ni

designs and data formats.

NoSQL databases may be broadly divided into four groups:
1. Document databases: These databases allow for querying using document-
U

oriented query languages and store data as semi-structured documents like

JSON or XML.
2. Key-value stores: These databases are designed for simple and quick read/
ity

write operations and include data in the form of key-value pairs.

3. Column-family stores: These databases store data in sets of columns known
as “column families,” which are treated as a single entity. They are designed to
query huge data collections quickly and effectively.
m

4. Graph databases: These databases use nodes and edges to store data and
handle complicated data interactions.
Applications like social media analytics, e-commerce and gaming typically employ
NoSQL databases to handle and analyse enormous volumes of data in real-time. They
)A

may also be used to other systems, such document management, content management
and customer relationship management.
Since they might not provide the same level of data consistency and transactional
guarantees as relational databases, noSQL databases might not be suitable for all
(c

applications. It is crucial to carefully assess the application’s unique needs before

choosing a database management system.

Amity Directorate of Distance & Online Education

 Data Engineering 17

Key Features of NoSQL

Notes

e
1. Dynamic schema: NoSQL databases don’t have a set schema; therefore they can
adapt to changing data structures without the requirement for schema migrations or

in
revisions.
2. Horizontal scalability: NoSQL databases are made to be expandable by adding
additional nodes to a database cluster, which makes them ideal for handling massive

nl
amounts of data and high traffic levels.
3. Document-based: Using a document-based data paradigm, certain NoSQL databases,
like MongoDB, store data in semi-structured forms like JSON or BSON.

O
4. Key-value-based: Using a key-value data format, some NoSQL databases, like Redis,
store data as a collection of key-value pairs.
5. Column-based: A few NoSQL databases, including Cassandra, use a column-based
data paradigm, where data is structured into columns rather than rows.

ty
6. Distributed and high availability: NoSQL databases are typically made to be highly
available and to automatically handle node failures and data replication across several
nodes in a database cluster.

si
7. Flexibility: NoSQL databases allow for the flexible and dynamic storing and retrieval
of data by programmers. These databases can adapt to changing data architectures
and accommodate a wide range of data types.
er
8. Performance: NoSQL databases can handle enormous volumes of reads and writes
and are geared for high speed, making them suitable for real-time and massive data
applications.
v
The many varieties of NoSQL databases are listed below, along with the name of the
database management system that goes with each type:
ni

1. A Few Examples of Graph Databases, Including Neo4j and Amazon Neptune

2. Key value stores such as Memcached, Redis and Coherence are examples.
U

3. Tabular, including Hbase, Big Table and Accumulo as a few examples.

4. Document-based databases, such as MongoDB, CouchDB and Cloudant are a few
examples.
ity

1.4.2 Introduction to Data Marts and Data Lakes

Introduction
The data mart is a component of the data warehouse that was designed and
m

constructed for a specific operational activity or a business function, purpose, or

community of users. The establishment of data marts is justified for a variety of reasons,
one of which is the facilitation of quick access to data on a regular basis. The second
benefit is a faster response time for end users. Thirdly, the production of a data mart
)A

is much simpler than the creation of a data warehouse. While the creation of a data
warehouse requires a significant amount of effort and resources, the creation of a
data mart is far less complicated. Data mart is associated with a negligible amount of
additional expense.
(c

Types of Data Mart

There are typically three types:

Amity Directorate of Distance & Online Education

 18 Data Engineering

1. Dependent Data Warehouse

Notes

e
™™ A dependent data mart is derived entirely from the data warehouse and the
enterprise data warehouse will be formed by grouping all of the dependent data

in
marts together. Since it is derived from the core DW, it can only be considered a
subset of the data warehouse.
™™ The ETT procedure, which stands for “Extract, Transform and Transportation,”

nl
is made easier due to the fact that data that has been cleaned and summarised
is already available in the central data warehouse. Simply locating this
particular subset and then doing ETT on top of it is all that is required of us
here. These data marts are often constructed with the goal of achieving more

O
availability as well as a significant increase in performance while simultaneously
improving both control and efficiency.
2. Independent Data Mart

ty
™™ This does not originate from the centralised data warehouse and its source may
be changing from time to time. The ETT procedure is slightly altered due to the
fact that the data come from sources other than the central DW.

si
™™ A smaller number of organisations make use of the majority of the independent
data mart and there are additional restrictions on the sources for this
information. In most cases, we turn to the creation of an independent data mart
er
when we are under pressure to find a solution in a relatively short amount of
time.
3. Hybrid Data Mart
v
™™ The hybrid data mart will provide you the ability to aggregate the data that
comes from all of the other sources except the core data warehouse DW.
ni

™™ When we are working with ad hoc integration, this will be of tremendous help to
the top work on all of the items that were brought into the organisations from the
outside.
U

Data Lake
We can store all of our structured and unstructured data in a data lake, which is
a centralized data repository, on a huge scale. This greatly increases our flexibility.
ity

You may utilize a range of analytics, from dashboards and visualisations to big data
processing, real-time analytics and machine learning, to help you make better decisions
without first structuring your data. The data that makes up a data lake is frequently
stored in the form of files or objects in a storage system with a flat design, as opposed
to a traditional data warehouse, which arranges its data into hierarchical dimensions
m

and tables. Users now have a larger range of options for managing, storing and using
their data.

Development of Data Lakes

)A

The shortcomings of data warehouses directly prompted the creation of data lakes.
Data warehouses are expensive, proprietary and unable to handle the contemporary
use cases that the great majority of organisations are trying to address, despite the fact
that they provide organisations with very effective and scalable analytics. A data lake
(c

enables all of an organization’s data to be kept in a single, central location where it may
be retained in its present condition, as opposed to a data warehouse which imposes
a schema (i.e., a formal structure for how the data is arranged) at the beginning. In
Amity Directorate of Distance & Online Education
 Data Engineering 19

contrast, a data warehouse necessitates that the data be first arranged in accordance
Notes

e
with the schema.

in
nl
O
ty
si
Source: A Detailed Introduction on Data Lakes and Delta Lakes - Analytics Vidhya

All phases of the data refinement process, including the intermediate data tables
er
created throughout the process of refining raw data, can be stored in a data lake. An
organization’s unstructured, tabular data sources (such database tables) can be used
in conjunction with its structured, tabular data sources to be consumed and stored. For
v
today’s machine learning and advanced analytics use cases, data lakes must be able to
analyse all forms of data, including unstructured and semi-structured data like photos,
ni

video, audio and documents. This is different from the vast majority of databases and
data warehouses.

Why should we use Data Lakes?

U

Data lakes are highly useful and practical, therefore let’s look at the prerequisites
and uses of these systems.
●● Data lakes and Hadoop-based computing platforms are frequently addressed
ity

together. Data is loaded into the Hadoop Distributed File System (HDFS) and
then stored on the several computer nodes that make up a Hadoop cluster in
deployments based on the distributed processing architecture. However, data lakes
are increasingly being created with the aid of cloud object storage providers rather
than using Hadoop. Some NoSQL databases can also act as the foundation for data
m

lakes within a company.

●● Data lakes are frequently used to store large quantities of data, which may include
structured, unstructured, or semi-structured data. Semi-structured data may also
)A

be present in these lakes. Relational databases, which form the backbone of the
great majority of data warehouses, are not best suited for handling situations like
these. Relational systems frequently can only store data that is ordered and tied to
transactions since they need a specified data schema. A data lake does not require
an initial definition and is capable of handling a variety of schemas. They can now
(c

manage many sorts of data provided in different formats as a result of this.

●● As a result, data lakes are a crucial part of the data infrastructures of many different

Amity Directorate of Distance & Online Education

 20 Data Engineering

businesses. Enterprises often utilise them as a platform for big data analytics and
Notes

e
other applications involving data science. Large volumes of data are required for
these applications, which also use advanced analytics techniques like machine
learning, predictive modelling and data mining.

in
●● A data lake provides data scientists and analysts with a centralised space in which
they can identify, prepare and evaluate relevant data. Without one, carrying out that
procedure is made more difficult. In addition, it is becoming increasingly challenging

nl
for businesses to make full use of the data assets they possess in order to support
better informed business decisions and strategies.

O
1.4.3 ETL and Data Pipelines
What is a Data Pipeline?

Data Pipeline

ty
si
er
Source: ETL vs Data Pipeline : A Comprehensive Guide 101 ([Link])
v
The Data Pipeline serves as the means via which you may transmit data from
ni

the source system or application to the data repository of your choice. This can be
accomplished by using the Data Pipeline. The architecture of a data pipeline is composed
of software tools that work together to automate the process of moving data from one
U

location to another. It’s possible for a Data Pipeline to incorporate a number of different
sub-processes, such as data extraction, transformation, aggregation, validation and so
on. It is a catch-all phrase for any and all data-related operations that may take place
throughout the movement of data from its point of origin to its final destination.
ity

The phrase “Data Pipeline” can be used to refer to a variety of different sets of
operations that, in the end, move data from one area to another. This suggests that a
Data Pipeline does not require the data to be transformed as it is being sent. This
highlights the distinction between an ETL and a Data Pipeline, the latter of which is
m

itself a subcategory of Data Pipeline. In general, any of the subprocesses, such as Data
Replication, Filtering, Transformation, Migrations, etc., can be present in any sequence
as part of a Data Pipeline. This is because the order in which they are performed does
not matter.
)A

Data Pipeline Types and Use Cases

Processing in batches and processing in real time are the two primary types of
pipelines. This is why:
(c

●● Data pipelines are used to implement data integration.

●● Data integration is the process of combining data from numerous sources to create

Amity Directorate of Distance & Online Education

 Data Engineering 21

a dataset that is complete and accurate for use in business intelligence (BI), data
Notes

e
analysis and various other applications and business activities.
●● Real-time data and historical data are the two primary categories into which the

in
needs and use cases of various analytics, applications and processes may be
classified.
●● As a result, it makes sense to categorise pipelines similarly.

nl
1) Batch processing: historical data
Using historical data to explore, research and gain insights into prior events and
information is a frequent practice in business intelligence (BI) and data analytics.

O
Consequently, it is sufficient to do batch processing, which entails routinely extracting,
converting and loading data into a target system. These runs can be programmed to
occur automatically in accordance with a specified schedule, at the request of the user,

ty
or at the beggining of an application. Batch processing enables complex analysis to be
performed on large datasets.
ETL pipelines may help with use cases that could rely on historical data and they are
especially useful when working with small data sets that demand intricate adjustments.

si
Converting raw data to be compatible with the target system before putting it into the
target repository enables systematic and accurate data analysis in the target repository.
This pipeline’s three processes are generally referred to by the acronym ETL, which
stands for “Extract, Transform and Load”:
er
●● Extract: Extracting raw data from a source (such as a database, an XML file, or a cloud
platform that saves data for applications like marketing tools, customer relationship
v
management systems, or transactional systems) is the process of doing so.
●● Transform: The process of modifying the dataset’s structure and format to make it
ni

compatible with the target system

●● Load: Adding a dataset to a target system, such as an application, database, data
lakehouse, data lake, or data warehouse, is known as loading it.
U
ity
m
)A
(c

Source: What is a Data Pipeline? Definition, Types & Use Cases ([Link])

Amity Directorate of Distance & Online Education

 22 Data Engineering

2) Streaming: real-time data

Notes

e
Streaming data pipelines are put into action whenever the analytics, application,
or business process in question calls for data to be continuously updated and flowed.

in
Streaming pipelines do not load data in batches like traditional pipelines do; rather, they
transport data constantly and in real time from source to goal. The “Extract, Load and
Transform” (ELT) procedure is preferable to the “Extract, Load and Transform” (ETL)
approach when dealing with big, unstructured data sets and when timely processing is

nl
essential.

O
ty
si
v er
ni
U
ity
m

Source: What is a Data Pipeline? Definition, Types & Use Cases ([Link])
)A

Without having to wait for IT to finish extracting, converting and loading more data,
users may analyse or report on their whole dataset, including any real-time data. The
main benefit of streaming pipelines is this. Additionally, compared to batch-oriented
pipelines, the cost of operation and maintenance is reduced. Given that all of the data
is always accessible and the transformation process is typically automated and cloud-
(c

based, cloud-based platforms considerably reduce the expenses associated with storing
and processing data. Additionally, cloud-based technologies provide considerably more
affordable data processing and storage.

Amity Directorate of Distance & Online Education

 Data Engineering 23

What is an ETL Pipeline?

Notes

e
in
nl
O
ty
Source: ETL vs Data Pipeline : A Comprehensive Guide 101 ([Link])

Your company creates and gathers enormous amounts of data on a daily basis.

si
Now, in order to extract any useful information from this vast ocean of data, you will need
to do the following:
●● Extraction and collection of data from a wide variety of sources that are in some
manner pertinent to your company’s operations.
er
●● Make the extracted data appropriate for data analysis by cleaning and transforming
it in accordance with the instructions provided.
v
●● Create a single source of truth by loading the changed datasets into the repository
that you have selected, which may be a Data Lake or a Data Warehouse, for
ni

example.
Now, these procedures operate together in a coordinated fashion to transform
your raw data into a format that is suitable for analysis. However, if you carry out the
U

aforementioned operations manually, you run the risk of making a variety of blunders.
When dealing with a manual ETL technique, your process code may throw abrupt errors,
specific data values may go missing, data discrepancies may emerge and many more
bottlenecks of a similar kind are likely.
ity

An ETL pipeline is what most businesses rely on to automate the three-step process
and convert their data in a safe manner. An ETL Pipeline is a collection of tools that work
together to extract raw data from various sources, convert and aggregate that raw data
and ultimately load the transformed and aggregated raw data into the storage location
of your choice. The best ETL pipelines will also provide you with administration for the
m

entire process, from beginning to end, as well as an error-handling system.

1.5 Big Data and Processing Tools

)A

Introduction
Big data is a collection of organised, semi-structured and unstructured information
that businesses have acquired. In projects involving machine learning, predictive
modelling and other applications of advanced analytics, this data may be mined for
(c

information.

Sources of Big Data

Amity Directorate of Distance & Online Education
 24 Data Engineering

Many different sources, including

Notes

e
●● Social networking websites: These sites, such as Facebook, Google and LinkedIn,
each produce enormous amounts of data every day since they have billions of users

in
worldwide.
●● E-commerce website: Websites like Amazon, Flipkart and Alibaba generate a lot of
logs that may be utilised to analyse user purchasing behaviour.

nl
●● Weather Stations: Each and every satellite and weather station produces enormous
volumes of data, which are then preserved and altered to produce weather
predictions.

O
●● Telecom company: In order to analyse user trends and promote new services in line
with those results, telecom juggernauts like Airtel and Vodafone preserve the data of
their millions of clients.
●● Share Market: The regular operations of stock exchanges throughout the world

ty
generate a tremendous amount of data.

3V’s of Big Data

1. Velocity: The amount of data being collected is rapidly growing at this point. It is

si
anticipated that the quantity of data would increase by a factor of two every two years.
2. Variety: These days, data are not kept in rows and columns as they were in the past.
er
There are both organised and unstructured forms of data. Unstructured data includes
things like log files and images from CCTV cameras. Structured data, such as the
bank’s transaction data, are examples of the types of data that may be recorded in
tables.
v
3. Volume: The quantity of data that we work with is on the order of peta bytes, which is
a very enormous size.
ni

1.5.1 Foundations of Big Data

What is Big Data?
U

A collection of data is described as “big data” if it is not only extremely large but
also grows exponentially over time. None of the traditional techniques for managing data
are able to store or handle it effectively because of its large amount and high degree of
ity

complexity. Big data is simply exceptionally enormous amounts of data that have been
saved. The data gathered on people through social media platforms is one example of
big data. Big data makes it simpler to see patterns in massive volumes of data, which in
turn enables a greater understanding of both human and corporate behaviour. This helps
ensure that orders are processed quickly, which leads to satisfied consumers. Big data
m

can be organised or unstructured, come from raw or processed sources and be related to
the passage of time.

Basic Units of Memory Measurements

)A

1 Bit = Binary Digit (0 or 1)

8 Bits = 1 Bite = 2 nibble
1024 Bytes = 1 KB (Kilobyte)
(c

1024 KB = 1 MB (Megabyte)
1024 MB = 1 GB (Giga Byte)

Amity Directorate of Distance & Online Education

 Data Engineering 25

1024 GB = 1 TB (Tera Byte)

Notes

e
1024 TB = 1 PB (Peta Byte)
1024 PB = 1 EB (Exa Byte)

in
1024 EB = 1 ZB (Zetta Byte)
1024 ZB = 1 YB (Yotta Byte)

nl
1024 YB = 1 (Bronto Byte)
1024 Brontobyte = 1 (Geop Byte)

Main Components of Big Data

O
Given below are the main components of big data:
1. Machine Learning
Artificial intelligence (AI) is the study of how to programme computers to learn on

ty
their own. The fields of machine learning, algorithm development and statistical
modelling all anticipate that a computer will carry out predetermined tasks without
being given any explicit instructions. Applications that use machine learning produce

si
outcomes that are based on experience. For instance, in today’s world, certain mobile
applications will provide you with a summary of your finances and bills will remind you
of your bill payments and also may provide you with advice to go for various savings
er
programmes. The reading of your emails and text messages is how these functions
are carried out.
2. Natural Language Processing (NLP)
v
A computer is capable of comprehending human language in its spoken form. The most
obvious examples that people may relate to in this day and age are digital assistants
ni

such as Google Home and Amazon Alexa. Both Siri and Alexa provide us with a
virtual assistant experience by utilising NLP and other technologies. Without even
being aware of it, we are constantly surrounded by NLP. While you are composing an
U

email, if you make any mistakes, it will automatically repair them for you. These days,
it provides auto-suggestions for finishing the emails and automatically intimidates
us when we try to send an email without the attachment that we mentioned in the
text. This is because Natural Language Processing Applications are operating in the
ity

background and they are doing so as a function of the technology.

3. Business Intelligence
Business intelligence (BI) is a technology-driven method or process for gaining
insights by analysing data and presenting it in a way that end-users (typically high-
m

level executives), such as managers and corporate leaders, can use it to form sound
business judgments. BI software, which may be installed on a PC or a server, can be
used to do this.
)A

4. Cloud Computing
While it is true that we are not talking about physical clouds in this instance, the
cloud in this context refers to the Internet. If we take the term into consideration, it
should be computing that is done on clouds. What we mean when we talk about cloud
(c

computing is the provision of computing services such as servers, storage, databases,

networking, software, analytics, intelligence and many more through the Internet (also
known as “the cloud”) in order to provide faster innovation, flexible resources and

Amity Directorate of Distance & Online Education

 26 Data Engineering

economies of scale.
Notes

e
Characteristics of Big Data
The following is a list of features that Big Data possesses:

in
●● Volume: When determining the worth of the data, one must first take into account
its size, which is an important factor. The magnitude of the data must also be
considered when determining whether or not a specific type of data is in the

nl
category of an introduction to Big Data.
●● Variety: Different data kinds according to their nature (structured and unstructured)
is what is meant by variety in this context. In the past, the only sources of data

O
that were taken into consideration by the majority of applications were those that
were in the form of rows and columns. These rows and columns often came from
spreadsheets and databases. However, in today’s world, data may be found in a
variety of formats, such as emails, photographs, movies and audio recordings, etc.

ty
●● Velocity: Velocity refers to the rate at which new data is generated, as suggested by
its name. The potential of data is determined by how quickly it can be created from a
source and how quickly it can be processed after it has been generated.

si
●● Variability: Data can be variable, which means that it might be inconsistent and
out of flow, which could obstruct or interfere with the process of processing and
managing data effectively.
er
Applications of Big Data
The following examples illustrate how analytics are being used to big data:
v
●● Medical Care: In the present day, sensors and wearable technology may provide
real-time updates to a patient’s health status.
ni

●● Education: By employing big data analytics to do the required analysis, the

development of a student may be tracked and improved.
●● Weather: Weather sensors and satellites have been put all over the world to monitor
U

the weather and environmental conditions and to either predict or forecast the
weather for the upcoming few days.

1.5.2 Overview of Big Data Processing Tools

ity

1) Hadoop
A framework for large amounts of data is the Apache Hadoop software library.
Distributed processing of big data sets across clusters of computers is made possible by
this technology. It is one of the most effective tools for working with large amounts of data
m

and is intended to expand easily from individual servers to thousands of computers.

)A
(c

Source: [Link]

Features:

Amity Directorate of Distance & Online Education

 Data Engineering 27

™™ Enhancements to the authentication process when utilising an HTTP proxy

Notes

e
server.
™™ The Specification for the Effort to Create a Hadoop-Compatible Filesystem.

in
™™ Support for expanded characteristics on filesystems in the POSIX style.
™™ It provides a comprehensive environment that is ideally adapted to satisfy the
analytic demands of developers, thanks to its big data technologies and tools.

nl
™™ It allows for greater flexibility in the processing of data.
™™ It makes it possible to process data more quickly.

2) Zoho Analytics

O
The Zoho Analytics platform provides users with self-service access to business
information and analytics. Users are able to construct intelligent dashboards and
graphically analyse any data in a matter of minutes with this tool. It provides customers

ty
with the ability to ask questions and receive intelligent responses in the form of useful
reports via an assistant that is powered by artificial intelligence (AI).

si
er
Source: [Link]
v
Features:
™™ More than one hundred pre-built connections for the most common corporate
ni

applications, cloud storage and databases.

™™ A large number of different visualisation choices, including charts, pivot
tables, summary views, key performance indicator widgets and individualised
U

dashboard themes.
™™ Business analytics unified for the purpose of analysing data from all business
applications.
ity

™™ Enhanced analytics through the use of AI, ML and NLP.

™™ Embedded analytics systems and white label business intelligence platforms.

3) [Link]
[Link] is a research programme that handles everything. Utilising this big data
m

analysis tool will provide you access to all available platforms. You may use it for mixed
techniques and qualitative data analysis in academic, market and user experience
research.
)A

Source: [Link]
(c

Features:
™™ You have access to information about every data source that you can export.
Amity Directorate of Distance & Online Education
 28 Data Engineering

™™ It offers a comprehensive approach to managing the data you have available.

Notes

e
™™ Allows you to change a Code’s name that is situated in the Margin Area.
™™ Helps you with efforts involving the handling of tens of thousands of documents

in
and chunks of coded data.

4) HPCC
The big data tool for HPCC was developed by LexisNexis Risk Solution. It only uses

nl
one platform, one architecture and one programming language to carry out all of the data
processing activities.

O
Source: [Link]

ty
Features:
™™ It is one of the very effective tools for big data that allows users to do jobs

si
involving huge data with significantly less code.
™™ It is one of the instruments for processing large amounts of data that provides a
high level of redundancy and availability
er
™™ It is capable of being utilised for both complicated data processing on a Thor
cluster as well as for
™™ Graphical integrated development environment that makes development,
testing and debugging easier
v
™™ It optimises code in an automated fashion so that it may be processed in
ni

parallel.
™™ Ensure improved scalability as well as overall performance
™™ The code for ECL can be compiled into an optimised form of C++ and it is also
U

capable of being extended using C++ libraries.

5) Storm
A system for computing enormous volumes of data that is free and open-source is
ity

called Storm. It is one of the best tools for handling enormous amounts of data since it
offers a distributed, fault-tolerant real-time processing system. having the capacity to do
calculations instantly.
m

Source: [Link]
)A

Features:
™™ This tool is one of the best on the list of big data tools, processing one million
100-byte messages per second per node as the criterion for judging its
performance.
(c

™™ It is furnished with big data tools and technologies that employ parallel
calculations carried out over a cluster of computers.

Amity Directorate of Distance & Online Education

 Data Engineering 29

™™ Should a node die, it will restart itself right away. The worker will start up a fresh
Notes

e
instance on a separate node.
™™ Using Storm guarantees that every single piece of data will be processed

in
exactly once or at least once.

6) Cassandra
Today, the database known as Apache Cassandra is utilised extensively for its ability

nl
to enable efficient administration of significant volumes of data.

O
Source: [Link]

ty
Features:
™™ Support for duplicating data across several data centres by lowering the amount

si
of delay experienced by users
™™ In order to ensure availability at all times, data is automatically duplicated
across a number of nodes.
er
™™ It is one of the greatest big data tools and it is most ideal for applications that
cannot afford to lose data, even when an entire data centre is offline.
™™ Support contracts and services are available from third parties and may be
v
obtained through Cassandra.
ni

1.6 Gathering and Importing Data

Introduction
U

Data collection may be seen in action when information is gathered for research,
strategic planning and decision-making inside an organisation. It is a crucial element
of any data analytics applications as well as research projects: Effective data collection
provides the information needed to provide answers to questions, assess business
ity

performance or other outcomes and foresee future trends, actions and scenarios.
Inside of businesses, there are many levels of data collection. Information
technology (IT) systems frequently gather data about the company’s clients, employees,
sales and other aspects of business operations as transactions are carried out and data
is submitted. Aside from surveys, organisations also utilise social media monitoring and
m

customer activity tracking to get customer feedback. Then, from within the company’s
systems and, if necessary, from external data sources, data scientists, other analysts and
business users gather pertinent data for analysis. The second duty comprises gathering
)A

data and getting it ready for usage in business intelligence (BI) and analytics applications.
This is the initial stage of data preparation. The first phase in data preparation is the
second duty.
For study on topics like science, health, higher education and a number of other
sectors, data collection is frequently a more specialist process. In this procedure,
(c

researchers develop and use methods to collect particular types of data. However, the
data that was acquired must be accurate in both the business and the research contexts

Amity Directorate of Distance & Online Education

 30 Data Engineering

in order to guarantee that the inferences made from analytics and the results of research
Notes

e
are trustworthy.

Data Import

in
You may transfer data from other sources and combine it with the data you’ve
gathered using Analytics by utilising Data Import. After that, you may use Analytics to
structure and evaluate your data more effectively so that it more accurately represents

nl
your business. You may integrate the data generated by your offline business systems
with the data that Analytics collects online by using Data Import. You will be better
able to arrange, assess and respond based on this unified data viewpoint in ways that

O
are more in line with the unique needs of your firm. As a web publisher, you may, for
instance, combine Analytics’ hit data with the data dimensions generated by your content
management system (CMS) and customer relationship management system (CRM) to
examine the relative contributions of your site’s writers.

ty
Types of data you can import
Through the use of Data Import, you will have the ability to upload data and link it
with your Analytics account at three distinct junctures along the data gathering and

si
processing chain.
1) Hit-data import er
Using the import hit-data capability, you may send hit data directly into Analytics. There
is a choice to use this in place of the tracking code, Collection API, Mobile SDKs, or
the Measurement Protocol. Before any processing is done, imported hits are changed
in your Analytics property; as a result, filters and other processing-related operations
v
may have an impact on your imported data. All reporting views for that property will be
able to access this data since you are importing hits (unless you specifically filter them
ni

out of selected views) because you are importing hits.

With this type of import, the information types listed below can be uploaded:
™™ Refund Data - Importing refund data from your internal e-commerce platform
U

will help you combine that platform’s reporting with Analytics.

2) Extended-data import
Through the use of extended-data import, data that has already been collected and
ity

processed, or is now being processed, for the designated reporting views is added to
(extended). However, there are times when you might want to change the default data
that has already been collected (for example, when importing a campaign’s Source or
Medium dimension). This extended data is frequently recorded in a custom dimension
or metric. The following data formats are available for upload:
m

™™ User Data: Create segments and remarketing lists that use imported user
metadata, such as a lifetime customer value or a loyalty rating, as an illustration.
™™ Campaign Data: By importing ad campaign-related characteristics like source,
)A

you may improve and reuse your current non-Google campaign codes. This
enables you to utilise your current marketing data more effectively.
™™ Geographical Data: By creating distinct geographic zones, you may report on
and analyse Analytics data in a way that is more directly related to the structure
(c

of your business.
™™ Content Data: Based on imported content information, including author,
publication date and article type, it organises the material into categories.
Amity Directorate of Distance & Online Education
 Data Engineering 31

™™ Product Data: By including product metadata, such as the product’s size,

Notes

e
colour and style, as well as any other product-related criteria, it improves your
merchandising insights.

in
™™ Custom Data: It provides help importing user-defined data sets.
3) Summary-data import
You may use the summary-data import capability to construct supplied metrics. The

nl
imported summary data is then applied to the chosen reporting views once all the
gathered data has been processed and aggregated. When you import summary data,
you may update or add to your data as new information becomes available. This
feature might be useful when you get data in batches following hit collection. When

O
you receive data in batches some time after hit collection, this might be useful. The
following import types are currently compatible with summary data import:
™™ Cost Data—to provide you a more complete picture of your advertising

ty
expenses, including clicks, prices and impression data from ad networks that
are not affiliated with Google.

1.6.1 Identifying Data for Analysis

si
The process of gathering, modelling, and analysing data using various statistical and
logical methodologies is known as data analysis. Businesses rely on analytics procedures
and equipment to gather information for operational and strategic decision-making.
er
Data collecting, cleansing, and organisation are common actions included in data
analysis techniques. The data must go through these procedures, which often involve
using data analysis tools, in order to be ready for commercial use. Data analytics,
v
another name for data analysis, is defined as the science of examining raw data in order
to make defensible inferences about the data.
ni

Data is available in a variety of forms, formats, and kinds, such as the following:
●● Big data. Big data is characterised as a huge data set that keeps expanding
exponentially over time. Big data has four main characteristics: volume, variety,
U

velocity, and variability. Quantity is referred to by volume, data growth is measured

by velocity, and data sources are identified by diversity. Veracity refers to the data’s
quality, indicating whether or not it has any practical application for business.
ity

●● Structured/unstructured data. A typical row-column database is an example of

structured data, which has a specified data model. Unstructured data might
comprise movies, photographs, music, text, and more in a format that does not
fit in rows and columns. Structured data is simpler to handle and analyse when
compared to unstructured data, according to research.
m

●● Metadata. Data that describes and offers details about other data is known as
metadata. For instance, the author, picture type, and creation date can all be
included in the metadata for an image. Users may categorise unstructured data
)A

using metadata, which makes it easier to deal with.

●● Real-time data. Real-time data is information that is shown as soon as it is obtained.
When making judgements that call for current facts, this kind of knowledge is
helpful. For instance, a stockbroker can follow the most active equities in real time
using a stock market ticker.
(c

●● Machine data. Data can be created automatically by manufacturing systems and

other machinery, information technology and telecommunications infrastructure,

Amity Directorate of Distance & Online Education

 32 Data Engineering

smart automobiles, handheld devices, and more thanks to the Internet of Things
Notes

e
(IoT), sensors, and other technologies. Because it is completely generated by
machines without human guidance, this form of data is known as machine data.

in
For gaining insights from data, such as measurements, facts, and numbers, data
analysis tools and procedures are helpful.
In order to do an accurate analysis of the data, follow these steps

nl
1. Set a target for yourself: First things first: figure out why you’re doing this data analysis
and what your primary goals are. Consider whether you have any doubts or worries,
as well as the objective that you hope to accomplish by carrying out this investigation.

O
One such objective of yours may be to broaden the scope of your consumer base.
2. Determine the method of data analytics that will be used: Determine the kind of data
that might provide the answers to your inquiries. An analysis that is descriptive might
be helpful if all you need is a basic overview of the situation. It’s possible that you

ty
already have access to data from a project you worked on in the past. You might
wish to take into consideration making use of this information if these figures are still
applicable.

si
3. Formulate a strategy for the production of the data: The amount of time, effort and
resources necessary to collect data might vary significantly depending on the sort
of information that you are attempting to get. Think about the project’s demands in
er
terms of both its budget and its timeline before making any financial commitments. It
is possible to guarantee that you will have the capacity to finish the data analysis job
if you plan these factors in advance.
v
4. Accumulate the information: Establish the required procedures for the collecting
of data and put together a group of individuals who are capable of amassing and
ni

managing the information. You could decide to purchase access to a data collection,
such as demographic information for a certain group of customers, in order to save
some of your valuable time. In addition, you may easily deploy, simplify and monitor
U

processes by making use of software that does data analytics.

5. Straighten out the data: After you have compiled a sizable data collection, the following
step is to process or clean the data to guarantee that it is complete, correct and devoid
of any mistakes. Check for any duplication or other problems and then standardise the
ity

data so that it will be simpler to evaluate in the future. You can further verify that there
are no errors in the formatting or typography.
6. Conduct an analysis of the data: Investigate thoroughly the information that you have
acquired. Apply a variety of mathematical techniques in order to investigate potential
m

connections between locations. Provide an interpretation of what the data could signify
and the reasons why it might be significant to the organisation.
7. Visualise the data: After you have gathered, cleansed and analysed the data, you can
)A

next use a method such as data visualisation to summarise the information and verify
that it has significance. You can better comprehend the data and display it in a way
that is interesting to others with the assistance of visualisation tools.
Charts and graphs in vivid colours may represent hundreds of data points, each of
(c

which can be used to illustrate a pattern or a compelling story. You might, for example,
elect to reallocate marketing spending based on the results of a sales data run that
was related to a geographic location.

Amity Directorate of Distance & Online Education

 Data Engineering 33

1.6.2 Data Sources and Collection Methods

Notes

e
●● Sources of Data
The process of acquiring, collecting, extracting and storing huge amounts of data

in
is known as data collection. The data may be in the organised or unstructured form,
such as text, video, audio, XML files, records, or other picture files that will be utilised in
subsequent stages of data analysis.

nl
“Data collection” is the first step in the process of analysing large amounts of data
before moving on to analyse the patterns or other information that may be gleaned from
the data. It is required that the data be gathered from a variety of reliable sources before

O
it can be analysed.
The information that is gathered is referred to as “raw data,” and it is now useless.
However, once the impure data are cleansed and the data are used for further analysis,
the information known as “knowledge” is formed. There are many different connotations

ty
associated with the word “knowledge,” including “business knowledge” or “sales of
enterprise products,” “disease treatment,” etc. The acquisition of information-dense data
is the primary objective of the data collecting process.

si
The process of collecting data begins with the gathering of basic questions, such
as what kind of data is going to be gathered and where the data is going to come
from. “Qualitative data” refers to a collection of non-numerical data such as words
er
and sentences that primarily focus on the behaviour and actions of the group and
“quantitative data” refers to data that is in numerical forms and can be calculated using
a variety of scientific tools and sampling data. The majority of the data collected are
of these two types and they are known as “qualitative data” and “quantitative data,”
v
respectively.
The actual data is then further divided mainly into two types known as:
ni

1. Primary data
2. Secondary data
U
ity
m
)A

Source: [Link]
(c

1. Primary data:

Amity Directorate of Distance & Online Education

 34 Data Engineering

The term “primary data” refers to information that has not been altered in any
Notes

e
way and has not been derived from any official sources. Performing methods like
questionnaires, interviews and surveys are some of the direct collection methods that are
used for this sort of data. If the data obtained are not in accordance with the demands

in
and requirements of the target audience on whom the analysis is done, then the data
processing will be hampered. Examples of Primary Data

nl
Some real-time examples of primary data are:
™™ Customer surveys: When a business gathers information via questionnaires
or surveys, it is doing it in the form of primary data. A restaurant may, as an

O
illustration, invite patrons to assess their meal experience.
™™ Market research: To comprehend consumer patterns or market demand,
businesses may carry out primary research. For instance, a business may hold
focus groups or interviews to learn more about consumer preferences.

ty
™™ Scientific experiments: Scientists can use experiments to get primary data,
for as by monitoring animal behaviour or testing novel medications on human
volunteers.

si
™™ Traffic counts: To figure out how to enhance traffic flow, traffic engineers may
gather primary data by observing the movement of automobiles on a certain
road.
er
™™ Consumer behaviour: Businesses may utilise primary data to monitor consumer
behaviour, including how users engage with a product or a website.
™™ Social media analytics: To understand how their customers are interacting with
v
their brand, businesses may gather primary data by examining social media
indicators like likes, comments, and shares.
ni

Few methods of collecting primary data:

1. Interview method
U

Interviews are conducted with members of the target audience by a person referred
to as an interviewer; the individual who responds to the interview is referred to as
an interviewee. The data obtained during this process is known as interviewee data.
Some fundamental questions concerning the company or the product in question are
ity

posed, followed by their notation in the form of notes, audio, or video and then the data
collected is saved for processing. Personal interviews or formal interviews conducted
over the phone, in person, by email, or any other medium can fall under this category.
These interviews can also be unstructured.
m

2. Survey method
The research procedure known as a survey is one in which a set of pertinent questions
are posed to participants and their responses are documented in the form of written
)A

text, audio recordings, or video recordings. Both online and offline methods, such as
filling out website forms or sending an email, are available for obtaining the survey
technique. After then, the survey responses are saved so that they may be analysed
later. There are further examples such as polls on social media platforms and online
surveys.
(c

3. Observation method

Amity Directorate of Distance & Online Education

 Data Engineering 35

The observation method is a type of data gathering approach in which the researcher
Notes

e
carefully observes the behaviour and practices of the target audience using some
type of data collecting instrument and then records the observed data in the form
of text, audio, video, or any other raw forms. This type of data collection method

in
is known as the observation method. Using this approach, the data is gathered in
an unfiltered manner by the participants themselves, in the form of a few questions.
Consider the following scenario: you are monitoring a group of clients and how they

nl
react to the merchandise. The data that was acquired will be transmitted so that it may
be processed.
4. Experimental method:

O
The data collection procedure that constitutes the experimental technique includes
activities such as carrying out experiments, conducting research and conducting
investigations. CRD, RBD, LSD and FD are the experiment procedures that are

ty
utilised the most commonly.
™™ CRD- The Completely Randomised design, sometimes known as CRD for
short, is a straightforward experimental design utilised in data analytics. It is

si
founded on the concepts of randomization and replication. The majority of its
applications revolve around contrasting the many tests.
™™ RBD- RBD stands for Randomised Block Design, which is a type of
er
experimental design in which the experiment is broken down into a series of
discrete sections known as blocks. Analysis of variance (ANOVA) is a statistical
method that is used to compile the findings of randomised experiments that
were conducted on each of the blocks. The agricultural industry was where
v
RBD first got its start.
ni

™™ LSD – LSD, also known as Latin Square Design, is an experimental design that
is comparable to CRD and RBD blocks but incorporates rows and columns into
its layout. It is a configuration of NxN squares with an equal number of rows
and columns that contains letters that appear just once in a column or in a row.
U

As a result, the differences may be readily seen with a reduced risk of making
mistakes in the experiment. The popular puzzle format known as Sudoku is an
example of a Latin square.
ity

™™ FD- Factorial design is a type of experimental design in which each trial

consists of two factors, each of which can take on a variety of values. Based on
the results of the experiments, further combinational factors are determined.

2. Secondary data:
m

Secondary data is data that has previously been acquired but is being utilised
again for another reason that is considered to be legitimate. This kind of data was once
recorded from primary data and it comes from two different kinds of sources, which are
)A

respectively known as internal source and external source.

Internal source:
These many kinds of data are simple to find inside the organisation and include
things like market records, sales records, transaction records, customer data, various
(c

accounting resources and so on. Obtaining internal sources requires less time and
money than looking outside the company.

Amity Directorate of Distance & Online Education

 36 Data Engineering

External source:
Notes

e
The term “external source data” refers to the information that cannot be obtained
from within an organisation but may be obtained from resources provided by third parties

in
located outside of the organisation. Because it includes an extremely large quantity of
data, this requires a greater investment of both money and effort. The Registrar General
of India, the Planning Commission, the International Labour Bureau, Syndicate Services
and other non-governmental publications are all examples of external sources. Other

nl
examples of external sources include government publications, news publications and
other non-governmental publications.

O
●● Data Collection Methods
1. Surveys
Questionnaires, either paper or digital, are used in surveys to collect qualitative and
quantitative information from the people being surveyed. In order to get feedback

ty
from attendees after an event, one scenario in which you may conduct a survey is
described below. This can provide you a sense of what attendees loved, what they
wished was different and areas in which you can improve or save money during your

si
next event for an audience that is comparable to the one you just had.
Although participants can be given hard copies of surveys, the distribution of surveys
at a larger scale can be accomplished through the use of online surveys. They may
er
also be affordable; if you utilise a platform that is free to users, the cost of conducting
a survey may be nothing. If you have a certain demographic in mind that you’d want to
gather feedback from, it can be worthwhile to work with a market research company
to distribute your survey to people who fit that profile. When it comes to designing and
v
carrying out surveys, one thing to keep an eye out for is the influence of bias, which
might include the following factors:
ni

Something to watch out for when crafting and running surveys is the effect of bias,
including:
U

™™ Collection bias: It is very uncommon for researchers to unintentionally include

bias into the survey questions they design. Be wary of this while formulating
questions to guarantee that the people you’re interviewing will respond honestly
and won’t be affected by the way you phrase things.
ity

™™ Subject bias: Since your respondents are aware that you will read their
comments, it is possible that their responses will be skewed towards what
appears to be socially acceptable. Because of this, it is important to consider
combining the results of surveys with the information gained from other types of
m

data collecting in order to get a complete picture.

2. Transactional Tracking
You will have a better understanding of your client base and be able to make more
)A

informed judgements regarding targeted marketing campaigns if you keep track of the
data that comes from your customers’ purchases. This is a painless technique of data
collecting that may pay off in the form of consumer insights since it allows you to retain
data as soon as it is created by the platform. This is true of e-commerce and point-of-
sale systems.
(c

3. Interviews and Focus Groups

Face-to-face conversation with participants on a particular problem or topic is the basis

Amity Directorate of Distance & Online Education

 Data Engineering 37

of both interviews and focus groups as a research method. Focus groups often consist
Notes

e
of a number of participants, whereas interviews typically involve only one individual.
You can collect qualitative and quantitative data using one or all of these methods.

in
You may get input from members of your target audience on new product features by
conducting interviews and focus groups with those individuals. Observing how users
engage with your product in real time and capturing their replies to both open-ended

nl
and closed-ended questions can give insightful data on the aspects of your product to
prioritise developing further.
As is the case with surveys, you are free to ask your respondents whatever question

O
you choose about their ideas, motives and sentiments in relation to your brand or
product while using these other gathering techniques. In addition to this, it creates the
possibility of prejudice. Your goal should be to ask questions that don’t force them to
respond in a predetermined way.

ty
One of the potential drawbacks of holding focus groups and conducting interviews
is that they might take up a lot of time and money. It is a potentially time-consuming
endeavour if you want to carry them out on your own. You may steer clear of this

si
situation by working with a market research facilitator who will organise and carry out
interviews on your company’s behalf.
4. Observation
er
Because of the openness it provides, seeing users as they engage with your
website or product may be a valuable data collecting tool. You may see the effects
of any ambiguity or difficulty in the user interface immediately. However, scheduling
v
observations might be challenging. Users’ interactions with a beta version of your site
or product may be recorded and analysed with the help of an external technology.
ni

Observations allow you to witness firsthand how consumers engage with your product
or site, but they are less convenient than other data collecting methods. You may use
the qualitative and quantitative insights gained from this to strengthen the areas that
are working and strengthen the weaker areas.
U

5. Online Tracking
Pixels and cookies can be used to track user activity. Both of these programmes
monitor a user’s actions as they navigate different websites, revealing the kind of
ity

material that commonly attract and hold their attention. User activity on your company’s
website may be monitored to see things like which pages receive the most attention,
if any users get stuck and how long they spend on each product page. You may use
this information to refine the site’s layout and make it easier to use.
m

A pixel may usually be added for no cost and with little effort on your part. Although
there may be costs associated with using cookies, the improved information quality
may be well worth them. Pixels and cookies, once installed, collect information
)A

automatically and require minimal upkeep. There are ethical and legal considerations
related to tracking people’s internet activities that should not be overlooked. Make
sure you’re following local and industry data privacy rules before beginning to track
individuals’ internet activities.
6. Forms
(c

When you need to collect detailed information about a group of people, including their

Amity Directorate of Distance & Online Education

 38 Data Engineering

demographics or contact details, online forms are a great tool to employ. You may
Notes

e
use them to restrict access to material or collect signups for things like webinars and
email newsletters and they don’t cost anything to establish up. This information may
subsequently be used for remarketing purposes, such as email workflows and content

in
suggestions, as well as to get in touch with those who might be interested in your
offering.

nl
7. Social Media Monitoring
Checking the activity of your company’s followers on social media platforms is a
simple approach to gather insights on your target demographic’s preferences and

O
habits. Although there are analytics features in many social media sites, there are
also third-party systems that provide more comprehensive, streamlined information
culled from a variety of sources. Information gleaned from social media may help you
prioritise your time and energy on topics that truly matter to your audience. If your

ty
organisation regularly writes about its sustainability initiatives, you could find that the
number of engagements spikes when it does so.

1.7 Data Wrangling and Cleaning

si
Introduction

What is Data Cleaning?

er
When data are compiled from a variety of sources for the purpose of conducting an
analysis, there is always the possibility that the data will get tainted during the process
of integrating them. The act of removing inaccurate, damaged, incorrectly formatted,
v
duplicate, or missing data from previously gathered datasets is referred to as “data
cleaning,” and it is strongly advised that this step be taken.
ni

Due to the fact that the methods for cleaning data vary from dataset to dataset, there
is no universally applicable strategy for prescribing the specific steps that make up the
U

data cleaning process. Having said that, it is very necessary to design a pattern for the
process of data cleansing in order to ensure that it is carried out appropriately each and
every time.

What Is Data Wrangling?

ity

The act of integrating data from a variety of sources and cleaning it so that it can be
readily accessed and analysed is referred to as “data wrangling.” Another name for this
process is “data preparation.” The purpose of this procedure is to pick the information
that is the most pertinent and valuable, as well as analyse the data in order to obtain the
m

outputs that are the most desirable, dependable and relevant so that they may assist in
decision-making. These days, corporations are extraordinarily vast and as a result, they
produce enormous amounts of data inputs. As a consequence of this, data wrangling is
)A

becoming an increasingly critical step in the process of creating valuable data.

Objectives of Data Wrangling

1. Make the data you provide to business analysts as accurate and helpful as possible
and do it in a timely way.
(c

2. Decrease the amount of time spent on data collection and organisation.

3. Devote a greater portion of your time to the analysis of the data rather than the data
Amity Directorate of Distance & Online Education
 Data Engineering 39

processing.
Notes

e
4. Improve the quality of your judgements in a timely manner by basing them on the
facts.

in
1.7.1 Introduction to Data Wrangling
Data wrangling is the act of cleaning and merging disparate data sets so that they

nl
are more manageable and ready for analysis. The need to store and organise massive
amounts of data for analysis is growing as the number of data and data sources
continues to proliferate at an astounding rate. Data wrangling, often called data munging,

O
is the act of cleaning, changing and mapping data from one “raw” form to another so that
it may be better utilised in subsequent processes, such as analytics.
The term “data wrangling” refers to the steps used to prepare raw data for analysis
and speedy decision-making by analysts. Data wrangling, which is synonymous with data

ty
cleaning and data munging, helps organisations deal with more complicated data in less
time, with more reliable outcomes and more sound judgements. The specific procedures
rely on the information available and the desired outcome of the project. The use of data

si
wrangling tools is on the rise as businesses prepare their data for further analysis.

Data Wrangling Process er

Wrangling the data is both the first and the last phase in every data analysis project.
Data wrangling entails six crucial procedures before data can be analysed.
1. Data Discovery
v
The process of learning the significance of your data in general. At this early stage,
you are only getting to know your data.
ni

2. Data Organization/Structuring
At the outset, raw data is chaotic and difficult to make sense of. This information has
to be transformed so that it may be used by the company’s chosen analytical model.
U

3. Data Cleaning
The inconsistencies in raw data prevent it from progressing to the next phase. The
term “cleaning” refers to the process of removing anomalies, modifying, or otherwise
ity

improving data quality.

4. Data Enriching
You presumably know the data you’re dealing with a little better now. At this point, you
should evaluate whether or not you need to add anything else to the basic data.
m

5. Data Validating
Data quality issues are exposed at this stage, requiring corrective transformations
)A

to be implemented. In order to guarantee the accuracy and reliability of the data,

validation rules demand iterative programming processes.
6. Data Publishing
Finally, the result of all that data manipulation is sent further to be analysed.
(c

1.7.2 Tools for Data Wrangling

Data Wrangling Tools
Amity Directorate of Distance & Online Education
 40 Data Engineering

Wrangling data typically takes up the majority of a data analyst’s work, rather than
Notes

e
doing actual analysis of the data. Knowledge of a statistical language such as R or
Python, in addition to knowledge of SQL, Scala, PHP and other programming languages,
is typically required for employment as a data wrangler. R and Python are two examples

in
of statistical languages. In addition to these abilities, it is beneficial to have familiarity with
the tools that are typically utilised in the process of data wrangling. The following is a list
of tools that fall within this category.

nl
Tabula
Tabula is a programme that can extract data from files with [Link] extension. Tabula

O
offers a straightforward and user-friendly interface for the process of extracting data into
a spreadsheet formatted in either CSV or Microsoft Excel. Tabula is a cross-platform
application that supports Mac, Windows and Linux.

ty
Talend
Data wrangling, data preparation and data purification are the three primary
functions that Talend’s array of tools is designed to accomplish. It is a browser-based

si
platform that features an easy-to-use interface that just requires point-and-click actions,
making it perfect for companies. This makes data manipulation far less complicated than
it would be if it relied on programmers who relied heavily on code.
er
Parsehub
If you are unfamiliar with Python or are having difficulty using it, Parsehub is a
fantastic place to begin. Data may be scraped and extracted from a wide range of
v
interactive websites with the help of the user-friendly desktop interface provided by the
online scraping and data extraction application known as Parsehub. You don’t need
ni

to know any code to gather the data you want to collect; all you have to do is click on
the data you want to collect and then you can extract it into JSON format, an Excel
spreadsheet, or API forms. The availability of a graphical user interface on Parsehub is
U

the product’s most appealing feature to users who are just getting started with it.

Scrapy
The popular online scraping programme known as Scrapy may be more difficult to
ity

use than code-free competitors such as Parsehub, but Scrapy offers a far wider range
of customization options. It is an open-source web scraping framework that is built
using Python and may be used without cost at any time. Due to the fact that it is both
lightweight and scalable, Scrapy is ideally suited for a diverse variety of tasks.
m

1.7.3 Data Cleaning Techniques

Steps of Data Cleaning
)A

The specific methods for cleaning data that your firm employs will depend on the
specific sorts of data it holds, however you may clean your data by following these
general steps:
1. Remove duplicate or irrelevant observations
(c

Remove them from your dataset, regardless of whether they are duplicates or just
pointless. Duplicate observations are frequently made when acquiring data. When

Amity Directorate of Distance & Online Education

 Data Engineering 41

many data sets are joined, when data is scraped, when data is obtained from clients or
Notes

e
other departments, the likelihood of producing duplicate data grows. De-duplication is
among the most crucial things to take into consideration. You have made an irrelevant
observation when you find data that is unrelated to the subject at hand.

in
To better understand millennial consumers, for instance, you may choose to exclude
older demographics from your information. This can reduce the time spent for analysis,

nl
let you focus on what’s most important and provide a dataset that is easier to work with.
2. Fix structural errors
When doing a measurement or data transfer, if you detect any unusual naming

O
conventions, typos, or wrong capitalization, you have encountered a structural
mistake. These discrepancies may lead to incorrectly labelled groups. The “N/A” and
“Not Applicable” notations that may appear in any given sheet should be grouped
together for purposes of analysis.

ty
3. Filter unwanted outliers
Sometimes you’ll come across outliers that, at first glance, don’t seem to belong in
the data you’re looking at. The efficiency of your data will improve if you get rid of an

si
outlier if you have a good cause to do so, such as incorrect data input.
But occasionally an anomaly may show up to validate your idea. And the existence of
er
an extreme case doesn’t prove that the norm is wrong. To verify the accuracy of the
figure, this procedure is essential. You may want to get rid of an outlier if it turns out to
be a mistake or that it doesn’t add anything to your study.
4. Handle missing data
v
Many algorithms will not function well with lacking data. There are a few strategies for
ni

handling missing information. Although neither is ideal, they each have some merit,
including:
™™ You can remove observations that have missing values, but doing so will result
U

in a loss of information, so tread carefully.

™™ You may fill in missing numbers by extrapolating from previous observations;
however, this might compromise the data’s veracity because it relies on
assumptions rather than hard evidence.
ity

™™ You may need to adjust your approach to data usage if you want to successfully
deal with null values.
5. Validate and QA
m

Basic validation includes answering questions like these, which should be possible
when the data cleansing procedure is complete:
™™ Do the numbers add up?
)A

™™ Does the information conform to standards established in the field?

™™ Does it support, refute, or otherwise provide light on your current working
hypothesis?
™™ If you’re looking for support for your next theory, can you spot any patterns in
the data?
(c

If not, may it be due to poor data quality?

Amity Directorate of Distance & Online Education

 42 Data Engineering

Due to bad business decisions and strategies might be based on inaccurate or noisy
Notes

e
data. If your data doesn’t hold up to scrutiny, you might be in for an awkward moment
at your next reporting meeting. You can’t get there without first establishing a company-
wide commitment to data quality. You may accomplish this by writing out the methods you

in
could employ to develop such a plan of action.

Methods of Data Cleaning

nl
The data can be processed through a variety of data cleansing processes. Here are
the procedures:
1. Ignore the tuples: Ignoring the tuples is an option, but it’s impractical because it’s only

O
useful when the tuple contains several attributes that are all blank.
2. Fill the missing value: It is also possible that this approach will take a long time. To
complete the method, one must guess the missing value. In addition to the manual

ty
method, attribute mean and the most probable value may be used for this.
3. Binning method: The third strategy, “binning,” is easy to grasp and use. Data that has
been sorted is smoothed using neighbouring values. Next, the information is chopped

si
up into manageable chunks. The subsequent step is the actual implementation of the
various strategies.
4. Regression: Next is Regression, wherein the data is smoothed with the use of the
er
regression function, is employed. Linear or multiple regression may be used. Different
from multiple regression, linear regression only uses a single independent variable
5. Clustering: Clustering is approach number five since it focuses on the collective.
Data is clustered together for easier analysis. Then, clustering is used to identify the
v
outliers. A “cluster” or “group” is then created out of the related values.
ni

Process of Data Cleaning

The stages involved in data cleansing for data mining are outlined below.
U

1. Monitoring the errors: Make a mental note of the areas with the highest error rates.
It will facilitate the identification and correction of skewed data. Particularly crucial is
data during the integration of a potential replacement with the current management
software.
ity

2. Standardize the mining process: Standardising the point of insertion in the mining
process can help with this and lessen the likelihood of duplicates.
3. Validate data accuracy: Make sure the data is correct by checking it against other
sources and investing in data cleaning technologies. AI-powered tools provided a
m

more thorough check for accuracy.

4. Scrub for duplicate data: Separate data erasing technologies that can analyse rough
data in bulk and automate the procedure can help eliminate the need to again attempt
)A

the same data.

5. Research on data: We need to standardise, verify and remove duplicates from our data
before we can proceed with this step. Data from our databases can be accessed by a
wide variety of approved and authorised third-party sources. To ensure completeness,
correctness and dependability for business decision making, they assist us in cleaning
(c

and compiling the data.

6. Communicate with the team: Be sure to keep the staff updated on your progress; their

Amity Directorate of Distance & Online Education

 Data Engineering 43

input will be invaluable as you work to refine the client and send out more specific
Notes

e
information to prospective clients.

1.8 Statistical Analysis and Data Mining

in
Introduction
What is Data Mining?

nl
The process of extracting useful information, patterns and trends from huge data
sets in order to use the results to make data-driven choices is known as data mining. The
phrase “data mining” covers a wide range of techniques, such as social media mining,

O
text mining and web mining. Software that is either simple or complex can be used to do
data mining. Knowledge Discovery in Data, or KDD, is the term used to describe the data
mining process.

ty
What is Statistical analysis?
Statistical analysis is the process of gathering and analysing a lot of data to find
reoccurring tendencies and patterns. This method eliminates any chance of subjectivity

si
from the data review process by using numerical analysis. This approach has useful
applications in the compilation of research interpretations, the creation of statistical
models and the planning of studies and surveys.
er
1.8.1 Overview of Statistical Analysis
Acquiring data, interpreting data and validating data are all components of statistics;
v
statistical data analysis refers to the method of carrying out a variety of statistical
operations; more specifically, it refers to in-depth quantitative research that makes an
ni

effort to quantify data and makes use of many types of statistical analysis. In this context,
descriptive data like survey data and observational data are examples of the types of
quantitative data that are often included.
U

In AI and ML, statistical analysis is a scientific tool that aids in the collecting and
analysis of enormous amounts of data in order to find common patterns and trends and
transform the data into meaningful information. In plain English, statistical analysis is a
method for data analysis that facilitates the process of drawing meaningful conclusions
ity

from unstructured, raw data.

Statistical Analysis Types

The six different types of statistical analysis are listed below:
m

●● Descriptive Analysis
Descriptive statistical analysis is the process of acquiring, analysing, analysing and
summarising data in order to present it in the form of diagrams, graphs and tables.
)A

Instead of drawing any conclusions, it simplifies the challenging information to make it

simpler to read and understand.
● Inferential Analysis
Making meaningful conclusions from the data that has been examined is the aim of
(c

inferential statistical analysis. Either it explores the relationships between a variety of

different elements or it offers population-wide projections.

Amity Directorate of Distance & Online Education

 44 Data Engineering

● Predictive Analysis
Notes

e
Predictive statistical analysis is a branch of statistical analysis that looks at data to find
historical trends and make predictions about the future based on those patterns. It makes

in
use of a number of approaches, such as machine learning algorithms, data mining, data
modelling and artificial intelligence, to finish the statistical analysis of the data.
● Prescriptive Analysis

nl
The prescriptive analysis analyses the data and, depending on the research’s
conclusions, suggests the best course of action. It is a type of statistical analysis that
makes it easier to make an educated decision.

O
● Exploratory Data Analysis
Inferential analysis and exploratory analysis are quite similar. The main distinction
between the two is that exploratory analysis focuses on uncovering previously

ty
undiscovered data correlations. It performs an analysis of the possible connections
found within the data.
● Causal Analysis

si
The purpose of the causal statistical analysis is to establish the nature of the cause-
and-effect relationship that exists between the many variables that are contained
within the raw data. To put it another way, it identifies the reasons why something
er
occurs as well as the influence that it has on other factors. Businesses are able to
discover the reasons for their failures by utilising this process.

1.8.2 Introduction to Data Mining

v
The process of looking through large data sets for patterns and correlations that,
ni

when evaluated, might help with problem-solving in commercial companies is known as

data mining. Thanks to data mining approaches and technologies, businesses can now
predict future trends and make more educated business decisions.
U

Data mining is one of the core subfields of “data science,” which is the process
of obtaining useful information from data collections. These two concepts both relate
to the application of advanced analytics techniques. More specifically, data mining is a
step in the knowledge discovery in databases (KDD) procedure, which is a data science
ity

method for gathering, processing and analysing data. Although they are commonly
used interchangeably, data mining and knowledge discovery and data mining are two
distinct ideas.

Data mining process: How does it work?

m

Along with other certified specialists in the business intelligence and analytics
disciplines, data scientists frequently perform data mining. It may also be done by data-
savvy business analysts, executives and staff members who work for an organisation and
)A

act as citizen data scientists.

Its core components include machine learning, statistical analysis and data
management activities that are performed to prepare the data for analysis. Using
machine learning algorithms and other artificial intelligence (AI) technologies has made
(c

it much easier to mine large data sets, including as customer databases, transaction
logs and log files from web servers, mobile apps and sensors. Process automation has
increased as a result of this.
Amity Directorate of Distance & Online Education
 Data Engineering 45

Data mining may be broken down into four fundamental steps, which are as follows:
Notes

e
1. The gathering of information. The data are compiled once it has been established
which ones are relevant for an analytics application. The information may be kept in

in
a number of different source systems, a data warehouse, or a data lake, the latter
of which is more common in big data situations and contains both structured and
unstructured data. The use of data from outside sources is another option. A data

nl
scientist will typically transfer the data to a data lake to complete the process after it
has been gathered from its original location.
2. The data preparation process. To prepare the data for the mining step, a number

O
of tasks are completed at this stage. It starts with data exploration, profiling and
preprocessing and then continues on to data purification to fix errors and other issues
with the quality of the data. Data transformation is carried out to guarantee that all
of the data sets are same, unless a data scientist is expressly assigned to analyse

ty
unfiltered raw data for one of their projects.
3. The data mining procedure. A data scientist will choose the best data mining strategy
after the data has been prepared and then they will use one or more algorithms to

si
mine the data. The algorithms used in machine learning applications frequently need
to be trained on smaller sample data sets in order to find the information needed
before being applied to the entire collection of data.
er
4. Data interpretation and evaluation. In order to inform decision-making and other
facets of company operations, data mining findings are incorporated into analytical
models. The data scientist or another member of the data science team is in charge of
communicating the findings to company executives and users. Data visualisation and
v
approaches built on data storytelling are frequently used to achieve this.
ni

Types of Data Mining Techniques

Data science has several potential applications for the vast topic of data mining.
Pattern recognition and anomaly detection are two popular applications of data mining
U

that look for odd data pieces. Examples of well-known data mining techniques include the
following:
●● Association rule mining. Data mining uses association rules, which are if-then
ity

statements, to establish causation between bits of information. The frequency with

which an associated item appears in a data collection and the number of times
an if-then statement is true are used to measure the support and confidence of a
relationship, respectively.
●● Classification. According to criteria developed during data mining, data set
m

components are categorised using this approach. Decision trees, Naive

Bayes classifiers, k-nearest neighbour and logistic regression are examples of
classification algorithms.
)A

●● Clustering. In this case, data mining methods cluster the data depending on how
similar the individual data points are to one another. Clustering techniques include
k-means clustering, hierarchical clustering and the Gaussian mixture model.
●● Regression. Another method for finding links within datasets is to predict values in
(c

the dataset using a set of variables. Regression analysis may be divided into two
categories: linear and multivariate. Decision trees are only one of the categorization
methods that may be used to perform regressions.

Amity Directorate of Distance & Online Education

 46 Data Engineering

●● Sequence and path analysis. The use of route and sequence analysis. Information
Notes

e
may also be mined for patterns showing how one collection of events or values
predicts another.

in
●● Neural networks. Computer programmes called neural networks make an effort to
imitate how the human brain functions. Deep learning, a kind of machine learning,
uses neural networks to solve complex pattern recognition problems.

nl
1.9 Communicating and Sharing Data Analysis Findings
Introduction

O
We routinely take on projects that need us to synthesise big volumes of complicated
data (both quantitative and qualitative) and use it in marketing materials to effectively
deliver essential themes and motivate target audiences to take action. Although the data
types, subject areas and target audiences for each communication product are unique,

ty
we nonetheless use a consistent process for creating high-quality output.

1.9.1 Introduction to Data Visualisation

si
What is data visualization?
The practice of displaying data in a way that is visually appealing and makes it
er
easier to understand and extract insights is known as data visualisation. The main
goal of data visualisation is to make it easier to find anomalies and hidden correlations
in large datasets. Statistical graphics and information visualisation are frequently used
interchangeably when referring to information graphics.
v
Data must be visualised after it has been collected, handled and modelled in order
ni

to yield insights. One of the steps in the data science process is this. Data visualisation
is a component of Data Presentation Architecture (DPA), which aims to identify, change,
package and transfer data in the most efficient way feasible.
U

Learning to visualise data will be beneficial for almost every profession. It may be
used by teachers to display their students’ test results, by AI researchers to uncover
novel possibilities and by corporate executives to inform their stakeholders. Additionally,
it is essential for large data projects. During the early stages of the big data movement,
ity

when businesses were gathering vast amounts of data, they required a quick way to get
a bird’s-eye view of everything. Making use of a visualisation tool was a no-brainer.
For similar reasons, visualisation is essential for sophisticated analytics. In
order to keep track of progress and double-check that their models are functioning as
m

expected, data scientists who develop cutting-edge predictive analytics or ML algorithms

must resort to visualising their outputs. This is due to the fact that pictures are worth a
thousand numbers when it comes to understanding complicated algorithms.
)A
(c

Amity Directorate of Distance & Online Education

 Data Engineering 47

Notes

e
in
nl
O
ty
si
v er
ni
U

A timeline depicting the history of data visualization

ity

Source: [Link]

Why is data visualization important?

Data visualisation allows for the efficient and immediate transmission of information
via the use of standardised visual cues. In addition, the method may assist organisations
m

learn what influences customers’ buying decisions, where to focus their efforts, how to
make data more memorable to key stakeholders, what times and places to introduce new
items and how to estimate future sales volumes.
)A

●● The following are some additional advantages of data visualisation:

●● The speed with which one can learn new material, grow in understanding and act;
●● A deeper appreciation for the measures that must be done to advance the company;
●● A more engaging presentation that is based on easily digestible facts;
(c

●● A streamlined dissemination of data that facilitates more open communication;

●● Do away with the necessity for data scientists since the information they analyse is

Amity Directorate of Distance & Online Education

 48 Data Engineering

more easily understood; and

Notes

e
●● A heightened capacity to implement results rapidly, allowing for more rapid success
with fewer missteps.

in
1.9.2 Communicating and Sharing Data Analysis Results
Data analysts gather and sort information in order to provide organisations with

nl
answers to pressing concerns. Data analysts typically have to explain their conclusions
to others who lack expertise in technical analysis. For a non-technical audience that has
to comprehend the data to make business choices, effectively communicating these
results is crucial. There is also a wide variety in the information delivery requirements

O
across team members. Let’s learn five strategies for communicating effectively with
stakeholders, which data analysts may use to maximise the impact of their work.
1. Storytelling

ty
Data analysts may win over stakeholders with an engaging tale that provides a full
explanation of the findings thanks to an abundance of dashboards, spreadsheets
and charts. By providing context through narrative, business executives get a deeper
understanding of data. It distils information down to its essentials without distraction.

si
2. Data Visualization
Data visualisation is an essential ability for data analysts since the human brain
er
processes graphics 60,000 times quicker than text. The ability to visually comprehend
data aids in the speedy assimilation of insights and the generation of business choices.
In addition, it provides a captivating story that helps business executives better grasp
the facts.
v
3. Critical Thinking
ni

Data analysts must be able to spot patterns and trends and ask pertinent questions.
One of the most sought-after abilities for data analysts is critical thinking, even if it is
not a direct data competence. The ability to analyse and assess business challenges
from all sides is a key skill for data analysts.
U

4. Present Data for Impact

Data analysts work with enormous volumes of information that must be clearly
explained to stakeholders. When presenting data in a manner that is both enticing
ity

and fully explains the findings, it is essential to keep the target audience in mind. If
visuals are made utilising reports and data analysis, the presentation will be easier to
understand.
5. Clarity and Concision
m

Data analysts frequently share their conclusions with non-technical team members
and business executives. Being succinct and straightforward is essential when
presenting data. The main goal is to swiftly provide the results while avoiding providing
)A

stakeholders with unnecessary information.

For organisations globally, data analysts gather, purify and analyse vast amounts of
data. It is crucial that corporate stakeholders comprehend the data offered to guide
business choices. The various methods of communication we discussed above will
make it easier for data analysts to collaborate with team members and give a clear
(c

presentation of the findings.

Benefits of Data Visualization

Amity Directorate of Distance & Online Education

 Data Engineering 49

Data visualisation has several uses in a variety of fields, including business,

Notes

e
government, retail, education, recreation and more. The following are some benefits of
data visualisation:

in
●● Storytelling: Colours and patterns in clothes, artwork, culture, architecture and more
appeal to people. The story contained inside the data may also be visualised using
colours and patterns.

nl
●● Accessibility: Information is shared in a way that is understandable and accessible
to a range of audiences.
●● Visualise relationships: When information is displayed as a graph or chart, it is
simpler to identify the linkages and patterns within a data collection.

O
●● Exploration: With more data available, there are more options for exploration,
collaboration and the development of practical judgements.

Types of Data Visualization

ty
Data visualisation can be as simple as a bar graph or scatter plot, but it becomes
effective when, for instance, contrasting the median ages of Americans with members of
the United States Congress. Here are a few prevalent data visualisation examples:

si
●● Table: A table may be easily created in an Excel spreadsheet or Word document
and shows data in rows and columns. er
●● Chart or graph: Data is shown as a tabular representation along an x and y axis,
usually with bars, points, or lines to provide data comparison. An infographic is a
type of chart that employs both text and visuals to present information.
™™ Gantt chart: In project management, a Gantt chart is a bar chart that shows a
v
schedule and tasks.
™™ Pie chart: A pie chart displays percentages of data as “slices” of a pie, all of
ni

which add up to 100%.

●● Geospatial visualisation: In a choropleth or heat map, data is displayed on a map
using shapes and colours to highlight the relationship between certain locations.
U

●● Dashboard: Information is shown for business purposes to help analysts analyse

and convey data.
ity

1.10 Communicating and Sharing Data Analysis Results

Introduction
Data quality and data analysis techniques are essential skills for capacity building,
but they are not enough to achieve your goals. You also need to communicate
m

and present your data analysis results to stakeholders effectively, so that they can
understand, trust and act on your findings.
)A

1.10.1 Overview of Career Paths in Data Analysis

What Is a Data Analyst?
A data analyst is anybody who is paid to examine data and draw conclusions from
it that others may use. Data analysts that are adept at their craft are in high demand
(c

across the globe. Data analysts are in great demand, yet there is a dearth of qualified
candidates; as a result, even entry-level data analysts may expect to earn above-average
pay and benefits.

Amity Directorate of Distance & Online Education

 50 Data Engineering

There is a wide variety of businesses and sectors that employ data analysts.
Notes

e
Some of the most in-demand data analysts use their findings to decide how to allocate
resources, who to market to, what risks to take and what investments to make.

in
Data Analyst Qualifications

Education

nl
If you graduate with honours from a data analysis programme and rank towards the
top of your class, you should have no problem finding an entry-level employment in the
field. A degree in mathematics, statistics, or economics from a respectable university,
even if it is not specifically related to the field, will be sufficient to get entry.

O
In the United States, students can major or minor in data analytics or data science
at most universities. There are several options for advanced education in data science
beyond the bachelor’s degree. There are several options for quick and flexible skill

ty
development, including certification programmes and courses offered by a wide range of
universities.

What does a Data Analyst do?

si
When it comes to assisting upper management in making sound business choices,
data analysts play a crucial role.
er
Data Analyst roles and responsibilities include -
™™ Work with IT, management and/or data scientist teams to figure out what the
organization’s goals are.
v
™™ To gather information from first-hand and second-hand sources
™™ To clean the data and get rid of anything that doesn’t belong.
ni

™™ To use normal statistics methods to analyse data and explain the results
™™ To show how trends, relationships and patterns are changing in large, difficult
data sets
U

™™ To plan how to improve a process

™™ To make sure that management can see the facts clearly
™™ To plan, make and keep up-to-date databases and data systems that are
ity

important and useful.

™™ To put jobs in order of importance and fix code problems and other data-related
problems.
™™ To find trends and make predictions.
m

™™ To make screens, graphs and other ways to look at data.

™™ To make key success markers and keep track of them
)A

What are the problems on which Data Analysts work?

™™ Four different kinds of data research are part of a Data Analyst’s job.
™™ Descriptive analysis: an analysis of what happened based on facts from the
past.
(c

™™ Diagnostic analysis is the process of looking at facts to figure out why

something happened.

Amity Directorate of Distance & Online Education

 Data Engineering 51

™™ Predictive analysis is the process of looking at data to see what will happen in
Notes

e
the future (trends and changes).
™™ Prescriptive analysis: figuring out what should be done in the future.

in
Key Data Analyst Skills
Listed below are some of the important skills to become a data analyst -

nl
Technical Skills
●● High-level skills in analysing and fixing problems.
●● A basic understanding of numbers and maths.

O
●● Hands-on experience with MS Excel or another spreadsheet programme.
●● Know how to use statistical programming languages like R, SAS and SPSS.
●● Knowledge of a programming language, such as Python, Java, or Scala.

ty
●● Know how to handle databases and how to use search languages like SQL or
NoSQL.
●● Know how to use tools like Tableau, QlikView and Power BI to show how data looks.

si
●● Mining for data
●● Statistics with many factors er
●● Hadoop

Soft Skills
●● Good speaking and writing skills
v
●● Good Analytical thinking
●● Interpretive skill
ni

●● Team management

1.10.2 The Many Paths to Data Analysis

U

Let’s look more closely at four possible job routes in the world of data.

1. Data Scientist
ity

Many data scientists begin their careers as researchers. Usually, to make this
change, you have to:
™™ Getting better at programming
™™ Learning more complicated maths
m

™™ Learning about how machine learning works

Many data scientists also have a degree in data science, computer science, or a
similar area. Even if you don’t need a degree, having one can open up more work
)A

possibilities.

2. Management
Moving into managerial positions is another well-liked path for professional
advancement for data scientists. Starting as a data analyst and progressing to a senior
(c

analyst, analytics manager, head of analytics, or even chief data officer (CDO), are all
possibilities.

Amity Directorate of Distance & Online Education

 52 Data Engineering

You should practice both your data and leadership abilities if you decide to choose
Notes

e
this route. Only those with a master’s degree in data analytics or business management
with an emphasis on data analytics may be eligible for these higher-level positions in
some organisations.

in
3. Specialist
Data researchers are needed in many different industries. You might occasionally be

nl
able to learn more about the particulars of that industry through your employment.
●● Business analysts utilise data to increase the efficacy and efficiency of an
organization’s organisational structures, personnel development and IT operations.

O
●● Financial specialists utilise data to aid in decision-making regarding investments,
identify opportunities for profit and lower financial risk.
●● Operations analysts are responsible for enhancing a company’s productivity by

ty
identifying and resolving technological, procedural and structural issues.
●● Marketing analysts, also known as market research specialists, examine market
trends to determine what goods and services to offer, how much to charge for them
and which clients to target.

si
●● Systems analysts, Cost-benefit analysis is a tool that systems analysts use to assist
companies match technological solutions to their needs.
er
●● Health care analysts utilize data from patient surveys, cost reports and medical
records to assist doctors provide better treatment.

4. Consultant
v
Consider becoming a data analytics analyst once you’ve spent a few years
analysing data for a firm (or numerous organisations). You would work as a freelancer
ni

or for a consulting firm rather than an organisation directly. You would do research for
several different clients. When you work as a consultant, you frequently have greater
independence and the opportunity to conduct various types of research (particularly if you
work for yourself).
U

Summary
●● A data ecosystem is a group of business infrastructure and apps that are used to
ity

collect and analyse data. It helps businesses learn more about their customers and
come up with better strategies for marketing, selling and running their business.
●● Data Ecosystem is the name for the computer languages, packages, methods,
cloud-computing services and general technology that an organisation uses to
collect, store, analyse and use data.
m

●● Sensing is the process of finding sources of information for your job. It means
figuring out how good the data is so you can figure out if it’s useful.
●● Data can come from internal sources like databases, files, customer relationship
)A

management (CRM) systems and other tools. It can also come from outside
sources, like websites or companies that gather info from other sources.
●● Services make it possible to show off apps that are running in pods. Their job is to
stand in for a group of pods that all do the same thing and set the rules for getting to
(c

those pods.
●● The Namespace object’s job is to divide up the resources in the cluster. A single
cluster can have more than one area. This lets managers better organise the cluster

Amity Directorate of Distance & Online Education

 Data Engineering 53

and make it easier to share resources.

Notes

e
●● Deployments are controller objects that tell Kubernetes how to handle the pods
where a containerized program is running.

in
●● ReplicationControllers make sure that the cluster always has the right number of
pod copies going. When a user makes a ReplicationController, they tell it how many
pods they want.

nl
Glossary
●● Algorithms: Algorithms are a set of steps or rules that must be followed in order
to solve a problem. In this case, the problem is figuring out how to evaluate and

O
change data.
●● Kubernetes is an open-source Container Management tool that handles container
release, scaling, descaling and load sharing. It is also called a container
orchestration tool. It is written in the language Golang.

ty
●● Pods: These are the smallest units of release in Kubernetes are pods. They live on
nodes in the cluster and have their own IP addresses, which lets them talk to the
rest of the cluster.

si
●● ConfigMaps are objects in Kubernetes that are used to store key-value pairs of data
about how containers are set up. ConfigMaps make it possible to make pictures that
are lighter and easier to move around by separating setup data from the rest of the
container image.
er
●● Data analysis is the science of looking at data to draw conclusions that can be used
to make choices or learn more about different topics. It means putting data through
v
processes.
●● Text Analysis: Text analysis is the process of getting useful information from text
ni

data that is not organised in a certain way. It uses different methods, such as natural
language processing (NLP), text mining, sentiment analysis and subject modelling,
to find trends and insights in text data.
U

Check Your Understanding

1. What is the full form of EDA?
a) Exploratory Data Analysis Project
ity

b) Exploratory Data Analysis Project

c) Exploratory Division Analysis Project
d) Exploratory Data Analytical Project
2. ______________ is information that cannot be measured or counted numerically.
m

These data categories are organised by category, not by number.

a) Text Data b) Quantitative Data
)A

c) Nominal Data d) Qualitative Data

3. _________________ is used to designate variables with no numerical or ordinal
value. The colour of hair can be regarded nominal data because no two colours can
be compared.
a) Quantitative Data b) Nominal Data
(c

c) Qualitative Data d) Predictive Data

4. ___________________ can be conveyed in numerical values, allowing it to be

Amity Directorate of Distance & Online Education

 54 Data Engineering

counted and incorporating statistical data analysis.

Notes

e
a) Analytical Data b) Kubernetes
c) Quantitative data d) Load Balancing

in
5. _____________________ is the process of connecting and synchronising two data
sources. A modification to an element in a data set automatically modifies the bound
data set when data binding is used.

nl
a) Data Analysis b) Data binding
c) Data Cleaning d) Data Replication

O
6. What is the full form of HTML?
a) Hypertext Markup Language b) Hidden Markup Language
c) Hyper Markup Language d) Hi Text Markup Language

ty
7. What is the full form of API?
a) Automated Programming Interface b) Analyst Programming Interface
c) Application Programming Interface d) Algorithm Programming Interface

si
8. What is the full form of XML?
a) Eliminating Markup Language b) Extensible Markup Language
c) Expression Markup Language d) Expertise Markup Language
er
9. What is the full form of DOM?
a) Division Object Model b) Data Object Model
v
c) Document Object Model d) Discrete Object Model
10. ______________________ is the process of storing digital information using computer
ni

devices. storing data enables the expedient completion of numerous digital duties.
a) Data Binding b) Data storage
c) Data analysis d) Data cleansing
U

11. ___________________ is a device for storing data that is directly connected to a

computer or server. The storage connects internally or externally to a single device
and are easier to install and manage than network-based storage.
ity

a) Direct-attached storage b) Network-Attached Storage

c) Data preparation d) Master-detail formats
12. _____________________ is a dedicated storage server connected to devices over a
local area network (LAN).
m

a) Data preparation b) Direct-Attached Storage

c) Network-attached storage d) Microsoft’s Visual Studio
)A

13. ____________________ is a storage network that connects directly to servers. The

storage type transfers data between a host and storage using distinct networking
protocols, such as Fibre Channel.
a) Storage Area Network b) Network-Attached Storage
c) Direct-Attached Storage d) One-way binding
(c

14. ____________________ is essentially a programme that enables the creation,

deletion and modification of relational databases.

Amity Directorate of Distance & Online Education

 Data Engineering 55

a) Regional Database Management Systems

Notes

e
b) Relational Database Marginal Systems
c) Relational Database Management Systems

in
d) Relational Database Managerial Systems
15. A ___________________ is a centralised data repository that gives us the ability to
store all of our structured and unstructured data on a huge scale.

nl
a) Dynamic schema b) Tabular
c) data lake d) Graph Databases

O
16. ______________ refers to a collection of data that is not only enormous in amount but
also expanding at an exponential rate over time.
a) Big Data b) Data Mart

ty
c) Entity Integrity d) Key-value stores
17. _________________ is the study of how to programme computers to learn on their
own. The fields of machine learning, algorithm development and statistical modelling
all anticipate that a computer will carry out predetermined tasks without being given

si
any explicit instructions.
a) Artificial intelligence b) E-commerce
er
c) Batch Processing d) Data Lake
18. What is the full form of NLP?
a) Natural Language Processing b) New Language Processing
v
c) Neutral Language Processing d) Natural Language Partner
19. _________________ is responsible for the development of the HPCC big data tool. It
ni

does all of the data processing tasks using a single platform, a single architecture and
a single programming language.
a) Zoho Analytics b) Hadoop
U

c) LexisNexis Risk Solutions d) Cassandra

20. ________________ is a free and open-source compute system for large amounts
of data. It is one of the greatest tools for working with large data since it provides a
ity

distributed real-time processing system that is fault-tolerant.

a) Storm b) Cassandra
c) HPCC d) [Link]

Exercise
m

1. Explain Modern Data Ecosystem.

2. Explain the Key Players in the Data Ecosystem.
)A

3. What are Kubernetes Objects?

4. What is Data Analysis?
5. What are the roles and responsibilities of Data Analyst?
6. What is difference between Data Analytics and Data Analysis?
(c

7. Explain the concept of Data processing and data storage.

8. What is RDBMS?

Amity Directorate of Distance & Online Education

 56 Data Engineering

Learning Activities
Notes

e
1. Create Two Table and explain the concept of Primary key and Secondary Keys.
2. Explain how would you clean data and explain ways that you can use to visualise it.

in
Also, take a raw data and represent it any data visualisation technique of your choice.

Check Your Understanding - Answers

nl
1. a) 2. d) 3. b) 4. c)
5. b) 6. a) 7. c) 8. b)
9. c) 10. b) 11. a) 12. c)

O
13. a) 14. c) 15. c) 16. a)
17. a) 18. a) 19. c) 20. a)

Further Readings and Bibliography

ty
1. Joe Reis and Matt Housley. (2022). Fundamentals of Data Engineering: Plan
and Build Robust Data Systems. First Edition. O’Reilly Publications.
2. Andreas Kretz. (2019). The Data Engineering Cookbook: Mastering the

si
Plumbing of Data Science.
3. Holden Karau andy Konwinski, Patrick Wendell and Matei Zaharia. (2015).
er
Learning Spark. First Edition. O’Reilly Publications.
v
ni
U
ity
m
)A
(c

Amity Directorate of Distance & Online Education

 Data Engineering 57

Module - II: Relational Database Basics

Notes

e
Learning Objectives

in
At the end of this module, you will be able to:
●● Analyse introduction to relational databases

nl
●● Recognise understanding tables, rows and columns
●● Identify primary keys, foreign keys and relationships
●● Define data manipulation with SQL

O
●● Analyse querying data with SELECT, INSERT, UPDATE and DELETE
●● Analyse filtering and sorting data
●● Recognise overview of MySQL and PostgreSQL

ty
●● Identify installation and setup
●● Define working with databases in MySQL and PostgreSQL
●● Analyse designing a relational database schema

si
●● Analyse Entity-Relationship (ER) diagrams
●● Analyse creating and implementing the database er
Introduction
A relational database may be used to store and retrieve data sets that are related to
one another. A straightforward and fundamental method for tabularly expressing data is
v
the relational model, which serves as the foundation for relational databases. A relational
database table’s entry each have their own distinct identifier, or key.
ni

2.1 Relational Database Concepts

It is simple to ascertain how the data is connected since the attributes of the data
U

are contained in the table’s columns and each record normally provides a value for
each characteristic. Relational Database Management System is the abbreviation for
this. Modern database management systems, like Microsoft SQL Server, IBM DB2,
Oracle, MySQL and Microsoft Access, are all built on top of RDBMS. In 1970, E. F.
ity

Codd established the relational model, the foundation of which is the relational database
management system (RDBMS).
The ability for IT staff and others to build, modify, administer and generally interact
with a relational database is made possible by a system called a relational database
m

management system (RDBMS). Most commercial RDBMSs employ Structured Query

Language (SQL) to allow users to query stored data in tabular format. However, SQL
is optional for RDBMS implementation because it was developed after the relational
)A

concept was already established.

2.1.1 Introduction to Relational Databases

Companies all across the world use RDBMS more than any other database
management technology. It offers both high system speed and simple implementation,
(c

making it an ideal solution for storing and retrieving massive volumes of data.

RDBMS Fundamental Characteristics:

Amity Directorate of Distance & Online Education

 58 Data Engineering

●● Offers the option of preserving data in tables

Notes

e
●● It may be accessed by several users at once, all under the control of a single
person.

in
●● Digital tables may be used to preserve useful information.
●● Each row is uniquely identified using the primary key.
●● The data is consistently stored in rows and columns.

nl
●● Data retrieval uses indexes.
●● To make it easier for tables to share columns, keys are utilised.

O
Relational database management solutions imitate the following actions to guarantee
data consistency:
●● Entity Integrity: A database table’s entries cannot have the same data in two
different instances.

ty
●● Referential integrity makes sure that only those table entries may be deleted that
are not referenced by other tables. Otherwise, there can be discrepancies in the
statistics.

si
●● User-defined Integrity: The rules are set by the users, taking access and
confidentiality into account.
er
●● Domain integrity: Depending on default values, the kind of data, or ranges,
the columns of database tables are contained inside specific, predetermined
boundaries. This guarantees the reliability and accuracy of the data.

Characteristics
v
●● Data must be organised into rows and columns and kept in tabular form within a
ni

database file.
●● A record or tuple is a single row in a table. The number of such records collected is
referred to as the table’s cardinality.
U

●● Each table column is referred to as an attribute or field. The table’s arity is a

grouping of these columns.
●● There cannot be duplicate records in the DB table. Using a candidate key thereby
ity

prevents data duplication. A candidate key is the bare minimal collection of

characteristics needed to identify every record specifically.
●● Foreign keys are used to connect tables to one another.
●● Database tables also permit NULL values, meaning that if any of the elements’
m

values are blank or missing, the value is converted to a NULL value, which is not the
same as zero. IMPORTANT: A primary key cannot contain a NULL value.

2.1.2 Understanding Tables, Rows and Columns

)A

What is a table?
Tables are the data storage elements of a relational database management system.
The many columns and rows in this table represent a collection of data items about a
common topic. Keep in mind that a table is the most basic and ubiquitous data storage
(c

structure in any relational database. An example CUSTOMERS table with columns for
CustomerId, Name, Age, Salary, City and Country is shown below−

Amity Directorate of Distance & Online Education

 Data Engineering 59

ID Name Age Salary City Country

Notes

e
1 Ramesh 32 2000.00 Hyderabad India
2 Mukesh 40 5000.00 New York USA

in
3 Sumit 45 4500.00 Muscat Oman
4 Kaushik 25 2500.00 Kolkata India

nl
5 Hardik 29 3500.00 Bhopal India
6 Komal 38 3500.00 Saharanpur India
7 Ayush 25 3500.00 Delhi India

O
8 Javed 29 3700.00 Delhi India

What is a field?
Fields are the building blocks of every table. A field is a column in a database table
that stores a single piece of data for each entry. Fields in our CUSTOMERS table include

ty
ID, Name, Age, Salary, City and Country, among others.

What is a Record or a Row?

si
Each piece of information included in a database table is referred to as a record. The
aforementioned CUSTOMERS table, for instance, contains 7 entries. Here we have one
entry, or row, from the CUSTOMERS database − er
ID Name Age Salary City Country
1 Ramesh 32 2000.00 Hyderabad India

Each record in a table occupies the same vertical position.

v
What is a column?
ni

A column is an upright part of a table that holds all the information about a certain
field. In our CUSTOMERS table, for example, there are different entries for ID, Name,
Age, Salary, City and Country.
U

What is a NULL value?

A NULL value in a table is a value in a field that seems to be empty. This implies
that a field with the value NULL is empty. It is crucial to understand that a NULL value is
ity

distinct from a zero value or a field with blank spaces. A field that has the value “NULL”
was left empty when the record was created. The next table has three records. Salary for
the first record is “NULL,” and salary for the second record is “0.”

ID Name Age Salary City Country

m

1 Ramesh 32 Hyderabad India

2 Mukesh 40 00.00 New York USA
3 Sumit 45 4500.00 Muscat Oman
)A

2.1.3 Primary Keys, Foreign Keys and Relationships

Keys are one of the most important parts of a relational database because they keep
track of how the tables relate to each other and help identify each piece of data in a table.
The primary key is a key that helps identify each tuple in the database individually. The
(c

foreign key, on the other hand, is a key that shows the link between tables by using the
primary key of one table as a foreign key for another table. Now, let’s talk more about
both of them.
Amity Directorate of Distance & Online Education
 60 Data Engineering

What is Primary Key?

Notes

e
A main key is used to make sure that each piece of information in a certain area is
unique. A column can’t have a number of “NULL.” It is either a table column that already

in
exists or a column that the database creates based on a set order.
Example: For example, both STUD_NO and STUD_PHONE are possible keys for
the relation STUDENT, but only STUD_NO can be chosen as the main key. (only one out

nl
of many candidate keys).
Table Student

Stud_ Stud_name Stud_phone Stud_state Stud_count Stud_age

O
no
1 RAM 9865278251 Haryana India 20
2 RAM 9655470231 Punjab India 19

ty
3 SUJIT 7514290359 Rajasthan India 18
4 SURESH 8564103258 Punjab India 21

si
Table Student_course

Stud_no
er Course_no Course_name
1 C1 DBMS
2 C2 Computer Networks
1 C2 Computer Networks
v
What is Foreign Key?
A relational database table’s foreign key is a column or set of columns that creates
ni

a connection between the data in two tables. It is a column (or columns) that refers to
a column in another table, often the primary key. A foreign key is a column, or set of
columns, in a relational database table that provides a link between data stored in two
U

separate databases.
Example: STUD_NO in STUDENT_COURSE is a foreign key to STUD_NO in
STUDENT relation.
ity

Difference between Primary Key and Foreign Key

Primary Key Foreign Key

A primary key is utilised in order to verify that A foreign key is a column, or set of columns, in
each piece of data included inside a certain a relational database table that provides a link
m

column is distinct. between data stored in two separate databases.

It uniquely identifies relational database It refers to the field in a table that is the main key
records. of another table.
)A

A table can only have one main key. While a table can have more than one foreign key.
It is a mix of the requirements UNIQUE and It can have the same number more than once and
Not Null. a table in a SQL database.
It doesn’t accept NULL numbers. It can also have values that are NULL.
(c

Its value in the parent table can’t be removed. The value in the child’s database can be dropped.
It can be set up automatically on the temporary It can’t be set up on either the local or the global
tables. temporary tables.

Amity Directorate of Distance & Online Education

 Data Engineering 61

In DBMS, a link can be either “One-to-Many” or “Many-to-One.” Let’s figure out what
Notes

e
it means:
™™ One-to-One relationship

in
™™ One-to-Many relationship or Many-to-One relationship
™™ Many-to-Many relationship

One-to-One Relationship

nl
One-to-one refers to the relationship between a single record in Table A and a
single record in Table B. For instance, if there are two objects labelled “Person” and
“Aadhar card” (name, Aadhar number), these are both examples of “entities.” As a result,

O
each individual is only permitted to own one Aadhar card. This is how the one-to-one
connection in DBMS looks:

ty
si
er
Source: [Link]

For security purposes, this kind of communication is formed. The aadhar card number
may be placed in the same object (‘Person’), but because it can include personally
v
sensitive information, we segregated it out into a new database (‘Aadhar card’).
ni

One-to-Many Relationship
A sort of relationship between two database entities is a one-to-many relationship.
One instance of the first entity—the “parent” entity—can be associated to several
U

instances of the second entity—the “child” entity—in a one-to-many connection. For

instance, a consumer may place many orders, but only one customer may place a single
order.
ity

The traits of a one-to-many connection are as follows:

™™ The parent entity has a single primary key.
™™ The main key in the parent entity is referenced by a foreign key in the child
object.
m

™™ The child entity’s foreign key may have different values.

™™ The parent entity may have one or no offspring.
One popular method for representing relationships between things in a database
)A

is through one-to-many relationships. They may be used to depict a wide range of

interactions and are simple to comprehend and put into practise.

Examples of one-to-many connections include the following:

(c

™™ A consumer could place several orders.

™™ A book may include several pages.
™™ A course may have a large enrollment.
Amity Directorate of Distance & Online Education
 62 Data Engineering

™™ A corporation may employ a sizable workforce.

Notes

e
™™ A nation may contain several cities.
Different databases have different ways of implementing one-to-many relationships.

in
The fundamental ideas, nevertheless, remain the same. Each instance of the parent
object is uniquely identified using the main key of the parent entity. The main key in
the parent entity is referred to using the foreign key in the child object. As a result, the
database is able to trace the connections between the two entities.

nl
One-to-many relationships are an effective technique for database data modelling.
They are simple to comprehend and use, and they may be used to express a wide range
of relationships.

O
One row of the first table is connected to one or more rows of the second table in the
illustration below, however the rows of the second tables can only be related to the one
row in the first table. Another name for it is a many-to-one connection.

ty
si
v er
Source: [Link]
ni

Each entry in table A is linked to at least one record in table B if tables A and B
are related. There will, however, only be one link between the data in tables A and B.
The most popular kind of connection is this one. The term “many-to-one relationship” can
also be used to describe a one-to-many relationship in a database management system
U

(DBMS).
In a database management system (DBMS), a one-to-many relationship develops
when a single record in one table is connected to several entries in the other table. For
ity

instance, when it comes to the terms “Customer” and “Account,” only one Customer
may be the owner of a particular Account, albeit a single Customer may have several
Accounts.
The one-to-many connection may be modelled as follows in DBMS:
m
)A

Source: [Link]
(c

Many-to-Many Relationship
The two tables are said to have a many-to-many relationship if a single record

Amity Directorate of Distance & Online Education

 Data Engineering 63

in one table is linked to several entries in the other table and if a single record in the
Notes

e
second table is linked to numerous records in the first table. Consider Tables A and B as
examples. A many-to-many connection allows each entry in table A to be linked to several
entries in table B and vice versa. Additionally, an N-to-N link is depicted.

in
Consider the two tables as an example: a table for students and a table for lessons.
A student may enrol in one or more courses and each course has this option. An
excellent example of a many-to-many link is this one.

nl
The many-to-many connection may be shown as follows in DBMS:
The relationship is many to many from the perspective of table A.

O
ty
si
Source: [Link]

A many-to-many relationship from the perspective of table B.

er
v
ni

Source: [Link]
U

2.2 Using Relational Databases

Introduction
The best thing about a relational database is that you can link information from
ity

different tables to learn useful things. This method helps organisations of all kinds and
types figure out how different data sets from different areas relate to each other. The way
data is organised in a relational database can also help manage who has access to what.
Users only get relevant, pre-screened information because the connections between data
m

points are already set and require a unique ID to look up.

Benefits of Relational Databases

Here are some more reasons why relational systems are useful:
)A

●● Simple and centralized database: Relational databases are easy to use and can be
kept in one place. By switching between tables, you can get a lot of information that
can be used in many ways. Plus, ERP systems are built on relational databases,
which help users handle clients, goods and a lot more.
(c

●● Easy to use: To organise and manage massive volumes of data, many businesses
employ relational databases and ERP. Their ongoing usage encourages the
development of these technologies, such as the cloud migration. Users may
Amity Directorate of Distance & Online Education
 64 Data Engineering

simply search through data sets using SQL to find, filter and generate the
Notes

e
information they want.
●● Save time and money: Companies can stay organised and efficient when they

in
use relational systems. When tracking a customer’s sale or a museum guest, the
unique IDs help get rid of similar information. A relational database cuts down on
duplication, which saves workers time because they don’t have to spend time
entering customer data into logs. By putting that work somewhere else, a company

nl
can save money.

2.2.1 Data Manipulation with SQL

O
Structured Query Language (SQL), as we all know, is the language for databases.
With SQL, we can do certain things with a current database and we can also use it to
make a new one. SQL uses instructions like CREATE, DROP, INSERT, etc. to do the
things that need to be done.

ty
SQL statements are like telling a table what to do. It is used to talk to the database
and do some things with it. It can also be used to do certain jobs, processes and data
queries. SQL can do a lot of different things, like make a table, add data to it, drop it,

si
change it and set permissions for people.
Most of these SQL commands can be put into five main groups:
er
1. DDL – Data Definition Language
2. DQL – Data Query Language
3. DML – Data Manipulation Language
v
4. DCL – Data Control Language
5. TCL – Transaction Control Language
ni

DDL (Data Definition Language)

Data Definition Language, or DDL, is a set of SQL statements that may be used
U

to describe a database’s structure. It is used to develop and modify the structure of

database objects in the database and exclusively deals with specifics of the database
architecture. It is possible to create, modify and remove database structures using the
DDL set of SQL commands, but not data. These tools are rarely used by users. They
ity

should instead connect to the database via an application. The DDL instructions are as
follows:
™™ CREATE: This programme is used to create databases and the objects included
inside them, such as tables, indexes, functions, views, store procedures and
m

triggers.
™™ DROP: This function allows you to remove things from the database.
™™ ALTER: This is used to modify the database’s configuration.
)A

™™ TRUNCATE: This function deletes all of the records in a table together with the
spaces designated for them.
™™ COMMENT: This is how comments are added to the data dictionary.
™™ RENAME: This is used to change the name of an item in the database that
already exists.
(c

DQL (Data Query Language)

Amity Directorate of Distance & Online Education

 Data Engineering 65

With DQL statements, you can ask questions about the data in schema objects. The
Notes

e
DQL Command’s job is to get a model relationship based on the query that is given to it.
Here’s how we can describe DQL: It is a part of a SQL query that lets you get information
from the database and put it in order. The SELECT statement is part of it. With this function,

in
you can get info out of the database so you can work with it. When a SELECT command
is sent to a table or tables, the results are put into a temporary table. This table is then
presented or maybe sent to the programme, which is called a front-end. A list of DQL:

nl
●● SELECT: It gets information from the database.

DML (Data Manipulation Language)

O
List of commands in DML:
●● The INSERT: This command is used to add information to a table.
●● UPDATE: This is usually used to change data that is already in a table.

ty
●● DELETE: This command is used to delete data from a database table.
●● LOCK: Control interaction with the table.
●● CALL: Calls a subprogram written in PL/SQL or JAVA.

si
●● EXPLAIN PLAN: It tells you how to get to the facts.

DCL (Data Control Language) er

DCL features commands like GRANT and REVOKE that primarily deal with the
rights, permissions and other limitations of the database system.

Here are some DCL commands:

v
GRANT is a command that gives people permission to view the database.
ni

Syntax:
GRANT SELECT, UPDATE ON MY_TABLE TO SOME_USER, ANOTHER_USER;
REVOKE: This command takes away the entry rights that the GRANT command
U

gave to the person.

Syntax:
REVOKE SELECT, UPDATE ON MY_TABLE FROM USER1, USER2;
ity

TCL (Transaction Control Language)

Transactions put together a group of jobs so that they can be done all at once. Each
transaction starts with a certain job and finishes when all the tasks in the group are done
m

properly. If any of the steps don’t work, the deal doesn’t work. So, there are only two
possible outcomes of a transaction: success or loss. Here, you can read more about
deals. So, these TCL commands are used to control how a deal is carried out:
)A

BEGIN: Starts a new transaction.

COMMIT: Commits a Transaction.

Syntax:
COMMIT;
(c

ROLLBACK: Reverses a transaction if something goes wrong.

Amity Directorate of Distance & Online Education

 66 Data Engineering

Syntax:
Notes

e
ROLLBACK;
SAVEPOINT: Sets a saving point within a transaction.

in
Syntax:
SAVEPOINT SAVEPOINT_NAME;

nl
2.2.2 Querying Data with SELECT, INSERT, UPDATE and DELETE
DML Commands in SQL

O
Data Manipulation Language is what DML stands for. Structured Query Language’s
DML commands change the data in the SQL database. Using DML commands, we
can quickly view, store, change, update and delete the records that are already in the

ty
database.
Following are the four main DML commands in SQL:
1. SELECT Command

si
2. INSERT Command
3. UPDATE Command er
4. DELETE Command

SELECT DML Command

In Structured Query Language, the SELECT statement is the most important way
v
to change data. The SELECT function displays the table’s records. Using the WHERE
clause, it also shows the specific record of a certain field.
ni

Syntax of SELECT DML command

SELECT column_Name_1, column_Name_2, ….., column_Name_N FROM Name_
of_table;
U

Here, column_Name_1, column_Name_2, ….., column_Name_N are the names of

those columns whose data we want to retrieve from the table.
If we want to get the information from all of the table’s columns, we must use the
ity

following SELECT command:

SELECT * FROM table_name;

INSERT DML Command

m

Structured Query Language’s INSERT command is another important one for

manipulating data because it lets users add information to database tables.
Syntax of INSERT Command
)A

INSERT INTO TABLE_NAME ( column_Name1 , column_Name2 , column_Name3 ,

.... column_NameN ) VALUES (value_1, value_2, value_3, .... value_N ) ;

Examples of INSERT Command

Example 1: This illustration shows how to insert the record in the database table.
(c

Let’s look at the following table of students, which only has two records.

Amity Directorate of Distance & Online Education

 Data Engineering 67

Stu_Id Stu_Name Stu_Marks Stu_Age

Notes

e
101 Ramesh 92 20
201 Jatin 83 19

in
Let’s say you want to add a new record to the table of students. To do this, you need to write
the following DML INSERT command:

nl
INSERT INTO Student (Stu_id, Stu_Name, Stu_Marks, Stu_Age) VALUES (104,
Anmol, 89, 19);

UPDATE DML Command

O
Structured Query Language’s (SQL) UPDATE command is another one of its most
important data manipulation commands. It lets users update or change the data that is
already in database files.

ty
Syntax of UPDATE Command
UPDATE Table_name SET [column_name1= value_1, ….., column_nameN =
value_N] WHERE CONDITION;

si
Here, ‘UPDATE’, ‘SET’ and ‘WHERE’ are SQL keywords and ‘Table_name’ is the
name of the table whose values you want to change. er
Examples of the UPDATE command
Example 1: This illustration shows how to change the value of only one field.
Let’s look at a table called Product that has the following records:
v
Product_Id Product_Name Product_ Product_
Price Quantity
ni

P101 Chips 20 20
P102 Chocolates 60 40
U

P103 Maggi 75 5
P201 Biscuits 80 20
P203 Namkeen 40 50
ity

Let’s say you want to change the Product_Price of the item with Product_Id P102. To
do this, you need to write the following DML UPDATE command:
UPDATE Product SET Product_Price = 80 WHERE Product_Id = ‘P102’ ;

DELETE DML Command

m

SQL users have the ability to delete a single record or a set of records from the
database tables using the DELETE DML function. The data is partially removed from the
database by this Data Manipulation Language order. By combining the WHERE clause
)A

with the DELETE statement, we may select which table rows to remove.
Syntax of DELETE Command
DELETE FROM Table_Name WHERE condition;
(c

Examples of DELETE Command

Example 1: This example shows how to delete a single item from a table.

Amity Directorate of Distance & Online Education

 68 Data Engineering

Let’s look at a table called Product that has the following records:
Notes

e
Product_Id Product_Name Product_ Product_
Price Quantity

in
P101 Chips 20 20
P102 Chocolates 60 40
P103 Maggi 75 5

nl
P201 Biscuits 80 20
P203 Namkeen 40 50

O
Let’s say you wish to get rid of the item with the Product_Id of P203 from the Product
database. This requires the following DML DELETE command to be entered:
DELETE FROM Product WHERE Product_Id = ‘P202’ ;

ty
Example 2: In this second example, we’ll see how to remove numerous rows or
records at once from a database table.
Consider a table called Students with the following entries:

si
Stu_Id Stu_Name Stu_Marks Stu_Age
101 Ramesh er 92 20
201 Jatin 83 19
202 Anuj 85 19
203 Monty 95 21
v
102 Saket 65 21
103 Sumit 78 19
ni

104 Ashish 98 20

Let’s say you decide you no longer want any information on pupils whose Marks are
U

higher than 70. The necessary DML Update command is as follows:

DELETE FROM Student WHERE Stu_Marks > 70;

2.2.3 Filtering and Sorting Data

ity

SQL’s built-in sort and filter functions are crucial for working with massive datasets.
Filtering allows you to pull out particular information from your data based on certain
criteria, while sorting helps you to organise your data in a specified order, such as
alphabetically or numerically.
m

The ORDER BY clause in SQL is used to order data by a column or columns of your
choosing. The WHERE clause is used to filter the results by specifying the criteria that a
row must fulfil in order to be returned.
)A

Together, sorting and filtering allow you to construct highly effective queries that
may mine your data for the precise answers you need. You may narrow your search
results even more by utilising sophisticated tactics like multiple criteria, wildcards and
subqueries. Now we’ll talk about how to sort and filter data in SQL, covering topics like
(c

the ORDER BY and WHERE clauses, wildcards in the WHERE clause and subqueries
for more intricate filtering.

Amity Directorate of Distance & Online Education

 Data Engineering 69

Sorting Data in SQL

Notes

e
Using SQL, you may sort rows of data into a particular order according to the
values of one or more columns. The ORDER BY clause is the data-sorting part of a SQL

in
statement. The SELECT statement’s ORDER BY clause allows you to order the results
by a specified column or columns.
The ORDER BY clause’s fundamental syntax looks like this:

nl
SELECT column1, column2, ...
FROM table_name

O
ORDER BY column1 [ASC|DESC], column2 [ASC|DESC], ...;
After the ORDER BY keyword, you may name the columns that will be used to
sort the data. Ascending (ASC) is the default sort order, although descending (DESC)
is an option. The following SQL query, for instance, applies an ascending sort to the

ty
“employees” table based on the “last_name” column:
SELECT first_name, last_name, salary
FROM employees

si
ORDER BY last_name;
By adding additional columns after the first column, separated by commas, you may
er
sort the data based on several columns. Data in the “employees” database, for instance,
may be sorted using the following SQL statement, which first sorts the “last_name”
column in ascending order and then the “first_name” column:
v
SELECT first_name, last_name, salary
FROM employees
ni

ORDER BY last_name, first_name;

Remember that case matters when sorting; if you need to ignore case when sorting,
use the LOWER() or UPPER() method to turn the column into a case-insensitive sorting
U

field. In this topic, you learned the fundamentals of using SQL to sort data. You now know
how to use the ORDER BY clause to sort records by one or more columns, in either
ascending or descending order. Now that you know how SQL works, you can use it to
organise data and pull the information you need.
ity

Filtering Data in SQL

Using SQL, you may filter data to get records from a table that meet specified
criteria. The WHERE clause is the SQL instruction used to filter information. In SELECT,
m

UPDATE and DELETE statements, the WHERE clause filters the rows that are modified,
updated, or deleted.
WHERE clauses follows the basic syntax mentioned below:
)A

SELECT column1, column2, ...

FROM table_name
WHERE condition;
The WHERE keyword is used to restrict the number of returned rows based on the
(c

condition that follows it. The condition can be a logical operator like AND, OR, or NOT, or
a comparison operator like =, >, >,, >=, =, BETWEEN, LIKE, or IN.

Amity Directorate of Distance & Online Education

 70 Data Engineering

For example, To retrieve all entries from the “employees” database where the
Notes

e
“salary” column value is more than $50,000, you may use the following SQL:
SELECT first_name, last_name, salary

in
FROM employees
WHERE salary > 50000;

nl
The WHERE clause supports numerous criteria with the use of logical operators
like AND and OR. When applied to the “employees” database, the following SQL query
retrieves all entries where the “first_name” column value is “John” and the “salary”
column value is more than 50,000.

O
SELECT first_name, last_name, salary
FROM employees
WHERE first_name = ‘John’ AND salary > 50000;

ty
The IN and BETWEEN operators can also be used to refine search results. To
retrieve all rows from the “employees” table where the “salary” column value is between
40,000 and 60,000, you may use the following SQL:

si
SELECT first_name, last_name, salary
FROM employees
er
WHERE salary BETWEEN 40000 AND 60000;
The LIKE operator is another useful tool for pattern-based data filtering. For
instance, the following SQL query retrieves all entries from the “employees” table where
v
the “first_name” field contains the letter ‘J’.
SELECT first_name, last_name, salary
ni

FROM employees
WHERE first_name LIKE ‘J%’;
U

2.3 MySQL and PostgreSQL

Introduction
ity

●● MySQL is a user-supported database management system (DBMS), while

PostgreSQL is an Object Relational Database Management System (ORDBMS).
●● While MySQL only supports JSON, current application capabilities such as those
found in PostgreSQL are supported.
m

●● When comparing PostgreSQL vs MySQL, you’ll find that PostgreSQL excels

at running sophisticated queries whereas MySQL shines in online analytical
processing and online transaction processing (OLAP) environments.
)A

●● When compared to MySQL, ACID compliance is limited to the InnoDB and NDB
storage engines in PostgreSQL.
●● MySQL lacks support for Materialised Views, PostgreSQL has it built right in.

2.3.1 Overview of MySQL and PostgreSQL

(c

A database is a compiled set of data that is organised for simple access,

maintenance and modification. Many different Database Management Systems (DBMS)

Amity Directorate of Distance & Online Education

 Data Engineering 71

have been developed to facilitate this process. The likes of MySQL, PostgreSQL,
Notes

e
MongoDB, Redis, etc., fall under this category.

MySQL

in
It’s a well-known RDMS (Relational Database Management System) and it’s free
and open-source. The CRUD (Create, Read, Update, Delete) actions on the data in this
table are rather straightforward. MySQL has a number of features, including:

nl
™™ Quick, simple and trustworthy.
™™ Adaptable to projects big and small.

O
™™ Allows for easy expansion

PostgreSQL
It’s a robust Object-relational database that’s freely available to the public. Because

ty
of its excellent stability, it delivers solid performance with little upkeep requirements.
PostgreSQL pioneered the use of MVCC, or multi-version concurrency control, in
database management systems. PostgreSQL’s many features include:
™™ It works with a wide variety of languages.

si
™™ It has state-of-the-art safety functions.
™™ It allows for geotagging. er
Features of MySQL
●● In-Memory Database Management System (DBMS) driven by the community,
MySQL
v
●● All major platforms, languages and middleware are supported.
●● Multi-Version Concurrency Control is supported.
ni

●● Standardised on the ANSI SQL language

●● SSL replication with logs and triggers
U

●● Supports objects and ANSI-SQL2008

●● Layered architecture with decoupled components
●● Completely threaded, with support for Kernel Threads
ity

●● The server may function as either an embedded DB or a client server.

●● Provides in-built capabilities for examination of both queries and spatial relationships
●● Data sets with 50 million rows or more are no problem for it.
●● MySQL is compatible with a wide range of UNIX distributions, in addition to other
m

operating systems including Windows and OS/2.

Features of PostgreSQL
●● An active neighbourhood that’s rapidly expanding
)A

●● Oracle, DB2 and SQL Server Alternative Popularity

●● Compatible with any of the most popular operating systems
●● Multiple, simultaneous users are supported by MVCC.
●● Large-scale indexing for efficient data reporting
(c

●● Help for cutting-edge formats (like XML and JSON)

●● With ANSI SQL, your knowledge and programmes are easily transferable.

Amity Directorate of Distance & Online Education

 72 Data Engineering

●● Data efficiency thanks to support for foreign keys

Notes

e
●● Joining tables and creating displays for versatile data retrieval
●● Complex programmes and transactions can benefit from triggers and stored

in
procedures.
●● Data backup and scalable reading through replication

nl
2.3.2 Installation and Setup
How to install MySQL

O
Among the many relational database management systems in use today, MySQL
is at the top of the list. It supports multiple users and several different types of storage
engines. The Oracle Corporation supports it. This section will teach you the fundamentals
of downloading and setting up MySQL.

ty
Prerequisites
MySQL requires the following components to function properly on your system:

si
™™ Installing MySQL Server
™™ [Link] Framework 4.5.2 by Microsoft
™™ Visual Studio 2019 Visual C++ Redistributable by Microsoft
er
™™ Minimum 4GB of RAM (recommended 6GB)
Download MySQL
v
Follow these steps:
Step 1: The first thing you need to do is head over to MySQL’s site and grab a copy
ni

of the community server edition. Select an OS like Windows from the drop-down menu
here.
Step 2: The second step is to choose one of two different download locations. Select
U

the desired version of the MySQL community server. Select the mysql-installer-web-
community if you have reliable internet service. In that case, go with the alternative.
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 73

Installing MySQL on Windows

Notes

e
Step 1: After downloading the setup, step one is to unzip it to a convenient location
and run the MSI installer. This is the screen that appears:

in
nl
O
Source: [Link]

ty
Step 2: Select the desired setup method in the subsequent wizard. MySQL product
and features installation requires selecting the correct kind from a list of available options.
We’re going to choose “Full” and move forward from here.

si
v er
ni
U
ity
m

Source: [Link]

MySQL Server, MySQL Shell, MySQL Router, MySQL Workbench, MySQL

Connectors, documentation, samples and examples and much more will all be installed if
)A

you choose this option.

Step 3: We will click the Next button and perhaps see a warning about features that
won’t install because your computer doesn’t meet the minimum specifications. We can
either click the Execute button, which will automatically install all prerequisites, to fix
(c

them, or we may ignore them. Proceed by selecting the Next option.

Amity Directorate of Distance & Online Education

 74 Data Engineering

Notes

e
in
nl
O
ty
si
er
Source: [Link]

Step 4: In the subsequent wizard, we will be presented with a dialogue box asking
for our approval to skip the installation of some goods. We must now select the “Yes”
v
option.
ni
U
ity

Source: [Link]

The goods that will be installed will be displayed after the Yes button is clicked. So,
m

hit the Execute button if we require all products.

)A
(c

Amity Directorate of Distance & Online Education

 Data Engineering 75

Notes

e
in
nl
O
ty
si
Source: [Link]
er
Step 5: The fifth step is to click the Execute button, at which point all the goods will
be downloaded and installed. When you’ve finished setting up, proceed by clicking the
Next button.
v
ni
U
ity
m
)A

Source: [Link]

Step 6: Now we will set up the MySQL server and router in the subsequent wizard.
(c

In this topic, the Router will not be setup because MySQL does not need its use. Only the
server setup will be covered in this topic. Proceed by selecting the Next option.

Amity Directorate of Distance & Online Education

 76 Data Engineering

Notes

e
in
nl
O
ty
si
Source: [Link]
er
Step 7: The screen below will appear once you click the Next button. The MySQL
Server has to be set up now. Click Next after selecting Standalone MySQL Server/Classic
MySQL Replication. In this section, you may also select the InnoDB Cluster that best
v
suits your requirements.
ni
U
ity
m
)A

Source: [Link]

Step 8: In this step, the system will prompt you to select the Config Type and other
(c

connectivity preferences on the following page. Config Type: “Development Machine,”

Connectivity: “TCP/IP,” Port Number: “3306,” and Next.

Amity Directorate of Distance & Online Education

 Data Engineering 77

Notes

e
in
nl
O
ty
si
er
Source: [Link]

Step 9: Select an Authentication Method and proceed to Step 9. In this case, I’m
going with the first choice.
v
ni
U
ity
m
)A

Source: [Link]
(c

Step 10: On the next screen, you must enter the MySQL Root Password. Select the
Next button once you’ve entered the correct password.

Amity Directorate of Distance & Online Education

 78 Data Engineering

Notes

e
in
nl
O
ty
si
Source: [Link]
er
Step 11: In this step, you will be prompted to set up the Windows Service that will
launch the server. Do nothing and proceed with the default settings by clicking Next.
v
ni
U
ity
m
)A

Source: [Link]

Step 12: After this, you’ll be prompted to apply the Server Configuration in a
(c

subsequent step. To proceed with this setup, please press the Execute button.

Amity Directorate of Distance & Online Education

 Data Engineering 79

Notes

e
in
nl
O
ty
si
Source: [Link]
er
Step 13: The following window will appear once setup is complete (Step 13). The
next step is to hit the Finish button.
v
ni
U
ity
m
)A

Source: [Link]

Step 14: Product Configuration is complete after Step 14, which will be displayed on
(c

the following screen. Leave everything as is and choose Next > Finish to finish installing
the MySQL package.

Amity Directorate of Distance & Online Education

 80 Data Engineering

Notes

e
in
nl
O
ty
si
er
Source: [Link]

Step 15: In Step 15, we’ll setup the Router using the subsequent wizard. To move
further, select Next > Finish and then click Next.
v
ni
U
ity
m
)A

Source: [Link]
(c

Step 16: The Connect to Server option will be available in the following wizard at
Step 16. Here, we must refer to the root password that was established before.

Amity Directorate of Distance & Online Education

 Data Engineering 81

Notes

e
in
nl
O
ty
si
er
Source: [Link]

In this window, you must also click the Check button to see if the connection was
successful. If the connection is established, you can proceed by selecting the Execute
v
button. The setup is finished; go to the next step by clicking Next.
Step 17: In the subsequent wizard, choose the appropriate settings and hit the Run
ni

button.
U
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 82 Data Engineering

Step 18: After we finish the preceding step 18, we’ll see this screen. A Finish button
Notes

e
has appeared; please click it.

in
nl
O
ty
si
er
Source: [Link]
v
Step 19: At this point, MySQL should be fully installed. Just hit the “Finish” button.
ni
U
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 83

2.3.3 Working with Databases in MySQL and PostgreSQL

Notes

e
Creating databases, defining tables, adding, updating, and querying data are all
standard actions when working with databases in MySQL and PostgreSQL. However, the two

in
database management systems (DBMS) also differ in terms of syntax and functionalities. A
brief description of these common tasks in both MySQL and PostgreSQL below.

1. Connecting to the Database:

nl
MySQL: To connect to a MySQL database using the command-line client, you
can use the following command, replacing username and password with your actual
credentials:

O
mysql -u username -p
PostgreSQL: To connect to a PostgreSQL database using the command-line client,
use the following command, replacing username with your actual username and dbname

ty
with your database name:
psql -U username -d dbname

2. Creating a Database:

si
MySQL: To create a new database in MySQL, you can use the following SQL
command: er
CREATE DATABASE dbname;
PostgreSQL: To create a new database in PostgreSQL, use this SQL command:
CREATE DATABASE dbname;
v
3. Selecting a Database:
ni

MySQL: To select a specific database for your session in MySQL, use the following
command:
USE dbname;
U

PostgreSQL: In PostgreSQL, you connect to a specific database when you log in.
There’s no separate USE command as in MySQL.

4. Creating Tables:
ity

MySQL: To create a table in MySQL, you can use the following SQL command:
CREATE TABLE tablename (
column1 datatype,
m

column2 datatype, ...

);
)A

PostgreSQL: To create a table in PostgreSQL, you can use the following SQL
command:
CREATE TABLE tablename (
column1 datatype,
(c

column2 datatype, ...

);

Amity Directorate of Distance & Online Education

 84 Data Engineering

5. Inserting Data:
Notes

e
MySQL: To insert data into a MySQL table, you can use the INSERT INTO
statement:

in
INSERT INTO tablename (column1, column2, ...)
VALUES (value1, value2, ...);

nl
PostgreSQL: To insert data into a PostgreSQL table, you can use the INSERT INTO
statement similarly to MySQL:
INSERT INTO tablename (column1, column2, ...)

O
VALUES (value1, value2, ...);

5. Inserting Data:
MySQL: To insert data into a MySQL table, you can use the INSERT INTO

ty
statement:
INSERT INTO tablename (column1, column2, ...)
VALUES (value1, value2, ...);

si
PostgreSQL: To insert data into a PostgreSQL table, you can use the INSERT INTO
statement similarly to MySQL: er
INSERT INTO tablename (column1, column2, ...)
VALUES (value1, value2, ...);

6. Querying Data:
v
MySQL: To retrieve data from a MySQL table, you can use the SELECT statement:
ni

SELECT column1, column2, ...

FROM tablename
U

WHERE condition;
PostgreSQL: To retrieve data from a PostgreSQL table, you can use the SELECT
statement just like in MySQL:
SELECT column1, column2, ...
ity

FROM tablename
WHERE condition;

7. Updating and Deleting Data:

m

Both MySQL and PostgreSQL support UPDATE and DELETE statements to modify
or remove records in a table. The syntax for these statements is similar in both DBMS.
)A

[Link]. MySQL PostgreSQL

1. It is the Database that most people use. The Database is the most advanced one.
2. It is a relational-based DBMS. It is an object-based relational DBMS
3. It only works with InnoDB and NDB cluster It was made to work with ACID from the
engines to be ACID-compliant. ground up.
(c

4. C/C++ is the language used to make it work. The language used to do the work is C.

Amity Directorate of Distance & Online Education

 Data Engineering 85

5. This supports the CASCADE option. The CASCADE choice is available.

Notes

e
6. The GUI tool provided is MySQL Workbench. PgAdmin is provided
7. It does not work with expression, partial, or It does work with expression, partial, or

in
image IDs. image IDs.
8. Materialised views and Table relations are PostgreSQL has both of these things.
not supported.

nl
9. SQL only works with Standard types of data. It works with advanced data types like
collections, hstore and user-defined kinds.
10. In InnoDB, SQL has restricted support for Full support for MVCC.

O
MVCC.
11. It was made by a Swedish company called It was made by the Department of Computer
MySQL AB in 1995. Science at the University of California.
12. It works well, is easy and goes faster. It goes slowly and more complex.

ty
13. Troubleshooting MySQL is easy. It is difficult to troubleshoot PostgreSQL.
14. MySQL is licensed beneath GNU GPU. PostgreSQL is licensed beneath MIT style.
15. It works best for easy tasks like reading and It is often used for tasks that are big and

si
writing. complicated.
16. Every link that is made in MySQL is a Every link made in PostgreSQL is a separate
separate OS thread. OS process.
er
PostgreSQL advantages over MySQL
Some reasons why PostgreSQL is better than MySQL are listed below:
v
●● PostgreSQL is an open-source database that is very safe, independent and full of
features.
ni

●● PostgreSQL allows a lot of advanced data types, such as multi-dimensional groups,

user-defined types, etc.
●● PostgreSQL does a good job of following SQL standards and can handle
U

“advanced” SQL things like window functions and common table expressions.
●● It lets you improve speed in the same ways that Oracle and SQL Server do.
●● Companies like Amazon, Redshift, Instagram and many more use PostgreSQL.
ity

MySQL advantages over PostgreSQL

Here are some reasons why MySQL is a better choice than PostgreSQL:
●● MySQL is a more popular database than PostgreSQL.
m

●● MySQL makes it easy to get help from the community and look for answers than
PostgreSQL.
●● MySQL does replication well, while PostgreSQL does not.
)A

●● Companies like Facebook, Twitter, Pinterest and many more use MySQL.

2.4 Database Design Project

Introduction
(c

Database design is a group of jobs or processes that help with designing,

developing, implementing and maintaining a business data management system. When

Amity Directorate of Distance & Online Education

 86 Data Engineering

you design a database well, the cost of upkeep goes down, which makes the data more
Notes

e
consistent. Disc storage room is a big factor in how cost-effective the design is. Because
of this, there must be a great way to build a database. The creator should stick to the
rules and figure out how the parts work together and what kind of data needs to be kept.

in
The main goals of database design are to make models of the suggested database
system’s physical and mental design. To explain this further, the logical model is mostly
focused on the needs of the data and the considerations must be made in terms of

nl
monolithic considerations. This means that the physical data that is saved must be kept
regardless of the physical conditions. On the other hand, the physical database design
model translates the logical database design model by keeping control of the actual

O
media using hardware tools and software systems like Database Management System
(DBMS).

2.4.1 Designing a Relational Database Schema

ty
Steps in the process of making a database schema
You can make a Database schema by doing any of the following:

si
1) Define the Objective
The first step in creating a database is to figure out what you want it to do. Before
making the database, you have to find out what the business needs. By taking a few
er
examples of questions and the results you want from them, you may be able to figure out
what kind of information you need to store in the database.
After you’ve collected the data, you’ll need to separate it into different tables and give
v
each column a unique name.

2) Find the Primary Keys and Organize the Tables

ni
U
ity

In the relational model, a table can’t have duplicate rows because that would cause
problems with queries. To keep things honest, every table should have a primary key field
that identifies all of the items in the table. From the picture above, you can see that the
Primary Key in the grocery list is the column grocery_id. This is because each grocery
item has a unique id.
m

3) Finding Relationships Between Tables

™™ One-to-one
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 87

Notes

e
in
nl
O
ty
Source: [Link]

si
A one-to-one relationship means that only one row of one table is linked to one
row of the other table and no more than that. For example, a buying list item can link
to a grocery item information table that has more information about the thing. If this
er
information isn’t available for a certain food item, it won’t be shown. The standard column
connects the tables above and grocery_id is the main key for both tables.
™™ One-to-Many
v
ni
U

Source: [Link]
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 88 Data Engineering

With a one-to-many relationship, one row in one table can link to more than one
Notes

e
row in another table. A one-to-many link lets you keep data you use often in the parent
database and refer to it more than once in the child table. The foreign key in the child
table points back to the main key in the parent table. For example, the above table has

in
an authorsList and a booksList and the foreign key is author_id.
™™ Many-to-Many

nl
O
ty
si
er
Source: [Link]

Through a many-to-many connection, zero, one, or multiple rows in one table may
be related to one or more rows in another table. To construct this relation, a bridge,
v
intermediary, or connector table is needed.

4) Refine the Design Using Normalization Techniques

ni

By eliminating duplicates, ensuring that the information in each column is the

same and combining the categories, the design may be improved. Data cleaning by
normalisation has various advantages. The following normalisation techniques are used
U

for refining:
●● First normal form (1NF): Each column in your table should contain exactly one
number and have the same value. Each column’s data must be of the same
ity

kind and have a unique name to prevent misunderstandings. One of 1NF’s key
advantages is that it prevents groups from appearing more than once in a table.
●● Second normal form (2NF): The concept of full functional dependency serves as the
foundation for the Second Normal Form (2NF). When a relationship’s main key is
composed of two or more qualities, the second normal form is utilised. A connection
m

must be in the first normal form and not have any ties in the second normal form.
Each column that isn’t a key in a database that is in the second normal form is
entirely dependent on the primary key.
)A

●● Third normal form (3NF): If a connection is in the First and Second Normal Forms
and no non-primary-key feature implicitly depends on the primary key, it is in the Third
Normal Form. A table is 3NF if its non-key entries are independent of one another.

Best Practices for Database Schema Design

(c

For database schema design to work well, it’s important to stick to best practices.
This gives writers a clear point of reference for tables and fields in a project. Here are a
few of the best ways to do things:
Amity Directorate of Distance & Online Education
 Data Engineering 89

Naming Conventions
Notes

e
●● Define and use the right name rules for your database schema designs to make
them work better. Even if you choose a certain style or follow an ISO standard, the

in
most important thing is that your name areas are always the same.
●● Try not to use protected words in table names, column names, field names, etc., as
this will likely cause a syntax mistake.

nl
●● Don’t use special characters, hyphens, quotes, or spaces. They will either need
more work or won’t be correct.
●● When naming tables, use single, not multiple, nouns. For example, use

O
StudentName instead of StudentNames. Since a table is a group of things, the term
doesn’t need to be in the plural.
●● Remove words from table names that aren’t needed. For example, use Department
instead of DepartmentList or TableDepartments.

ty
Security
●● A good database schema design is the first step in making sure data is safe. Use
encryption for private data like passwords and information that can be used to

si
identify a person. Don’t give every user master privileges; instead, ask users to sign
in before they can view the database. er
Documentation
●● Database designs are helpful even after you’ve made them and many people will
look at them. So it’s important to have good documents. Write out clear rules for
your database schema design and write comment lines for scripts, triggers and
v
other tasks.
ni

Normalization
●● Normalisation makes sure that things and connections that are not related to
each other are not put in the same table. This reduces redundancy and improves
U

the integrity of the data. Use normalisation as needed to improve the speed of a
database. Both too much and too little normalisation can cause trouble.

Expertise
ity

●● You can make the best schema design if you know your data and what each
element’s properties are. With a well-designed schema, your data can grow faster
and faster. As you keep adding more data, look at how each field fits in with the
other fields you’re adding to your schema.
m

2.4.2 Entity-Relationship (ER) Diagrams

The Entity Relational Model is a way to figure out what kinds of things will be in
the database and how those things are connected to each other. The ER data model
)A

describes enterprise schema, which is a pictorial representation of how a database’s

logical organisation works as a whole.
The Entity Relationship Diagram shows how the things in the database are linked
to each other. ER models are used to show how things in the real world, like a person, a
(c

car, or a company, relate to each other. In short, an ER diagram shows how the database
is set up.

Amity Directorate of Distance & Online Education

 90 Data Engineering

Why Use ER Diagrams In DBMS?

Notes

e
™™ ER diagrams are used to show the E-R model in a database. This makes it
easy to turn them into relations (tables).

in
™™ ER diagrams can be used to describe things in the real world, which makes
them very useful.
™™ You don’t need to know anything about computers or tools to use an ER

nl
graphic.
™™ These images are easy to understand and easy to make, even for people who
don’t know much about computers.

O
™™ It gives a standard way to show the facts in a sensible way.

Symbols Used in ER Model

ER describe is used to describe the system’s logical view from a data point of view. It

ty
is made up of these symbols:
●● Rectangles: In the ER Model, rectangles stand for “entities.”
●● Ellipses: In the ER Model, ellipses stand for Attributes.

si
●● Diamond: Diamonds stand for the connections between things.
●● Lines: Lines show the characteristics of entities and entity sets that have other types
of relationships.
er
●● Double Ellipse: A double ellipse means that the attribute has more than one value.
●● Double Rectangle: The double rectangle is a sign of a weak entity.
v
Components of ER Diagram
Entities, Attributes and Relationships between Entities in a Database System are
ni

what make up the ER Model.

Entity
U

An Entity can be a real object like a person, car, house, or employee, or it can be a
mental object like a company, job, or college course.
Entity Set: An Entity is an item of the Entity Type and an entity set is a group of all
entities. For example, E1 is an object with the object Type Student and the group of all
ity

students is called an Entity Set. Entity Type is shown in an ER diagram as:

Entity Set
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 91

1. Strong Entity
Notes

e
Entities that have a key attribute are called Strong Entities. Strong Entity doesn’t
depend on any other Entity in the Schema. It has a main key that makes it easy to

in
find and a square is used to represent it. These are what are known as Strong Entity
Types.
2. Weak Entity

nl
The key property of an Entity type is what makes it possible to identify each entity in
the entity set by itself. But there are some types of entities where key characteristics
can’t be set. These are called types of Weak Entities. For example, a company might

O
keep records about an employee’s parents, children, or spouse. But the kids wouldn’t
have been there if the job hadn’t been there. So Dependent will be a Weak Entity Type
and Employee will be the Identifying Entity Type for Dependent, making Employee a
Strong Entity Type.

ty
A Double Rectangle is used to show that an object type is weak. Weak object types
always take part in everything. The connection between a weak entity type and the
strong entity type that identifies it is called a “identifying relationship,” and it is shown by a
double diamond.

si
v er
Strong Entity and Weak Entity
Source: [Link]
ni

Attributes
Attributes are the type of entity’s traits. For example, the elements that make up
U

object type Student are Roll_No, Name, DOB, Age, Address and Mobile_No. An oval
stands for the characteristic in an ER picture.
ity

Attribute
Source: [Link]
m

1. Key Attribute
The key attribute is the part of each entity in the entity set that makes it unique.
For example, each student will have a different Roll_No. In an ER diagram, the most
)A

important characteristic is shown by a circle with lines going through it.

Key Attribute
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 92 Data Engineering

2. Composite Attribute
Notes

e
A combined attribute is an attribute that is made up of a lot of other characteristics.
For example, the Street, City, State and Country are all parts of the Address property of

in
the student Entity type. In an ER diagram, a composite characteristic is shown as an oval
that is made up of other ovals.

nl
O
ty
Composite Attribute
Source: [Link]

3. Multivalued Attribute

si
An attribute for a certain object that has more than one number. For example,
Phone_No (there can be more than one for a given student). In an ER diagram, a
property with more than one number is shown by a double oval.
v er
Multivalued Attribute
Source: [Link]
ni

4. Derived Attribute
A “derived attribute” is an attribute that can be made from other attributes of the
U

same object type. e.g., DOB can be used to figure out age. In an ER diagram, the derived
characteristic is shown by a circle with lines through it.
ity

Derived Attribute
Source: [Link]
m

The full Entity Type Student with all of its properties can be shown as:
)A
(c

Amity Directorate of Distance & Online Education

 Data Engineering 93

Notes

e
in
nl
O
ty
Entity and Attributes

si
Source: [Link]

2.4.3 Creating and Implementing the Database

er
Create Database
A database is a set of organised pieces of information that are kept on a computer
v
system. They are used to store and find info quickly and easily. Different query languages
can be used to make databases and SQL is one of them.
ni

The CREATE DATABASE statement is a DDL (Data Definition Language) statement

used to make a new database in SQL. If you create your database on Linux or Unix, the
case of the name matters, even though SQL keywords don’t care about case. This limit
U

does not apply if you are working on Windows.

Syntax
Here is the SQL code for Creating a database:−
ity

CREATE DATABASE DatabaseName;

Here, the “DatabaseName” is the name of the database that you want to create. The
database name can be any valid identifier, such as number, letters, or underscores.
m

Note − The “DatabaseName” cannot be a keyword in SQL.

When creating a database, you might get a warning like warning 1044 (42000):
Access refused for user “krishna”@”localhost” to database “DatabaseName.” This means
)A

that you do not have the right permissions to make a database.

Example
Let’s try to use the CREATE DATABASE line in SQL to make a database called
<testDB>.
(c

SQL> CREATE DATABASE testDB

Note: Before you create a database, make sure you have the right permissions.

Amity Directorate of Distance & Online Education

 94 Data Engineering

Once the database <testDB> is made, we can find it in the list of databases by doing the
Notes

e
following:
SQL> SELECT * FROM [Link]

in
+--------------------+
| Database |

nl
+--------------------+
| master |
| tempdb |

O
| model |
| msdb |
| testDB |

ty
+--------------------+
5 rows in set (0.00 sec)

si
By using the USE statement in SQL, we can now make testDB> the default
database−
SQL> USE testDB;
er
Database changed
That’s all. We were able to make a database in SQL. We can now make tables and
other types of database items in the new database.
v
Database Design Phase: Implementation
ni

During the Implementation phase, the DBMS is installed on the necessary hardware,
the database is optimised to run best on that hardware and software platform and
the database and its data are created and loaded. The first set of data could be new
U

information that was directly entered or old information that was brought in from a
MariaDB database or another DBMS. In this step, you also set up protection for the
database and give the different users you’ve found the access they need. At the end of
this time, you also start making backup plans.
ity

The following are steps in the implementation phase:

1. Make sure the DBMS is installed.
2. Adjust the variables of the configuration in accordance with the hardware,
m

software and use conditions.

3. Establish the database and table structures.
4. Start by loading the data.
)A

5. Configure the users and the security settings.

6. Implement the backup regime.

Summary
●● Relational Database Management System is the abbreviation for this. Modern
(c

database management systems, like Microsoft SQL Server, IBM DB2, Oracle,
MySQL and Microsoft Access, are all built on top of RDBMS.

Amity Directorate of Distance & Online Education

 Data Engineering 95

●● In 1970, E. F. Codd established the relational model, the foundation of which is the
Notes

e
relational database management system (RDBMS).
●● A main key is used to make sure that each piece of information in a certain area

in
is unique. A column can’t have a number of “NULL.” It is either a table column that
already exists or a column that the database creates based on a set order.
●● DCL has commands like GRANT and REVOKE that mostly deal with the database

nl
system’s rights, permissions and other limits.
●● Data Manipulation Language is what DML stands for. Structured Query Language’s
DML commands change the data in the SQL database. Using DML commands, we

O
can quickly view, store, change, update and delete the records that are already in
the database.
●● With the DELETE DML function, SQL users can remove a single record or a group
of records from the database tables. This Data Manipulation Language order does

ty
not fully remove the data from the database.
●● The ORDER BY clause in SQL is used to order data by a column or columns of your
choosing. The WHERE clause is used to filter the results by specifying the criteria

si
that a row must fulfil in order to be returned.
●● MySQL is a user-supported database management system (DBMS), while
PostgreSQL is an Object Relational Database Management System (ORDBMS).
er
Glossary
●● Entity Integrity: No two rows in a database table can contain identical information.
v
●● Referential integrity ensures that only rows from tables that are not referenced by
other tables may be erased. Inconsistencies in the data might result otherwise.
ni

●● User-defined Integrity: Rules established by the users, with consideration given to

access and confidentiality.
●● Domain integrity: The columns of the database tables are contained within certain
U

specified limitations, depending on default values, the kind of data, or ranges. This
ensures that the data is accurate and reliable.
●● Tables are the data storage elements of a relational database management system.
The many columns and rows in this table represent a collection of data items about
ity

a common topic.
●● Fields are the building blocks of every table. A field is a column in a database table
that stores a single piece of data for each entry. Fields in our CUSTOMERS table
include ID, Name, Age, Salary, City and Country, among others.
m

●● A foreign key is a column or group of columns in a relational database table that

provides a link between data in two tables. It is a column (or columns) that
references a column (most often the primary key) of another table.
)A

●● One-to-one means that a single record in Table A is linked to a single record in Table
B and vice versa.
●● Structured Query Language (SQL), as we all know, is the language for databases.
With SQL, we can do certain things with a current database and we can also use it
(c

to make a new one. SQL uses instructions like CREATE, DROP, INSERT, etc. to do
the things that need to be done.

Amity Directorate of Distance & Online Education

 96 Data Engineering

Check Your Understanding

Notes

e
1. A ________________ is an attribute that is made up of a lot of other characteristics.
For example, the Street, City, State and Country are all parts of the Address property

in
of the student Entity type.
a) Combined Attribute
b) Derived Attribute

nl
c) Simple Attribute
d) Composite Attribute

O
2. A ____________________ is an attribute that can be made from other attributes of
the same object type.
a) Multi-Valued Attribute
b) Nominal Attribute

ty
c) Derived attribute
d) Simple Attribute

si
3. The _________________ is a tool for handling and controlling SQL server instances
and databases. It lets database managers, developers and users connect to SQL
Server instances and do things like create, change and delete tables.
er
a) Postgre SQL
b) SQL Server Management Studio
c) MongoDB
v
d) Oracle Database
4. With the _________________ DML function, SQL users can remove a single record
ni

or a group of records from the database tables.

a) INSERT
U

b) CREATE
c) DELETE
d) SELECT
ity

5. What is the full form of DCL?

a) Data Call Language
b) Deemed Control Language
c) Data Control Language
m

d) Division Control Language

6. What is the full form of TCL?
)A

a) Transaction Command Language

b) Typed Control Language
c) Transaction Control Language
d) Transmission Control Language
(c

7. The ________________ command is used to add information to a table.

a) DELETE

Amity Directorate of Distance & Online Education

 Data Engineering 97

b) INSERT
Notes

e
c) SELECT
d) CALL

in
8. _______________ is used to change data that is already in a table.
a) UPDATE

nl
b) DELETE
c) SELECT
d) REVOKE

O
9. What is the full form of DQL?
a) Data Query Language
b) Demand Query Language

ty
c) Deep Query Language
d) Data Question Language
10. What is the full form of DDL?

si
a) Data Definition Language
b) Define Definition Language er
c) Divide Definition Language
d) Data Demand Language
11. What is the full form of DML?
v
a) Data Manipulation Language
ni

b) Data Mining Language

c) Define Mining Language
d) Demand Manipulation Language
U

12. A ________________ is a column or group of columns in a relational database

table that provides a link between data in two tables. It is a column (or columns) that
references a column (most often the primary key) of another table.
ity

a) Foreign key
b) Composite Key
c) Primary Key
d) Entity Key
m

13. ___________________ are the building blocks of every table. A field is a column in a
database table that stores a single piece of data for each entry.
a) Rows
)A

b) Fields
c) Columns
d) Blocks
14. ____________________ is used to change the name of an item in the database that
(c

already exists.
a) TRUNCATE

Amity Directorate of Distance & Online Education

 98 Data Engineering

b) SELECT
Notes

e
c) RENAME
d) UPDATE

in
15. What is the full form of SQL?
a) Simple Query Language

nl
b) Structured Query Language
c) Select Query Language
d) Select Question Language

O
16. What is the full form of RDBMS?
a) Rational database management system
b) Relational database manipulating system

ty
c) Relational database management system
d) Regional database management system
17. ___________________ ensures that only rows from tables that are not referenced by

si
other tables may be erased. Inconsistencies in the data might result otherwise.
a) User-defined Integrity er
b) Domain Integrity
c) Referential integrity
d) Entity Integrity
v
18. A ________________ is an upright part of a table that holds all the information about
a certain field. In our CUSTOMERS table, for example, there are different entries for
ni

ID, Name, Age, Salary, City and Country.

a) Column
b) Row
U

c) Field
d) Block
19. What is the full form of DBMS?
ity

a) Data Management System

b) Database management system
c) Data Managerial System
m

d) Database Manipulation System

20. ________________ is used to get rid of all the records in a table, along with the spots
that were set aside for the records.
)A

a) TRUNCATE
b) COMMEBT
c) DROP
d) ALTER
(c

Exercise
1. What is the difference between Primary key and Foreign Key?
Amity Directorate of Distance & Online Education
 Data Engineering 99

2. What is RDBMS?
Notes

e
3. What is Deep Learning?
4. How to design Relational Database Schema?

in
5. Explain Data Manipulation with SQL.
6. How to query data with INSERT and UPDATE FUNCTION?

nl
7. How do you Filter and Sort data in SQL?
8. What is the difference between MySQL and PostgreSQL?

Learning Activities

O
1. Write a SQL statement to create a simple table countries including columns country_
id,country_name and region_id.
2. Create a table employee including First_name, Last_Name employee_ID, Manager_

ty
ID, Hire_Date, and Salary. Insert values. Update First name name to Sakshi and Last
name to Arora where employee ID is 7054.
3. Create table named “Café Day” of a confectionery including category_ID. Category_

si
Name and Item_Description. Then show results for Drop table and Truncate table
commands.

Check Your Understanding – Answers

er
1. a) 2. c)
3. b) 4. c)
v
5. c) 6. c)
7. b) 8. a)
ni

9. a) 10. a)
11. a) 12. a)
13. b) 14. c)
U

15. b) 16. c)
17. c) 18. a)
19. b) 20. a)
ity

Further Readings and Bibliography

1.. Joe Reis and Matt Housley. (2022). Fundamentals of Data Engineering: Plan
and Build Robust Data Systems. First Edition. O’Reilly Publications.
m

2. Andreas Kretz. (2019). The Data Engineering Cookbook: Mastering the

Plumbing of Data Science.
3. Holden Karau andy Konwinski, Patrick Wendell and Matei Zaharia. (2015).
)A

Learning Spark. First Edition. O’Reilly Publications.

(c

Amity Directorate of Distance & Online Education

 100 Data Engineering

Module - III: SQL for Data science

Notes

e
Learning Objectives

in
At the end of this module, you will be able to:
●● Know databases and database management systems (DBMS)

nl
●● Learn SQL and its importance in data manipulation
●● Understand the SELECT, COUNT, DISTINCT, LIMIT, INSERT, UPDATE and
DELETE statements

O
●● Recognise
●● Identify working with string patterns and ranges in SQL
●● Analyse sorting result sets with ORDER BY and grouping result sets with GROUP

ty
BY
●● Understand relational database concepts and database instance on Cloud
●● Analyse getting started with the Db2 console and discuss types of SQL statements

si
●● Explain CREATE TABLE statement, ALTER, DROP and TRUNCATE tables
●● Summarise built-in database functions and time built-in functions in SQL
●● Recognise databases using Python and DB-API
●●
er
Learn creating tables, loading and querying data from databases in Python

Introduction
v
In the broadest sense, a database is a set of data that is organised in a certain way.
A database is more specifically a computer system that makes it easy to view, change
ni

and add to data. SQL is a language for working with databases. It can be used to create
databases, delete databases, get data rows, change and delete data rows and more.
Structured Query Language, or SQL, is a computer language used to store, change and
U

get information from a relational database.

3.1 Introduction to Databases and SQL

ity

Database
In other words, a database is a way for a company to store, organise and find
information electronically. The database is one of the most important parts of corporate
IT. Its ability to organise, process and handle information in a structured and controlled
way is the key to many aspects of modern business efficiency.
m

SQL
SQL was created by IBM computer scientists in the 1970s. It became a standard for
)A

both the American National Standards Institute (ANSI) and the International Organisation
for Standardisation (ISO) in 1986 and 1987, respectively.
SQL is the standard language that Relational Database Systems use to talk to
each other. SQL is the Standard Database Language used by all Relational Database
(c

Management Systems (RDMS) like MySQL, MS Access, Oracle, Sybase, Informix,

Postgres and SQL Server.

Amity Directorate of Distance & Online Education

 Data Engineering 101

3.1.1 Overview of Databases and Database Management Systems

(DBMS) Notes

e
Database Management System

in
●● A database management system is a piece of software used to run databases. For
example, MySQL, Oracle and other business databases are very common and are
used in many different ways.

nl
●● DBMS gives you a way to do things like create a database, store data in it, change
data, make a table in the database and a lot more.
●● It makes sure the information is safe and secure. It also keeps info the same when

O
there are different users.

DBMS allows users the following tasks:

™™ Data Definition: It is used to add, change, or remove definitions that describe

ty
how the data in the database is organised.
™™ Data Updating: This is used to add, change, or get rid of real data in the
database.

si
™™ Data Retrieval: It is used to get data from the database so that applications can
use it in different ways.
™™ User Administration: This is used to register and keep track of users, maintain
er
data integrity, enforce data security, deal with process control, keep track of
performance and recover information that was damaged by an unexpected
failure.
v
Characteristics of DBMS
ni

●● It stores and manages the information in a digital library that is set up on a

computer.
●● It can show how data is changed in a way that is clear and makes sense.
U

●● DBMS has backup and restore processes that run on their own.
●● It has ACID features that keep data in a good state in case something goes wrong.
●● It can make the connections between facts easier to understand.
ity

●● It is used to help with data editing and processing.

●● It is used to make sure that info is safe.
●● It can show the information from different points of view, depending on what the user
wants.
m

Advantages of DBMS
●● Controls redundancy in the database: It can control redundancy in the database
because it saves all the data in a single database file and puts the recorded data in
)A

the database.
●● Sharing data: With DBMS, the people who are allowed to use an organization’s data
can share it with other people who are allowed to use it.
●● Easy to Maintain: Because the computer system is centralised, it can be easy to
(c

maintain.
●● Save time: It cuts down on the time needed for creation and upkeep.

Amity Directorate of Distance & Online Education

 102 Data Engineering

●● Backup: It has backup and recovery modules that automatically back up data in
Notes

e
case of hardware or software failure and can retrieve the data if needed.
●● Multiple user interface. For example, it has graphics user interfaces and application

in
programme interfaces.

Disadvantages of DBMS
●● Costs of Hardware and Software: To run DBMS software, you need a fast processor

nl
and a lot of memory.
●● Size: It takes up a lot of room on discs and needs a lot of memory to run well.
●● Complexity: Using a database system adds to the complexity and needs.

O
●● Higher impact of failure: Failure has a bigger effect on the database because most
organisations store all of their data in a single database. If the database is damaged
by a power outage or a mistake in the database, the data may be lost forever.

ty
3.1.2 Introduction to SQL and Its Importance in Data Manipulation
Structured Query Language (SQL) is a standard computer language used to handle

si
relational databases and do different things with the data in them. SQL was made in the
1970s and is used by database managers, developers who write tools for integrating data
and data analysts who want to set up and run analytical queries. SQL helps control the
er
information that is saved in databases, so users can get the exact information they need
when they need it.
SQL is a powerful computer language that is easy to use. In fact, SQL can add
data to database tables, change data in current database tables and delete data from
v
SQL database tables. SQL can also change the layout of the database by adding,
changing and removing tables and other database items. SQL lets you change the data
ni

in a database by using a set of commands. SQL INSERT lets you add information to
database tables, SQL SELECT lets you get information from database tables and SQL
UPDATE lets you change current database records.
U

SQL is a data management language (DML) that is used to search for and change
database information. Data Manipulation Language is what DML is short for. Structured
Query Language’s DML commands change the data in the SQL database. Using DML
ity

commands, we can quickly view, store, change, update and delete the records that are
already in the database.
Here are SQL’s four most important DML commands:
1. SELECT Command
m

2. INSERT Command
3. UPDATE Command
4. DELETE Command
)A

1) SELECT DML Command

In Structured Query Language, the SELECT command is the most important way
to change data. The SELECT command displays the table’s records. Using the WHERE
clause, it also shows the specific record of a certain field.
(c

Syntax of SELECT DML command

Amity Directorate of Distance & Online Education

 Data Engineering 103

SELECT column_Name_1, column_Name_2, ….., column_Name_N FROM Name_

Notes

e
of_table;
Here, column_Name_1, column_Name_2, ….., column_Name_N are the names of

in
those columns whose data we want to retrieve from the table.
If we want to get the information from all of the table’s columns, we must use the
following SELECT command:

nl
SELECT * FROM table_name;

2) INSERT DML Command

O
Structured Query Language’s INSERT command is another important one for
manipulating data because it lets users add information to database tables.
Syntax of INSERT Command

ty
INSERT INTO TABLE_NAME ( column_Name1 , column_Name2 , column_Name3 ,
.... column_NameN ) VALUES (value_1, value_2, value_3, .... value_N ) ;

3) UPDATE DML Command

si
Structured Query Language’s (SQL) UPDATE command is another one of its most
important data manipulation commands. It lets users update or change the data that is
already in database files.
Syntax of UPDATE Command
er
UPDATE Table_name SET [column_name1= value_1, ….., column_nameN =
value_N] WHERE CONDITION;
v
4) DELETE DML Command
ni

With the DELETE DML command, SQL users can remove a single record or a group
of records from the database tables.
This Data Manipulation Language command does not fully remove the data from the
U

database. We choose which rows from the table to delete by using the WHERE clause
with the DELETE command.
Syntax of DELETE Command
ity

DELETE FROM Table_Name WHERE condition;

3.1.3 Understanding the SELECT Statement

Retrieving data using the SQL Select Statement
m

SQL is a language for all kinds of databases. SQL, which is called “Sequel” or
just “S-Q-L,” is a computing language used to query relational databases in a way that
doesn’t follow a procedure. When you use SQL to get information out of a database, you
)A

are “querying” the database.

A Relational Database Management System (RDBMS) is used to set up and run
a relational database. An RDBMS does all the basic things that the DBMS software
listed above does, plus a lot more that makes the relational model easy to understand
and use. Users of an RDBMS change data by using a special language for doing so. A
(c

data description language is used to describe how a database is set up. Users can put
commands into a terminal with an RDBMS interface or use a graphical interface to store
and access data. After that, the DBMS runs the commands.
Amity Directorate of Distance & Online Education
 104 Data Engineering

Capabilities of the SELECT Statement

Notes

e
SQL is used in the right way and in a fast way to get data from a database. The
SELECT statement can do three things that come from relationship theory: projection,

in
selection and joining.
●● Projection: In a project process, only certain sections (called “fields”) from a table
are chosen. The output table has a selection of the available columns, which can be

nl
anything from a single column to all available columns.
●● Selection: A select action chooses a group of rows (records) from a table (relation)
that meet a selection condition. Selection is the power to pick rows out of a whole

O
set of results. It uses data preparation and filters based on conditions. If none of the
rows meet the selection condition, the group can be all of the rows in a table.
●● Joining: A join action mixes data from two or more tables based on the values of
one or more columns that are the same in both tables. A user of an information

ty
system can use a join procedure to work with the connections between tables. The
join process is very useful because it lets users look into relationships between data
elements that might not have been planned for when the database was made.

si
v er
ni
U

Think about how the above tables are set up. Projection is the process of getting the
first_name, name, department_id and pay for a single employee from the EMPLOYEES
table. Selection means to get information from the EMPLOYEES table about employees
ity

whose salaries are less than 5,000. Joining EMPLOYEES and DEPARTMENTS lets you
get the first name and name of the employee’s department.

Basic SELECT statement

Here are the basic syntax for writing a SELECT sentence.
m

SELECT [DISTINCT | ALL] {* | select_list}

FROM {table_name [alias] | view_name}
)A

[{table_name [alias] | view_name}]...

[WHERE condition]
[GROUP BY condition_list]
[HAVING condition]
(c

[ORDER BY {column_name | column_# [ ASC | DESC ] } ...

Amity Directorate of Distance & Online Education

 Data Engineering 105

The SELECT clause is required and does the work for the database project. Also,
Notes

e
the FROM clause has to be there. It points to one or more tables and/or views where the
column data shown in a result table can be found. The relational select action is done by
the WHERE clause, which can be left out. It tells which rows should be chosen. You don’t

in
have to use the GROUP BY clause. It puts data into groups based on the names of one
or more columns in the SELECT clause.
The extra HAVING clause tells a result table how to decide which groups to include.

nl
The GROUP BY clause says what the groups are. You do not have to use the ORDER BY
clause. It puts question results in ascending or descending order by one or more fields.

O
3.1.4 Using COUNT, DISTINCT and LIMIT
SQL SELECT COUNT
The output of the SQL COUNT() function shows the amount of records in the table.

ty
The SQL SELECT sentence uses this function. Let’s look at a simple example: if you
have a list of voters in a certain area and you want to count how many voters there are,
it’s hard to do it by hand but easy to do with SQL SELECT COUNT query.

si
Select Count Function in SQL Syntax
SELECT COUNT(column_name) FROM table_name;
After the COUNT keyword, we have to write the name of the column and the name
of the table where the Count function will be used.
er
Example of the SQL Select Count Function
v
Now we will show you how to use the Count function in a query by using two SQL
examples:
ni

Example: This is a table called “Bike” that has three columns:

Bike_Name Bike_Colour Bike_Cost

U

Pulsar Black 185,000

Apache Black NULL
KTM RC Red 90,0000
Royal Enfield White NULL
ity

Livo Black 80,000

KTM DUKE Red 195,000

Let’s say you want to find out how many different colours of bikes there are in Bike
m

Table. You must write the following SQL code for this task:
SELECT COUNT (Bike_Colour) AS TotalBikeColour FROM Bikes ;
The following will show up on the screen when this query is run:
)A

TotalBikeColour
6

Distinct
(c

With the SELECT command and the SQL DISTINCT term, you can get rid of all the
duplicate records and get only the unique ones.

Amity Directorate of Distance & Online Education

 106 Data Engineering

There may be times when you have more than one copy of the same record in a table.
Notes

e
When getting these kinds of records, it makes more sense to get only the original ones
rather than getting duplicates. The general form of the DISTINCT keyword looks like this:

in
SELECT DISTINCT column1, column2,.....columnN FROM table_name
Where column1, column2, etc. are the names of the columns from which we want to
get unique or different numbers and table_name is the name of the table that holds the

nl
data.

DISTINCT keyword on Single Column

We can use the DISTINCT keyword on a single column to get all of the unique

O
numbers in that column, without any copies. This is often done to get a summary of the
different numbers in a field or to get rid of duplicate information.

Example

ty
Assume we made a table called “CUSTOMERS” in the SQL database by using the
CREATE TABLE statement, as shown below:
CREATE TABLE CUSTOMERS (

si
ID INT NOT NULL,
NAME VARCHAR (20) NOT NULL,
AGE INT NOT NULL,
er
ADDRESS CHAR (25),
SALARY DECIMAL (18, 2),
v
PRIMARY KEY (ID)
ni

);
Using the INSERT statement, the following query adds rows to this table−
insert INTO CUSTOMERS VALUES(1, ‘Ramesh’, 32, ‘Ahmedabad’, 2000.00);
U

insert INTO CUSTOMERS VALUES(2, ‘Khilan’, 25, ‘Delhi’, 1500.00);

insert INTO CUSTOMERS VALUES(3, ‘kaushik’, 23, ‘Kota’, 2000.00);
insert INTO CUSTOMERS VALUES(4, ‘Chaitali’, 25, ‘Mumbai’, 6500.00);
ity

insert INTO CUSTOMERS VALUES(5, ‘Hardik’, 27, ‘Bhopal’, 8500.00);

insert INTO CUSTOMERS VALUES(6, ‘Komal’, 22, ‘MP’, 4500.00);
insert INTO CUSTOMERS VALUES(7, ‘Muffy’, 24, ‘Indore’, 10000.00);
m

If we use the SELECT statement to check the data of the CUSTOMERS table, we
can see the new records as shown below:
SELECT * from CUSTOMERS;
)A

+----+----------+-----+-----------+----------+
| ID | NAME | AGE | ADDRESS | SALARY |
+----+----------+-----+-----------+----------+
(c

| 1 | Ramesh | 32 | Ahmedabad | 2000.00 |

| 2 | Khilan | 25 | Delhi | 1500.00 |

Amity Directorate of Distance & Online Education

 Data Engineering 107

| 3 | kaushik | 23 | Kota | 2000.00 |

Notes

e
| 4 | Chaitali | 25 | Mumbai | 6500.00 |
| 5 | Hardik | 27 | Bhopal | 8500.00 |

in
| 6 | Komal | 22 | MP | 4500.00 |
| 7 | Muffy | 24 | Indore | 10000.00 |

nl
+----+----------+-----+-----------+----------+
First, let’s look at how the SELECT query below finds the duplicate pay records−

O
SELECT SALARY FROM CUSTOMERS
ORDER BY SALARY;
This would lead to the following result, where the salary (2000) appears twice, which
is a duplicate record from the original table:

ty
+----------+
| SALARY |

si
+----------+
| 1500.00 | er
| 2000.00 |
| 2000.00 |
| 4500.00 |
v
| 6500.00 |
| 8500.00 |
ni

| 10000.00 |
+----------+
U

Now, let’s run the above SELECT query with the DISTINCT keyword and see what
happens−
SELECT DISTINCT SALARY FROM CUSTOMERS ORDER BY SALARY;
ity

Output
The following would happen if we do not have any similar entries:
+----------+
m

| SALARY |
+----------+
| 1500.00 |
)A

| 2000.00 |
| 4500.00 |
| 6500.00 |
| 8500.00 |
(c

| 10000.00 |
+----------+
Amity Directorate of Distance & Online Education
 108 Data Engineering

LIMIT
Notes

e
If there are a large number of tuples that fulfil the query requirements, it may be
helpful to examine only a small subset of them at a time using the SQL limit clause. This

in
is one of the many situations in which the SQL limit clause is quite helpful.

Important points to remember:

™™ The LIMIT clause may be utilised to place a maximum restriction on the number

nl
of tuples that will be returned by SQL.
™™ It is essential to take into consideration that certain versions of SQL do not
support this clause.

O
™™ The LIMIT clause may also be provided by utilising the OFFSET and FETCH
FIRST clauses in SQL 2008 respectively.
™™ The limit and offset expressions have to be a positive integer at all times.

ty
Example:
Let’s pretend that we have an example table with the name Student and for the sake
of better understanding, we’ll look at various queries with the limit clause. Imagine we are

si
in a relationship with one another, Student.

Student Table: er
CREATE TABLE student (
id INT PRIMARY KEY,
name VARCHAR(50),
v
age INT
ni

);
INSERT INTO student (id, name, age)
VALUES (1, ‘Shubham Thakur’, 18),
U

(2, ‘Aman Chopra’, 19),

(3, ‘Bhavika uppala’, 20),
(4,’Anshi Shrivastava’,22);
ity

Output:

id name age
1 Shubhom Thakur 18
m

2 Amen chopra 19
3. Shavika Uppola 20
)A

4. Anshi Shrivastava 22
Queries:
SELECT *
FROM student
(c

LIMIT 3;

Output:

Amity Directorate of Distance & Online Education

 Data Engineering 109

id name age
Notes

e
1 Shubhom Thakur 18
2 Amen chopra 19

in
3 Bhavika uppola 20

3.2 Data Manipulation with SQL

nl
Introduction
In order to query and alter database data, the data manipulation language (DML)

O
known as SQL is utilised.
The following should be included in the SQL DML statement:
●● A new line should be started for each clause that is included in a statement.

ty
●● It is important that the beginnings of each clause are aligned with the beginnings of
the other clauses.
●● If a sentence consists of many components, those components should be written on

si
different lines and indented under the beginning of the phrase to demonstrate their
connection to one another.
●● The representation of reserved terms is done entirely in upper case letters.
er
●● User-defined terms are always represented by their initials in lower case letters.

3.2.1 INSERT, UPDATE and DELETE Statements

The abbreviation for “Data Manipulation Language” is “DML.” It is a language that
v
may be used to pick data from a database, as well as insert, delete and update that data.
Data stored in a relational database may be retrieved from it and manipulated with its
ni

help.
DDL commands are as follows:
U

1. SELECT
2. INSERT
3. UPDATE
ity

4. DELETE

1. SELECT COMMAND
●● The SELECT command is what is utilised in order to get information from the
database.
m

●● With the help of this command, users of a database are able to access the particular
information that they want from a functional repository.
●● It brings back a collection of records that were found in one or more tables as a
)A

consequence.
The SELECT Command has a wide variety of optional clauses, some of which are
listed below:

Clause Description
(c

WHERE It identifies the rows that should be retrieved.

GROUP BY It is put to use in the process of categorising the data into groups.

Amity Directorate of Distance & Online Education

 110 Data Engineering

HAVING It makes a selection from among the groups that have been established
Notes

e
by the GROUP BY clause.
ORDER BY It identifies a certain order in which the rows should be returned.

in
AS It does so by providing an alias that may be employed for the purpose of
indefinitely renaming tables or columns.

nl
Syntax:
SELECT * FROM <table_name>;

Example: SELECT Command

O
SELECT * FROM employee;
OR

ty
SELECT * FROM employee
where salary >=10,000;

2. INSERT COMMAND

si
●● In order to put data into a table, the put command must first be executed.
●● You may add one or more records to any single table in a database by using this
er
command. You also have the option of adding multiple records.
●● Another thing you may do with it is add records to a code that already exists.

Syntax:
v
INSERT INTO <table_name> (`column_name1` <datatype>, `column_name2`
<datatype>, . . . , `column_name_n` <database>) VALUES (`value1`, `value2`, . . . ,
ni

`value n`);

Example:
U

INSERT INTO employee (`eid` int, `ename` varchar(20), `city` varchar(20))

VALUES (`1`, `ABC`, `PUNE`);

3. UPDATE COMMAND
ity

The UPDATE command is what’s utilised to make changes to the records that are
already in the table. This command modifies the data that is already included within a
table. It modifies the information in one or more of the records that are stored in a table.

Syntax:
m

UPDATE <table_name>
SET <column_name = value>
)A

WHERE condition;
Example: UPDATE Command
UPDATE employee
SET salary=20000
(c

WHERE ename=’ABC’;

Amity Directorate of Distance & Online Education

 Data Engineering 111

4. DELETE COMMAND
Notes

e
The DELETE command is what’s used to remove records from an existing table,
either one by one or completely. It removes each record from a table in its entirety.

in
Syntax:
DELETE FROM <table_name> WHERE <condition>;

nl
Example: DELETE Command
DELETE FROM employee
WHERE emp_id = ‘001’;

O
If we do not put the criteria for the WHERE clause, then all of the rows will be
erased.

ty
3.2.2 Working with String Patterns and Ranges in SQL
String Functions
These are called upon to do out an operation on a string of input and then return a

si
string of their own. The following is a list of all of the string functions that are defined by
SQL:
1. ASCII(): The ASCII() function is used to determine the value of a character in the
ASCII code set.
er
Syntax: SELECT ascii(‘t’);
Output: 116
v
2. CHAR_LENGTH(): The CHAR_LENGTH() function is not supported by SQL Server.
ni

For SQL Server, use the LEN() function. The length of a word may be determined with
the help of this function.
Syntax: SELECT char_length(‘Hello!’);
U

Output: 6
3. CHARACTER_LENGTH(): The CHARACTER_LENGTH() function is not supported
by SQL Server. For SQL Server, use the LEN() function. The length of a line may be
determined with the help of this function.
ity

Syntax: SELECT CHARACTER_LENGTH(‘live to learn’);

Output: 13
4. CONCAT(): The CONCAT() method is used to combine two words or strings into a
m

single value.
Syntax: SELECT ‘Geeks’ || ‘ ‘ || ‘forGeeks’ FROM dual;
Output: ‘GeeksforGeeks’
)A

5. CONCAT_WS(): The CONCAT_WS() method is used to combine two words or strings

together while using a single character as the concatenating symbol.
Syntax: SELECT CONCAT_WS(‘_’, ‘geeks’, ‘for’, ‘geeks’);
Output: geeks_for_geeks
(c

6. FIND_IN_SET(): The FIND_IN_SET() method is used to locate a specific symbol

inside a predefined collection of symbols.

Amity Directorate of Distance & Online Education

 112 Data Engineering

Syntax: SELECT FIND_IN_SET(‘b’, ‘a, b, c, d, e, f’);

Notes

e
Output: 2
7. FORMAT(): The FORMAT() method displays a number using the format that was

in
passed in as an argument.
Syntax: Format(“0.981”, “Percent”);
Output: ‘98.10%’

nl
8. INSERT(): Inserting data into a database is accomplished with the help of the INSERT()
method.

O
Syntax: INSERT INTO database (name_id, person_name) VALUES (5000, ‘abc’);
Output: successfully updated
9. INSTR(): The INSTR() method can be used to locate instances of a certain alphabet
in a string.

ty
Syntax: INSTR(‘geeks for geeks’, ‘e’);
Output: 2 (the first occurrence of ‘e’)

si
Syntax: INSTR(‘geeks for geeks’, ‘e’, 1, 2 );
Output: 3 (the second occurrence of ‘e’)
10. LCASE(): The LCASE() method is used to transform the provided string into lower
er
case. This function takes the string as its argument.
Syntax: LCASE (“SimilarWeb To Learn”);
Output: similarweb to learn
v
11. LEFT(): The LEFT() method is what you want to employ if you want to SELECT a
ni

substring from the left of a particular size or characters.

Syntax: SELECT LEFT(‘[Link]’, 5);
Output: geeks
U

12. LENGTH(): The length of a word may be determined with the help of the LENGTH()
function.
Syntax: LENGTH(‘SimilarWeb’);
ity

Output: 10
13. LOCATE(): The LOCATE() method is what you want to employ if you want to find the
provided word in a string at the nth place.
Syntax: SELECT LOCATE(‘for’, ‘similarweb’, 1);
m

Output: 6
14. LOWER(): The LOWER() method is what is used to change a string that is in upper
case to one that is in lower case.
)A

Syntax: SELECT LOWER(‘[Link]’);

Output: [Link]
15. LPAD(): The LPAD() method is used to add the supplied symbol to the given string in
order to make it the size specified by the size parameter.
(c

Syntax: LPAD(‘geeks’, 8, ‘0’);

Output:
Amity Directorate of Distance & Online Education
 Data Engineering 113

000geeks
Notes

e
16. LTRIM(): The LTRIM() method is what you want to employ if you want to separate the
specified substring from the whole text.

in
Syntax: LTRIM(‘123123new’, ‘123’);
Output: new
17. MID(): The MID() function is used to locate a word beginning at the specified position

nl
and having the specified length.
Syntax: Mid (“geeksforgeeks”, 6, 2);

O
Output: for
18. POSITION(): The POSITION() method is used to determine the location of the first
occurrence of the specified letter in the input string.
Syntax: SELECT POSITION(‘e’ IN ‘geeksforgeeks’);

ty
Output: 2
19. REPEAT(): The REPEAT() function is used to write the specified string an infinite

si
number of times, up to the maximum that was specified in the argument.
Syntax: SELECT REPEAT(‘aman’, 2);
Output: amanaman
er
20. REPLACE(): The method known as REPLACE() is used to “cut” the supplied string by
eliminating the given sub string from the string.
Syntax: REPLACE(‘123similar123’, ‘123’);
v
Output: similar
ni

3.2.3 Sorting Result Sets with ORDER BY

On the basis of one or more columns, the data can be sorted using the SQL
ORDER BY clause, which allows for the data to be arranged in ascending or
U

descending order. There are certain databases that automatically sort the query results
into an ascending order.
In addition to that, the ORDER BY clause in a database table may also sort the data
ity

in a table into a specific order of one’s choosing. It is possible that the records of a table
will not be sorted into any conventional order (such as alphabetical or lexicographical),
but the data can be sorted according to any external criterion that exists. For example,
in an ORDERS table that contains the list of orders placed by a variety of clients of an
organisation, the specifics of orders placed can be sorted according to the dates on
m

which such orders are placed. This does not need to be sorted in alphabetical order;
rather, it will be done on a first come, first served basis.

Syntax
)A

The following is a rundown of the fundamental syntax of the ORDER BY clause,

which may be used to sort the result in either ascending or descending order:
SELECT column-list
FROM table_name
(c

[WHERE condition]

Amity Directorate of Distance & Online Education

 114 Data Engineering

[ORDER BY column1, column2, .. columnN] [ASC | DESC];

Notes

e
Within the ORDER BY clause, you are able to make use of more than one column.
Check to see that the column you are using to sort is included in the list of columns.

in
Sorting Results in Ascending Order
If you use SQL’s Order By Clause, the records in a database table may be sorted in
ascending order and you can do this either by letting SQL do it for you automatically or by

nl
using the “ASC” keyword in the clause condition. Let’s look at an example of this to better
comprehend it.

O
Example
Take into consideration the CUSTOMERS table, which contains the following
records:

ty
+----+----------+-----+-----------+----------+
| ID | NAME | AGE | ADDRESS | SALARY |
+----+----------+-----+-----------+----------+

si
| 1 | Ramesh | 32 | Ahmedabad | 2000.00 |
| 2 | Khilan | 25 | Delhi | 1500.00 |
| 3 | kaushik | 23 | Kota | 2000.00 |
er
| 4 | Chaitali | 25 | Mumbai | 6500.00 |
| 5 | Hardik | 27 | Bhopal | 8500.00 |
v
| 6 | Komal | 22 | MP | 4500.00 |
| 7 | Muffy | 24 | Indore | 10000.00 |
ni

+----+----------+-----+-----------+----------+
The following is an example of how to arrange the results so that NAME comes
before SALARY in the ascending order.
U

SELECT * FROM CUSTOMERS

ORDER BY NAME;

Output
ity

As a result, the following would be the output −

+----+----------+-----+-----------+----------+
| ID | NAME | AGE | ADDRESS | SALARY |
m

+----+----------+-----+-----------+----------+
| 4 | Chaitali | 25 | Mumbai | 6500.00 |
| 5 | Hardik | 27 | Bhopal | 8500.00 |
)A

| 3 | kaushik | 23 | Kota | 2000.00 |

| 2 | Khilan | 25 | Delhi | 1500.00 |
| 6 | Komal | 22 | MP | 4500.00 |
(c

| 7 | Muffy | 24 | Indore | 10000.00 |

| 1 | Ramesh | 32 | Ahmedabad | 2000.00 |

Amity Directorate of Distance & Online Education

 Data Engineering 115

+----+----------+-----+-----------+----------+
Notes

e
3.2.4 Grouping Result Sets with GROUP BY

in
SQL Server GROUPING SETS
A GROUP BY clause extension is GROUPING SET. The GROUP BY clause is
extended by the GROUPING SET statement. When used with aggregate functions like

nl
SUM, AVG, COUNT and other similar functions, the GROUP BY statement is used to
summarise data. The result set is divided into categories depending on one or more
columns. The first version of the GROUPING SET appeared in SQL Server 2008.

O
The following example shows how SQL Server’s GROPING SET concept works:
We may use SQL Server to create a group by joining the department and city columns
of an employee database in order to extract the total number of employees employed
in a certain department in a city. Let’s say we want to locate every person working in

ty
Bangalore’s IT department. In addition, we may combine employee type (writer) with
department and city to get a list of every author in Bangalore’s IT department. Because
there are two separate groups, the GROUP BY clause cannot be used to aggregate them

si
in a single query. For each of these categories, we must develop two separate statements:
one for the department and city and another for the personnel type.
This example simplifies things by using the GROUPING SET. Through the definition
er
of several groups in a single query, it expands the possibilities of the GROUP BY clause.
It permits the creation of several groups and the examination and comparison of their
properties using a GROUPING SET in a single query since it is a collection of various
GROUPS.
v
Syntax
ni

The following SQL Server syntax illustrates the GROUPING SET:

SELECT column_list, aggregate(column_name)
U

FROM table_name
GROUP BY
GROUPING SETS (
ity

(column1, column2),
(column1),
(column2),
()
m

);
The following is an explanation of this syntax parameter:
)A

™™ Column_list: It specifies the column’s name that we want to obtain in a result set
from the table. o Column_name: It indicates the column name that we want to
retrieve.
™™ aggregate(column_name): This parameter specifies the column that the
aggregate function (such as SUM, COUNT, AVG, etc.) will be applied to.
(c

™™ GROUP BY: This allows you to specify a column or list of columns as a single
group. You can do this by using the notation.

Amity Directorate of Distance & Online Education

 116 Data Engineering

™™ GROUPING SET: This denotes the collection of columns that need to be

Notes

e
grouped with one another.
™™ (): This denotes an empty grouping set and aggregates columns separately; it is

in
an optional part of the expression.

GROUPING SETS Equivalents

The operation of GROUPING SETS is similar to that of the UNION ALL operator.

nl
When running the UNION ALL query, SQL Server generates the identical results every
time; these results have the GROUP BY list and the GROUPING SETS declared as
the GROUP BY list. ROLLUP and CUBE are two terms that can be interchanged with

O
GROUPING SETS.
The following table provides a more in-depth explanation of the GROUPING SETS
equivalents of UNION ALL, ROLLUP and CUBE:

ty
GROUPING SETS equivalent to UNION ALL
SELECT column1, column2, SELECT column1 NULL AS column2, SUM(column3)
SUM(colmn3) FROM table1 GROUP BY column1
FROM table1 GROUP BY UNION ALL

si
GROUPING SETS ((column1), SELECT NULL AS column1, column2, SUM(column3)
(column2)); FROM table1 GROUP BY column2;
GROUPING SETS equivalent to ROLLUP
er
GROUP BY ROLLUP (column1, GROUP BY GROUPING SETS ((column1, column2,
column2, column3) column3),
(column1, column2),
(column1),
v
());
GROUPING SETS equivalent to CUBE
ni

GROUP BY CUBE (column1, GROUP BY GROUPING SETS ( (column1, column2,

column2, column3) column3),
(column1, column2),
U

(column1, column3),
(column2, column3),
(column1),
(column2),
(column3),
ity

());

GROUPING SETS Example

Let’s have a look at a few examples to see how the GROUPING SETS functionality
m

works in SQL Server. In order to show this, we will begin by constructing a sample table
called ProductQty, which will include information on the purchased product and we will
then put some data into the table.
)A

To generate a ProductQty table, carry out the steps outlined in the following
statement:
CREATE TABLE ProductQty(
Id INT IDENTITY(1,1),
(c

Region VARCHAR(100),
Product VARCHAR(50),

Amity Directorate of Distance & Online Education

 Data Engineering 117

Year INT,
Notes

e
Quantity INT
)

in
Next, in order to put records into this table, run the statement that is listed below:
INSERT INTO ProductQty(Region, Product, Year, Quantity)

nl
VALUES(‘East’, ‘Computer’, 2020, 13000),
(‘South’, ‘Computer’, 2020, 45000),
(‘North’, ‘Computer’, 2020, 25000),

O
(‘East’, ‘Hard Disk’, 2020, 1900),
(‘West’, ‘Computer’, 2021, 25000),

ty
(‘South’, ‘Hard Disk’, 2021, 5500),
(‘West’, ‘Hard Disk’, 2021, 6500),
(‘East’, ‘Pen Drive’, 2021, 1200),

si
(‘North’, ‘Mouse’, 2019, 1600),
(‘South’, ‘Pen Drive’, 2019, 2700), er
(‘East’, ‘Mouse’, 2019, 2000),
(‘West’, ‘Pen Drive’, 2019, 1900);
Now that the SELECT query has been executed, the data can be checked for
v
accuracy:
ni
U
ity
m

Here, rather than beginning with the GROUPING SET, we will begin with the
GROUP BY clause so that we can gradually extend the logic and more clearly
comprehend the relevance and use of the GROUPING SET.
)A

1) Query to determine the total purchased quantity by region

This statement will generate a GROUP by use the column labelled ‘Region’ in order to
determine the total quantity of all purchases. For the purpose of computing the grand total
of the purchases, the SUM aggregate function is used to the ‘Quantity’ column of the table.
(c

1. SELECT Region, SUM(Quantity) AS TotalPurchase

2. FROM ProductQty

Amity Directorate of Distance & Online Education

 118 Data Engineering

3. GROUP BY Region;
Notes

e
When you run the query, the total purchase quantity will be shown as follows:

in
nl
O
ty
3.3 Relational Database Concepts and Database Instance on

si
Cloud
Introduction
er
A collection of data that is organised according to predetermined connections
is called a relational database. It is easy to see and understand how different data
structures are connected to one another since the data is stored in one or more tables
v
(or “relations”) that are made up of columns and rows. Relationships are a type of logical
relationship that are created between different tables based on how those tables interact
ni

with one another.

A cloud database is a database service that can be accessed through a cloud
platform and is developed on top of that platform. It accomplishes many of the same
U

tasks as a traditional database would, but with the extra adaptability that cloud computing
offers. By putting software on a cloud-based computer infrastructure, users will deploy
the database.
ity

3.3.1 Understanding Relational Database Concepts

The database is shown by the Relational Model, often known as RM, as a collection
of relations. A collection of values is all that a relation is; nothing more. Each row in the
table has a set of data values that are connected to one another. These rows in the table
each represent a connection or entity that exists in the actual world. The significance of
m

the data included in each row can be better understood with the assistance of the table
name and column names. The information is given in the form of a collection of relations.
The relational model organises its data into tables for storage. On the other hand, the
)A

logical organisation of the data is not affected by the storage method that is used for the
data physically.
Database management systems, sometimes known as DBMS, are pieces of
software that run on computers and enable users to interact with databases. Users are
granted the ability to govern access to a database, write data, conduct queries and carry
(c

out any other database management-related duties by utilising a database management

system (DBMS).

Amity Directorate of Distance & Online Education

 Data Engineering 119

To do any of these functions, however, the database management system (DBMS)

Notes

e
has to have some type of underlying model that specifies how the data should be
arranged. Since it was initially conceived in the late 1960s, the relational model has been
one of several methods for organising data that has seen widespread adoption in the field

in
of database software. This topic provides a conceptual overview of the relational model,
including its history, how relational databases organise data and how they are utilised in
the modern world.

nl
History of the Relational Model
Databases are collections of information, also known as data, that have been

O
modelled logically. Databases may be created from any collection of data, regardless
of how or where the data is kept. A database may be anything from a file cabinet with
payroll information to a stack of patient forms at a hospital to a company’s collection of
customer information distributed across various locations. Even a file cabinet can be

ty
considered a database.
Physical databases such as these were the only ones that were accessible to
government agencies and corporate organisations that required to keep information in

si
the days when it was normal practice to use computers to store and manage data. The
field of computer science had significant advancements around the middle of the 20th
century, which resulted in the creation of computers with increased processing power
as well as increased capacity for both internal and external storage. Because of these
er
developments, computer scientists began to realise the potential that these computers
had for storing and handling ever-increasing volumes of data.
On the other hand, there were no theories to explain how computers could organise
v
data in ways that were meaningful and logical. It is one thing to store unsorted data on a
machine, but it is an entirely other challenge to create systems that enable you to add,
ni

retrieve, sort and otherwise handle that data in ways that are consistent and useful. The
requirement for a logical structure for the storing and organising of data resulted in a
number of suggestions for how to utilise computers for the purpose of data management.
U

The hierarchical model was an early form of database organisation. This model,
which is analogous to the tree-like structure of current filesystems, was one of the first
database models. The following illustration provides an illustration of how the structure of
a portion of a hierarchical database that is used to classify animals may look:
ity
m
)A

Source: [Link]

The hierarchical model was widely used in the early database management
(c

systems, but it also turned out to be relatively rigid and unsuitable for modern use. In
this approach, individual records can have numerous “children,” but each record
can only have one “parent” in the hierarchy. This is true even if individual records can

Amity Directorate of Distance & Online Education

 120 Data Engineering

have multiple “children.” Because of this, the older hierarchical databases could only
Notes

e
reflect “one-to-one” and “one-to-many” connections between the entities they included
respectively. When working with data pieces that you would want to identify with more
than one parent, the absence of relationships that can be described as “many-to-many”

in
might cause you to run into some difficulties.
Edgar F. Codd, a computer scientist who worked at IBM at the time, came up with
the idea for the relational model of database management in the late 1960s. In addition to

nl
“one-to-many” links, the relational model developed by Codd allowed for “many-to-many”
relationships between data points. This was made possible by the fact that individual
records might be connected with more than one table. When it came to the design of

O
database structures, this allowed for greater flexibility than previous models that were
already in existence. As a result, relational database management systems (RDBMSs)
were able to satisfy a considerably larger range of requirements posed by businesses.
Codd is credited with having a hand in the creation of later database languages due

ty
to his proposal of a language known as Alpha for the management of relational data.
Donald Chamberlin and Raymond Boyce, both of whom worked for IBM and were
colleagues of Codd’s, developed one such language that was influenced by Alpha. They

si
originally referred to their language as SEQUEL, which is an abbreviation for Structured
English Query Language. However, in order to avoid confusion with an already
established trademark, they abbreviated the name of their language to SQL, which is
er
more properly known as Structured Query Language.
Early relational databases were still unusably sluggish because of limitations
imposed by the underlying hardware; hence, it was some time before the technology
became widely used. By the middle of the 1980s, however, Codd’s relational model had
v
already been incorporated into a variety of commercial database management solutions
produced by both IBM and its rivals.
ni

These suppliers, in addition to following IBM’s lead, developed and implemented

their own variants of SQL on their own initiative. By 1987, the American National
Standards Institute and the International Organisation for Standardisation had both
U

certified and published standards for SQL, therefore establishing SQL’s position as the
standard language for the management of RDBMSs. These developments helped to
strengthen SQL’s position as the acknowledged language.
ity

It is because of its widespread use across a variety of sectors that the relational
model is now considered to be the gold standard for data management models.
Relational databases continue to be the most common and widely used tools for storing
and organising data, notwithstanding the proliferation of many NoSQL databases in
recent years.
m

How Relational Databases Organize Data

Let’s take a deeper look at how the relational model organises data now that you
)A

have a broad idea of the relational model’s history. Relationships, also known as tables to
end users and contemporary relational database management systems (RDBMSs), are
the most fundamental building blocks of the relational model. A relation is a collection of
tuples, also known as rows in a table, in which all of the rows have the same properties,
also known as columns:
(c

Amity Directorate of Distance & Online Education

 Data Engineering 121

Notes

e
in
nl
Source: [Link]

O
A relational database’s smallest organisational structure is called a column and it is
used to represent the many aspects of a table’s entries that are used to describe the
table itself. Because of this, they have a more official name: attributes. One way to think
about each tuple is as a one-of-a-kind instance of the many types of persons, things,

ty
events and affiliations that are contained in the table. These examples may include things
like personnel working at a corporation, sales made by an internet business, or findings
from a laboratory test. For instance, the tuples in a table that stores employment records
of instructors at a school may include properties like as name, subjects, start_date and so

si
on. The database might also store other information.
When you create columns, you will be prompted to select a data type. This data
er
type will govern the types of data that may be entered into the column. RDBMSs
frequently implement their very own distinct data types, which may or may not be directly
interchangeable with data types that are used in other systems that are functionally
equivalent. Dates, texts, numbers and Booleans are examples of some of the most
v
prevalent forms of data.
In the relational model, every table has at least one column that is referred to as a
ni

primary key and may be used to identify each row in a way that is completely unique.
This is significant because it indicates that customers do not need to be aware of the
specific location on a computer where their data is physically kept. Instead, the users’
U

database management system (DBMS) may maintain track of each record and retrieve
it on an ad hoc basis. This, in turn, indicates that records do not have a predetermined
logical order and that users have the freedom to return their data in any order or through
any filters that they like.
ity

A foreign key is one method that may be utilised in the event that you have two
tables that you would want to connect to one another in some fashion. A primary key
from one table (referred to as the “parent” table) is copied and then put into a column
of another table (referred to as the “child” table) to create a foreign key. The following
m

illustration illustrates the connection between two tables, one of which is used to store
information on employees at a firm and the other of which is used to keep track of the
sales made by the company. The primary key of the EMPLOYEES database is utilised as
)A

the foreign key of the SALES table in this illustration:

(c

Amity Directorate of Distance & Online Education

 122 Data Engineering

Notes

e
in
nl
O
ty
si
er
Source: [Link]

If you try to add a record to the child table but the value you provide in the foreign
key column isn’t present in the primary key of the parent table, the insertion statement will
v
be invalid. This will occur if you try to add a record to the parent table. Because of this,
the entries in both tables will always be associated in the proper way, which is helpful for
ni

maintaining the relationship-level integrity.

The structural features of the relational model contribute to keeping the data that is
kept in an organised manner; nevertheless, the storage of data is only helpful if it can be
U

retrieved. You may obtain the information you need from an RDBMS by issuing a query,
which is a structured request for a specific collection of data. As was discussed before,
the majority of relational databases make use of SQL for data management and querying.
You have granular control over the data that will be included in the result set since SQL
ity

enables you to filter and alter query results using a wide variety of clauses, predicates
and expressions. This gives you more information about the data.

3.3.2 Creating a Database Instance on Cloud (e.g., IBM Db2)

m

To add an instance using the command line:

Enter the command: db2icrt instance_name.
When creating instance on an AIX® server, you must provide the fenced user id, for
)A

example:

DB2DIR/instance/db2icrt -u db2fenc1 db2inst1

When adding another Db2 instance using the db2icrt command, you are required
to enter the login name of the instance owner. Additionally, you have the option of
(c

specifying the authentication type of the instance. The authentication type is implemented
in each and every database that is produced by that instance. The authentication type

Amity Directorate of Distance & Online Education

 Data Engineering 123

is a declaration that identifies the location at which the users will have their identities
Notes

e
validated.
With the help of the DB2INSTPROF environment variable, you are able to alter the

in
path to the instance directory that is currently being used by DB2PATH. You need to
have read-and-write permissions for the instance directory. Before executing the db2icrt
command, you must first set the DB2INSTPROF variable if you want the folders to be
generated in a path that is different from the DB2PATH variable.

nl
Additionally, if you are using Db2 Enterprise Server Edition (ESE), you are required
to specify that the new instance you are installing is a partitioned database system.
Additionally, when working with an ESE instance that has more than one database

O
partition and when working with Fast Communication Manager (FCM), you may have
numerous connections between database partitions by setting extra TCP/IP ports while
the instance is being created. This is possible because ESE instances support multiple
database partitions.

ty
Use the db2icrt command with the -r port_range argument, for instance, if you’re
working with a Windows operating system. The port range is displayed as follows, with
the base_port being the initial port that FCM is able to use and the end_port being the

si
latest port in a range of port numbers that FCM is able to use:
-r:base_port,end_port er
3.3.3 Getting Started with the Db2 Console
IBM Db2 is a family of linked data management solutions that was created and is
marketed by IBM. Some of the products in this family include relational database servers.
v
IBM has been developing a whole family of database servers since the 1970s. These
database servers were initially implemented on mainframe systems and include Virtual
ni

Machine (VM), Virtual Storage Extended (VSE) and Multiple Virtual Storage (MVS).
1983 saw the introduction of IBM’s Db2 for MVS version 1, which the company
did. IBM’s transition from a hierarchical database like Information Management System
U

(IMS) to the new relational database was denoted by the adoption of the Db2 database
management system. Since that time, IBM has maintained the development of Db2 on
both the mainframe and distributed platforms. Open system platforms like Linux, UNIX
and Windows are examples of distributed systems, which are also known as open
ity

system platforms.
The fifth edition of IBM’s Db2 Universal Database (Db2 UDB) was made available
for distributed systems in the year 1996. This was the first version of Db2 that IBM built
specifically with web optimisation in mind. Since the release of DB2 UDB version 5, IBM
m

has continued to offer several editions of Db2 throughout the years, each one containing
a number of improvements such as pureXML, which supports XML documents and was
introduced in 2006. The Db2 SQL compatibility feature was first made available by IBM in
)A

2009 and it enables users to more easily transition from using other relational database
solutions to utilising Db2.
2012 saw the introduction of IBM Db2 10.1, which included a number of
enhancements designed to assist clients in cutting down on their operational expenses.
The ability to store items at a variety of temperatures was a standout feature. You might,
(c

for instance, divide the data into warm and cool categories. The data that you access
frequently is referred to as “hot data,” and it will be stored in the hardware that is both the

Amity Directorate of Distance & Online Education

 124 Data Engineering

most recent and the most expensive. On the other hand, “cold data” refers to information
Notes

e
that is accessed seldom and is thus kept on slower and less expensive devices.
IBM announced version 10.4 of their Db2 database management system in 2013,

in
which included the BLU accelerated memory-based column store. This memory-based
column store uses parallel vector processing power, dynamic memory capabilities and
sophisticated storage disc technology. Db2 11.1 was introduced by IBM in 2016 and it
brought with it some major innovations that helped lower the total cost of ownership.

nl
Additionally, Db2 11.1 improved analytics, as well as greater availability, stability and
security for the applications that are crucial to your organisation.

O
Download IBM DB2 Developer Edition
After downloading the installation file, you will need to extract the contents of the
downloaded file into a directory (for example, c:software), as indicated in the following
picture:

ty
si
v er
ni

Install DB2 on Windows

Step 1. To begin the installation of DB2, double-click the [Link] file. This will start
the DB2 setup launchpad, which includes the following steps:
U
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 125

Step 2. Navigate to the Release information menu by clicking on it. This will bring up
Notes

e
a window that lists the new features and updates that have been made to this version.

in
nl
O
ty
si
er
Source: [Link]

Step 3. If you are upgrading from an earlier version, click the upgrade information
v
link to obtain further details on the process of updating your DB2 system:
ni
U
ity
m
)A

Source: [Link]

Step 4. To begin using the DB2 11.1 version, click the Install a Product button.
(c

Amity Directorate of Distance & Online Education

 126 Data Engineering

Notes

e
in
nl
O
ty
si
Source: [Link]
er
Step 5. Move the scrollbar to the bottom of the page, then click the “Install New”
button.
v
ni
U
ity
m
)A

Source: [Link]

Step 6. Before beginning the installation of DB2, carefully read the licence
agreement, then click the box labelled “Next” to indicate that you accept the terms of the
(c

agreement.

Amity Directorate of Distance & Online Education

 Data Engineering 127

Notes

e
in
nl
O
ty
si
v er
ni
U
ity
m
)A

Source: [Link]

Step 7. In the install type drop-down menu, select Typical and then click the Next
button:
(c

Amity Directorate of Distance & Online Education

 128 Data Engineering

Notes

e
in
nl
O
ty
si
Source: [Link]
er
Step 8. To generate a response file, select the third option from the drop-down menu
and then click the Next button.
v
ni
U
ity
m
)A

Source: [Link]

Step 9. Choose the location of the installation folder, then keep everything else the
(c

same and click the Next button.

Amity Directorate of Distance & Online Education

 Data Engineering 129

Notes

e
in
nl
O
ty
si
er
Source: [Link]

Step 10. Choosing the IBM SSH server installation folder and the starting choice is
the tenth step. Select the option that says “Do not autostart the IBM SSH server,” as we
v
will only be using DB2 on the currently active server.
ni
U
ity
m
)A

Source: [Link]
(c

Step 11. Configure a user name and password for access to the DB2 administrative
server.

Amity Directorate of Distance & Online Education

 130 Data Engineering

Notes

e
in
nl
O
ty
si
er
Source: [Link]

Step 12. Configure DB2 instances, then click the Next button after leaving the field
v
as DB2.
ni
U
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 131

Step 13. Confirm that you do not want to receive alerts, then click the Next button.
Notes

e
in
nl
O
ty
si
er
Source: [Link]

d/install-db2/
v
Step 14. Activate the operating system security for the DB2 objects, deactivate the
checkbox labelled “Activate operating system security,” and click the Next button:
ni
U
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 132 Data Engineering

Step 15. Once you have reviewed the data, click the Finish button.
Notes

e
in
nl
O
ty
si
`
er
Source: [Link]

Step 16. At this point, step 16, the setup launchpad will begin transferring files and
installing the DB2 server:
v
ni
U
ity
m
)A

Source: [Link]
(c

Step 17. At the step, you need to wait for some time before the installation process is
finished. When you are finished, click the button that says “Finish”:

Amity Directorate of Distance & Online Education

 Data Engineering 133

Notes

e
in
nl
O
ty
si
er
Source: [Link]

3.3.4 Types of SQL Statements: DDL vs. DML

v
Data structures are defined using a language known as DDL, or Data Definition
Language. Examples of SQL commands include create table and modify table. Data
ni

may be changed using the Data Manipulation Language, or DML. Examples of SQL
commands include insert, update and delete. The distinction between DDL and DML is:

DDL DML
U

It is an abbreviation for Data Definition It is an abbreviation for Data Manipulation

Language. Language.
It may be used to provide limits in addition Data may be updated, retrieved, or inserted using it.
to being used to define database structure.
ity

In simple terms, it describes the table’s A table row is either added or changed. Tuples are
Attributes column. the name for these rows.
There is no additional classification. Procedural DML and non-procedural DML are the
two other categories.
m

The fundamental DDL commands are DML supports UPDATE, INSERT, MERGE and
CREATE, DROP, RENAME, ALTER and other BASIC operations.
others.
)A

The statement made by DDL lacks a WHERE The WHERE clause is used by DML in its statement.
clause.
The structure of a database is specified using Data that is kept in a database may be changed
DDL. using DML.
To create and modify database objects DML is used to perform operations on the data that
(c

like tables, indexes, views and constraints, is housed within database objects.
programmers utilise the DDL language.

Amity Directorate of Distance & Online Education

 134 Data Engineering

DDL statements frequently execute more DML commands are often used to query and
Notes

e
slowly than DML statements. change data.
DDL statements are often performed by DML statements are often carried out by application

in
database administrators. developers or end users.
Data changes are not made directly with DDL Data may be changed easily using DML statements.
statements.

nl
The contents of the database are not changed DML operations change the data in the database.
by DDL statements.
DDL commands include CREATE TABLE, DML commands include SELECT, INSERT,
ALTER TABLE, DROP TABLE, TRUNCATE UPDATE, DELETE and MERGE.

O
TABLE and RENAME TABLE.

3.4 Table Operations in SQL

ty
Introduction
Each table in a relational database, which is referred to as a relation, is used to store
data. A table may have as many as 1000 rows. Tables are a common option due to:

si
●● The tables are arranged neatly.
●● Depending on our options, we may divide the data into rows and columns.
●● Retrieval and manipulation of data are made simpler.
er
●● You can find out if any of the data is repeated.
●● The table’s columns can be expanded without affecting the data already there.
v
There are rows and columns in a table. Records are what are known as rows and
fields are what are known as columns. The database and schema names are used to
identify tables in MS SQL Server.
ni

3.4.1 CREATE TABLE Statement

U

SQL is used in relational databases to store data in the form of structures. These
structures are just tables with data in the form of fields and records. A field is a column
that defines the type of data that will be kept in a database, while a record is a row that
contains the actual data. SQL allows you to deal with data by creating tables, updating
ity

them, removing them and so on. The construct TABLE command is used in SQL to
construct a table.
In a SQL Server database, any number of tables may be created. However, there
is a limit to the number of items that may be stored in a database. A database, including
tables, views and indexes, cannot contain more than 2,147,483,647 items. As a result, a
m

single user-defined table can have up to 1024 columns.

A table’s structure must be defined in a SQL query to construct it. The structure
consists of the name of a table and the names of the table’s columns, as well as the data
)A

type of each column. In a database, each table must be individually titled.

Syntax
The fundamental syntax of a CREATE TABLE statement is as follows−
(c

CREATE TABLE table_name(

column1 datatype,

Amity Directorate of Distance & Online Education

 Data Engineering 135

column2 datatype,
Notes

e
column3 datatype,
.....

in
columnN datatype,
PRIMARY KEY( one or more columns )

nl
);
The database system can understand what you want to do by looking at the words
CREATE TABLE. You must create a new table in this case. The distinctive name or

O
identifier of the table comes after the CREATE TABLE statement.
Following in brackets is a list that describes each table column and the kind of data it
contains. The syntax gets easier with the following example.

ty
SQL CREATE TABLE Example
Let’s make a table to hold Customer data, thus the table name is Customer and the
columns are Name, Country, Age, Phone and so on.

si
CREATE TABLE Customer(
CustomerID INT PRIMARY KEY, er
CustomerName VARCHAR(50),
LastName VARCHAR(50),
Country VARCHAR(50),
v
Age int(2),
Phone int(10)
ni

);

Output:
U
ity

3.4.2 ALTER, DROP and TRUNCATE Tables

●● SQL ALTER TABLE
The ALTER TABLE command in Structured Query Language enables you to add,
m

modify and remove columns from an existing table. Additionally, database administrators
are able to add and remove SQL limitations from already-existing tables using this
statement.
)A

Any user may change the name of the table using this statement.

ALTER TABLE ADD Column statement in SQL

You may frequently need to expand an existing table’s columns. Without having to
(c

construct a whole table or database, you can easily add single and many columns by
using the ADD keyword.

Amity Directorate of Distance & Online Education

 136 Data Engineering

Syntax of ALTER TABLE ADD Column statement in SQL

Notes

e
ALTER TABLE table_name ADD column_name column-definition;
You can only add one column to an existing table using the code that is seen above.

in
In a single SQL query, use the following syntax to add more than one column to the table:
ALTER TABLE table_name

nl
ADD (column_Name1 column-definition,
column_Name2 column-definition,
.....

O
column_NameN column-definition);
●● DROP
It is a command in the data definition language (DDL). It is employed to take the

ty
entire table down. The identified components of the schema can be deleted all at once
by using the “DROP” command to drop (delete) the complete structure. The whole table’s
existence is terminated or lost when this command is used.

si
™™ SYNTAX –
If we want to drop the table:
er
DROP table <table_name>;
Note that the “ROLLBACK” command cannot be used to restore the table because it
automatically commits.
v
●● TRUNCATE
It is also a command in the Data Definition Language (DDL). It is used to
ni

simultaneously erase every row from a relation (table). Since the WHERE clause isn’t
being used in this case, we are unable to use the “TRUNCATE” command to delete even
one record. By using this command, all rows in the table are erased from existence.
U

Since it immediately deletes all the rows, it is quicker than the remove command.
™™ SYNTAX –
If we want to use truncate :
ity

TRUNCATE table <table_name>;

Note – Here we can’t restore the tuples of the table by using the “ROLLBACK”
command.
m

3.5 Built-in Database Functions and Date/Time Functions

Introduction
)A

A programming concept known as a built-in function in SQL is one that takes 0–n
inputs and outputs a result. An example of a built-in function is ABS(), which when given
a value computes the number’s absolute (non-negative) value.
Other functions, like GETDATE(), are used to get system values, such the current
(c

data, while still others, like LEFT(), are used to alter textual data. Some functions, like
ABS(), are used to do calculations.

Amity Directorate of Distance & Online Education

 Data Engineering 137

3.5.1 Using Built-in Database Functions (e.g., SUM, AVG, MAX and
Notes

e
MIN)
Database objects called SQL Server functions are collections of SQL statements

in
that carry out certain tasks. A function receives parameters, executes commands and
then returns the result. Functions always return a single value or a table, it’s important
to remember this. Reproducing common tasks rapidly is the main objective of functions.
Depending on our needs, we can write functions once and use them in several locations.

nl
The usage of functions to add, remove, or update records in database tables is not
supported by SQL Server.
The following are the rules for developing SQL Server functions:

O
●● The name of a function cannot begin with a special character, such as @, $, #, or
other like characters.
●● SELECT statements are the only ones that can be used with functions.

ty
●● The SQL SELECT query allows us to use functions like AVG, COUNT, SUM, MIN,
DATE and others.
●● Every time a function is called, it compiles.

si
●● Functions must return a value or outcome.
●● Only the input parameters are used by functions. er
●● Functions are not allowed to contain TRY or CATCH statements.

Types of Functions
SQL Server divides functions into two types:
v
●● System Functions
●● User-Defined Functions
ni

Let’s go into more depth about each category.

System Functions
U

System functions are those ones the system specifies. In other words, system
functions cover all of the server’s built-in features. As we complete the task, the built-in
functions help us to work faster. To compute numbers and change data, these functions
are frequently used in combination with the SQL SELECT statement.
ity

Here is the list of some system functions used in the SQL Server:
™™ String Functions (LEN, SUBSTRING, REPLACE, CONCAT, TRIM)
™™ Date and Time Functions (datetime, datetime2, smalldatetime)
m

™™ Aggregate Functions (COUNT, MAX, MIN, SUM, AVG)

™™ Mathematical Functions (ABS, POWER, PI, EXP, LOG)
™™ Ranking Functions (RANK, DENSE_RANK, ROW_NUMBER, NTILE)
)A

What is an Aggregate Function in SQL?

After computing many column values, an aggregate function in SQL produces a
single result. We frequently employ aggregate functions in conjunction with the SELECT
statement’s GROUP BY and HAVING clauses.
(c

SQL aggregate functions are classified into the following types:

™™ Sum()

Amity Directorate of Distance & Online Education

 138 Data Engineering

™™ Avg()
Notes

e
™™ Min()
™™ Max()

in
●● SUM() Function
The SUM() function calculates the total sum of a numerical column.

nl
Syntax:
SUM()
or

O
SUM( [ALL|DISTINCT] expression )

Example:

ty
The SQL query below computes the total of the “unit price” fields in the “products”
table:

si
This will result in the following outcome. Let’s see how we may utilise the GROUP
BY and HAVING functions in conjunction with the SUM function. Take a look at the
er
following dataset:
v
ni
U

The SQL command below will provide the number of customers in each city with a
ity

total of more than 3000 points.

m

This will result in the following outcome:

)A

●● AVG() Function
(c

The AVG() function computes the average of a collection of numbers.

Amity Directorate of Distance & Online Education

 Data Engineering 139

Syntax:
Notes

e
AVG()
or

in
AVG( [ALL|DISTINCT] expression )

Example:

nl
The SQL query below computes the average quantity in stock.

O
This will result in the following outcome.

ty
●● MIN() Function
The MIN() aggregate method returns the lowest (minimum) value in a collection of
non-NULL items.

si
Syntax:
MIN()
er
or
MIN( [ALL|DISTINCT] expression )
v
Example:
ni

The code above returns the minimum quantity in stock in the products table.
U

●● MAX() Function
The MAX() aggregate method returns the greatest (maximum) value from a group of
ity

non-NULL values.

Syntax:
AVG()
m

or
AVG( [ALL|DISTINCT] expression )
)A

Example:
The code shown below returns the maximum quantity in stock in the products table.
(c

This will result in the following outcome.

Amity Directorate of Distance & Online Education

 140 Data Engineering

Notes

e
in
3.5.2 Date and Time Built-in Functions in SQL
In SQL, we have a plethora of date and time functions at our disposal. These are

nl
supplied to ensure that the date and time module is accessible while creating and
accessing a SQL database.
The following are some of the most common date and time functions:

O
[Link] Function Description
1 NOW( ) The current date and time are displayed.
2 CURDATE( ) The current date is displayed.

ty
3 CURTIME( ) The current time is displayed.
4 DATE( ) The date from the Date/Date Time expression is shown.
5 EXTRACT( ) Displays the specified portion, such as the date and time.

si
6 DAY( ) The day from the specified date is shown.
7 MONTH( ) Displays the month as of the specified date.
8 YEAR( ) Displays the year as of the specified date.
9 DATE_ADD( )
er
Displays the date after adding the specified interval.
10 DATE_SUB( ) After subtracting the specified interval, the date is shown.
11 DATEDIFF( ) The interval between two dates is displayed.
v
12 DATE_FORMAT( ) Displays date/time data in a various formats.
ni

Dates in SQL are difficult for beginners to understand since, while dealing with a
database, the format of the data in the table must match the format of the input data
to enter. Datetime (time is also linked with date) is used in numerous contexts instead
U

of date. MySQL provides the following data types for storing a date or a date and time
value in a database:

DATE format YYYY-MM-DD

ity

DATETIME format: YYYY-MM-DD HH:MI: SS

TIMESTAMP format: YYYY-MM-DD HH:MI: SS
YEAR format YYYY or YY
m

Now, let’s look at some of the most used SQL date functions.
1) NOW()
The current date and time are returned.
)A

Query:
SELECT NOW();

Output:
(c

Amity Directorate of Distance & Online Education

 Data Engineering 141

Notes

e
in
2) CURDATE()

nl
The current date is returned.

Query:

O
SELECT CURDATE();

Output:

ty
si
3) CURTIME() er
The current time is returned.

Query:
SELECT CURTIME();
v
Output:
ni
U

4) DATE()
ity

The date portion of a date or date/time expression is extracted. As an example,

consider the ‘Test’ table below.

Id Name BirthTime
m

4120 Pratik 1996-09-26 16:44:15.581

Query:
)A

SELECT Name, DATE(BirthTime)

AS BirthDate FROM Test;

Output:
(c

Name BirthDate
Pratik 1996-09-26

Amity Directorate of Distance & Online Education

 142 Data Engineering

5) EXTRACT()
Notes

e
This function returns a single portion of a date/time.
Syntax

in
EXTRACT(unit FROM date);
Several units can be considered but only some are used such as MICROSECOND,
SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR, etc. And ‘date’ is

nl
a valid date expression. Example: For the below table named ‘Test’

Id Name BirthTime

O
4120 Pratik 1996-09-26 16:44:15.581

Query:
SELECT Name, Extract(DAY FROM

ty
BirthTime) AS BirthDay FROM Test;

Output:

si
Name Birthday
Pratik
er 26

Query:
SELECT Name, Extract(YEAR FROM BirthTime)
v
AS BirthYear FROM Test;

Output:
ni

Name BirthYear
Pratik 1996
U

Query:
SELECT Name, Extract(SECOND FROM
ity

BirthTime) AS BirthSecond FROM Test;

Output:

Name BirthSecond
m

Pratik 581
6) DATE_ADD()
This adds a time interval to a date.
)A

Syntax:
DATE_ADD(date, INTERVAL expr type);
Where date is a valid date expression and expr is the number of intervals to be
added. and one of the following types: SECOND, MINUTE, HOUR, DAY, WEEK,
(c

MONTH, QUARTER, YEAR and so on. Example: For the ‘Test’ table below

Amity Directorate of Distance & Online Education

 Data Engineering 143

Id Name BirthTime
Notes

e
4120 Pratik 1996-09-26 16:44:15.581

Query:

in
SELECT Name, DATE_ADD(BirthTime, INTERVAL
1 YEAR) AS BirthTimeModified FROM Test;

nl
Output:

Name BirthTimeModified

O
Pratik 1997-09-26 16:44:15.581

Query:
SELECT Name, DATE_ADD(BirthTime,

ty
INTERVAL 30 DAY) AS BirthDayModified FROM Test;

Output:

si
Name BirthDayModified
Pratik 1996-10-26 16:44:15.581
er
Query:
SELECT Name, DATE_ADD(BirthTime, INTERVAL
4 HOUR) AS BirthHourModified FROM Test;
v
Output:
ni

Name BirthSecond
Pratik 1996-10-26 20:44:15.581
U

7) DATE_SUB()
Subtracts a certain time period from a given date. DATE_SUB has the same syntax
as DATE_ADD. The only difference is that DATE_SUB is used to subtract a date interval.
ity

8) DATEDIFF()
The number of days between two dates is returned.

Syntax:
m

DATEDIFF(date1, date2);
date1 & date2- date/time expression
)A

Query:
SELECT DATEDIFF(‘2017-01-13’,’2017-01-03’) AS DateDiff;

Output:
DateDiff
(c

10
9) DATE_FORMAT()

Amity Directorate of Distance & Online Education

 144 Data Engineering

Displays date/time data in various forms.

Notes

e
Syntax:
DATE_FORMAT(date,format);

in
3.5 Accessing Databases Using Python and DB-API

nl
Introduction
Databases are extremely useful tools for data scientists. Python’s standard API for
accessing databases is DB-API. Instead of building a different programme for each type

O
of relational database, you may write a single programme that works with all of them. A
common user accesses databases in this manner by writing Python code in a Jupyter
notebook, a Web-based editor.

ty
si
Source: [Link]

The Python programme interfaces with the DBMS through the following mechanism:
●●
er
One or more API calls that connect the programme to the DBMS are the first step in
the application program’s database access.
●● The application transforms the SQL statement into a text string before calling an API
v
to send the contents to the DBMS.
●● The application software makes API calls to check the status of its DBMS request
ni

and handle problems.

●● By sending an API request to disconnect the application from the database, the
application programme ends its database access.
U
ity
m
)A

Source: [Link]

The Python DB-API’s two fundamental notions are:

1) Connection objects used for
™™ Connect to a database
(c

™™ Manage your transactions.

Here are a few examples of connection methods:
Amity Directorate of Distance & Online Education
 Data Engineering 145

™™ Cursor() leverages the connection to return a fresh cursor object.

Notes

e
™™ Commit(): Commit any pending transaction to the database using the commit()
method.

in
™™ rollback(): This command makes the database start any pending transactions
from the very beginning.
™™ Close(): It closes a database connection using the close() method.

nl
2) Query objects are used to run queries.
This is a Python programme that queries a database using the DB-API.
from dbmodule import connect

O
# Create connection object
connection = connect(‘databasename’, ‘username’, ‘pswd’)

ty
# Create a cursor object
cursor = [Link]()

si
# Run queries
[Link](‘select * from mytable’)
er
results = [Link]()

# Free resources
v
[Link]()
ni

[Link]()
1. To start, we use the connect API of the database module to import data. Use the
connection function with the database name, username and password as inputs to
U

establish a connection to a database. The connect function returns the connection

object.
2. Next, we create a cursor object on the connection object. The cursor is utilised to run
queries and get their outcomes.
ity

3. The cursor is used to execute the searches and to receive the results of the queries.
4. Finally, the system releases all resources by cutting off the connection after the
questions have been answered. Always remember to terminate connections to prevent
resource-hogging idle connections.
m

3.6.1 Introduction to Python’s DB-API

)A

What is Database?
A database is a well-organized group of data that is often saved and accessed
electronically by means of a computer system. More advanced databases are usually
built using formal design and modelling approaches.
(c

Amity Directorate of Distance & Online Education

 146 Data Engineering

Notes

e
in
nl
Source: [Link]

The database management system (DBMS) is the programme that communicates

O
with end users, applications and the database itself in order to store and analyse
data. The fundamental functions needed to manage the database are also included
in the DBMS software. A “database system” is made up of a database, a database
management system and any auxiliary applications. In a wide sense, the term “database”

ty
is typically used to describe any DBMS, the database system, or an application that uses
a database.
Let’s examine what API means.

si
API (Application Programming Interface) definition
v er
ni

Source: [Link]

Application Programming Interface is referred to as API. An application software

U

connects to the operating system or another control programme, such as a database

management system (DBMS) or communications protocol, using the programming
language and message format known as API. By creating function calls that provide a
connection to the required subroutine for execution, APIs are implemented in programmes.
ity

An API therefore denotes that a driver or programme module must be present in the
computer to do the activity, or that software must be integrated into an already-running
programme to carry out the necessary tasks.
m
)A
(c

sqlite3 — DB-API 2.0 interface for SQLite databases

Source: [Link]
Amity Directorate of Distance & Online Education
 Data Engineering 147

A lightweight disk-based database called SQLite is a C library that may be accessed

Notes

e
using a non-standard SQL query language and does not need a separate server process.
Some programmes can store internal data using SQLite. Before converting the code
to a more powerful database like PostgreSQL or Oracle, an application may also be

in
prototyped using SQLite.
The sqlite3 module was built by Gerhard Häring. It provides a SQL interface that
adheres to PEP 249’s DB-API 2.0 specification. A complete SQL database, including

nl
tables, indices, triggers and views, is included in a single disc file. Because the database
file format is cross-platform, it is simple to copy databases between big-endian and little-
endian architectures or between 32-bit and 64-bit platforms.

O
Natively supported data types for SQLite are NULL, INTEGER, REAL, TEXT and
BLOB. Thus, no problems should arise when delivering the following Python types to
SQLite:

ty
si
v er
Source: [Link]

Concepts of Python DB-API

ni

1. Connection Objects — Database Connections and Manage Transactions

●● cursor(factory=Cursor)-The cursor method takes one optional parameter, factory. If
U

provided, this must be a callable that returns an instance of Cursor or one of its
subclasses.
●● commit()-The commit() function commits the current transaction. If you do not call
this method, whatever you have done since the last commit() call is not accessible
ity

to other database connections. If you’re wondering why you can’t see the data
you’ve written to the database, make sure you called this function.
●● rollback()-This function undoes all database modifications made since the last
commit() call.
m

●● close()-This function terminates the database connection. It should be noted that

this does not automatically call commit(). Your modifications will be lost if you just
shut your database connection without first using commit()!
)A

2. Cursor Objects — Database Queries

●● execute(sql[, parameters])- Executes a SQL command. The SQL statement can be
parameterized (by replacing SQL literals with placeholders).
●● executemany(sql, seq_of_parameters)-Applies a SQL command to all parameter
(c

sequences or mappings found in the seq_of_parameters sequence. Instead of a

series, the sqlite3 module allows you to use an iterator that returns arguments.

Amity Directorate of Distance & Online Education

 148 Data Engineering

●● fetchone()-Retrieves the next row of a query result set, returning a single sequence
Notes

e
or None if there is no more data available.
●● fetchmany(size=[Link])-Returns a list after fetching the next set of

in
data from a query result. When there are no more rows available, an empty list is
returned.
The size option specifies the number of rows to retrieve each request. If it is not
specified, the arraysize of the cursor defines the number of rows to be fetched. The

nl
procedure should attempt to get as many rows as the size argument specifies. If this is
not feasible because the requested number of rows is not available, fewer rows may be
returned.

O
Note: It should be noted that the size parameter has performance implications.
It is typically advisable to utilise the arraysize property for best performance. If the size
argument is utilised, it should be kept constant from one fetchmany() call to the next.

ty
●● fetchall()-Returns a list of all (remaining) rows of a query result. It should be noted
that the cursor’s array size property might have an impact on the execution of this
operation. When there are no rows available, an empty list is returned.

si
●● close()-Close the cursor immediately (rather than when __del__ is invoked).
●● arraysize- A read/write property that determines how many rows are delivered by
fetchmany(). The default value is 1, which implies that just one record is obtained
er
every request.

STEPS with different operations used on DB:

1. Import sqlite3 and connect with a database

v
Creating a database-representative Connection object. In the example, the data will
ni

be saved [Link] file:

U

2. Create a Table
Creating a Cursor object and a table name material with characteristics such as
date, trans, symbol, quantity and price.
ity
m

3. Insert command
To execute SQL commands, use its execute() function.
)A
(c

4. Reading the Data

Amity Directorate of Distance & Online Education

 Data Engineering 149

Notes

e
in
nl
5. Update the Records (Optional)

O
Consider that the quantity was modified to 500 on 2008-05-10.

ty
si
6. Delete a record from a table (USE it carefully)
er
Let us remove the date record — 2006-01-05.
v
ni
U

7. Closing the Connection

ity

Benefits of the Python for Database Programming

●● Python is a widely used programming language for working with databases.
m

●● The Python environment is incredibly extensive and data science tools are simple to
use.
)A

●● Python is portable across numerous systems due to its open source nature.
●● Python supports the Relational Database System (RDBMS).
●● Creating Python code to access database API commands, also known as DB-API.

3.6.2 Connecting to a Database Using ibm_DB-API

(c

You must first connect to a database before you can perform SQL commands
to create, edit, remove, or retrieve data. The ibm_db API allows you to connect to a

Amity Directorate of Distance & Online Education

 150 Data Engineering

database using either a catalogued or uncataloged connection. You may also establish a
Notes

e
persistent connection to boost performance.
Before you start,

in
™™ Set up the Python environment for IBM® Database servers.
™™ In your Python script, use the import ibm_db statement.

nl
Procedure
To connect to an IBM database server, use one of the following functions.
Table. ibm_db connection functions

O
Function Description
ibm_db.connect Creates a nonpersistent connection.
ibm_db.pconnect Makes a long-lasting relationship. After the initial Python script request, a

ty
persistent connection stays open, allowing subsequent Python requests
to reuse the connection. The credentials for subsequent Python connect
requests must be the same.

si
The database value sent as an argument to these methods can be either a
catalogued database name or a whole database connection string for a direct TCP/
IP connection. Optional parameters can be used to influence the timing of committing
er
transactions, the case of the column names returned and the cursor type.
If the connection attempt fails, use the ibm_db.conn_error or ibm_db.conn_errormsg
functions to get diagnostic information.
v
Example
ni

Example 1
Connect to a database that is either local or categorised.
import ibm_db
U

conn = ibm_db.connect(“database”,”username”,”password”)

Example 2
ity

Connect to an uncataloged database.

import ibm_db
ibm_db.connect(“DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=
TCPIP;UID=username;
m

PWD=password;”, “”, “”)

What to do next
)A

If the connection attempt is successful, the connection resource can be used when
calling ibm_db functions that perform SQL queries. The following step is to prepare and
execute SQL statements.

Setting up the Python environment for IBM database servers

(c

Before you can connect to an IBM® database server and perform SQL commands,
you must first install the ibm_db (Python) driver. You may also download the ibm_db_sa
(SQLAlchemy) or ibm_db_django (Django) adapters.

Amity Directorate of Distance & Online Education

 Data Engineering 151

Before you begin

Notes

e
Make sure the following software is installed on your computer:
●● Python 3.0 or later. For Linux® operating systems, you also require the python3.0-

in
dev package.
●● The setuptools program or the distribute program. The setuptools program is
available at [Link] and the distribute program is

nl
available at [Link] You can use the setuptools
program or the distribute program to download, build, install, upgrade and uninstall
Python packages.

O
●● If your Python programme will connect to a distant IBM database, one of the
following items must be installed on the machine where your application will run:
™™ The IBM Data Server Client software
™™ The IBM Data Server Runtime Client application

ty
™™ The IBM Data Server Driver Package product
™™ The IBM Data Server Driver for ODBC and CLI product

si
If your Python application connects to an IBM database server on the local computer,
no additional IBM data server products are needed.

Procedure
er
To configure the Python environment
1. Install the ibm_db Python driver using the following procedure:
v
™™ Use the remote repository to install:
a. You can specify IBM_DB_HOME to avoid automatic installation of the
ni

clidriver and utilise an existing copy of the driver.

Set the IBM_DB_HOME environment variable by using the export command:
$ export IBM_DB_HOME=DB2HOME
U

where DB2HOME is the directory where the IBM data server product is
installed.
For example, issue the following command to set the IBM_DB_HOME
ity

environment variable:
$ export IBM_DB_HOME=/home/db2inst1/<dsdriver installation>/clidriver
b. Issue the following command:
$ pip install ibm_db
m

Or
$ easy_install ibm_db
)A

2. Optional: Using the following method, install the ibm_db_sa SQLAlchemy adapter or
ibm_db_django Django adapter:
™™ Install from the remote repository:
a. To install the SQLAlchemy adapter, issue the following command:
$ pip install ibm_db_sa
(c

Or
$ easy_install ibm_db_sa
Amity Directorate of Distance & Online Education
 152 Data Engineering

b. To install the django adapter, issue the following command:

Notes

e
$ pip install ibm_db_django
Or

in
$ easy_install ibm_db_django
3. Ensure that the Python driver can access the [Link] CLI driver file:

nl
™™ For 32-bit Linux and UNIX operating systems other than the AIX operating
system, set the LD_LIBRARY_PATH variable to the IBM_DB_HOME/lib32
directory by issuing the export command:
$ export LD_LIBRARY_PATH=IBM_DB_HOME/lib32

O
™™ For 64-bit Linux and UNIX operating systems other than the AIX operating
system, set the LD_LIBRARY_PATH variable to the IBM_DB_HOME/lib64
directory by issuing the export command:

ty
$ export LD_LIBRARY_PATH=IBM_DB_HOME/lib64
™™ For a 32-bit AIX operating system, set the LIBPATH variable to the IBM_DB_
HOME/lib32 directory by issuing the export command:

si
$ export LIBPATH=IBM_DB_HOME/lib32
™™ For a 64-bit AIX operating system, set the LIBRARY_PATH variable to the IBM_
DB_HOME/lib64 directory by issuing the export command:
er
$ export LIBPATH=IBM_DB_HOME/lib64

3.6.3 Creating Tables and Loading Data in Python

v
Python provides extensive support for libraries that may be used to create a variety
of applications. This topic will discuss two such modules that may be used to generate
ni

tables.

Method 1: Using Tabulate module

U

The tabulate() method is a tabulate module method that generates a text-based table
output within a Python programme using any specified inputs. It may be installed using
the following command.
pip install tabulate
ity

Here are some examples of how to make tables in Python:

Example 1
# import module
m

from tabulate import tabulate

# assign data
)A

mydata = [
[“Nikhil”, “Delhi”],
[“Ravi”, “Kanpur”],
[“Manish”, “Ahmedabad”],
(c

[“Prince”, “Bangalore”]
]

Amity Directorate of Distance & Online Education

 Data Engineering 153

# create header
Notes

e
head = [“Name”, “City”]
# display table

in
print(tabulate(mydata, headers=head, tablefmt=”grid”))

Output:

nl
O
ty
Loading Data in Python Using 5 different methods
1) Manually loading a file: This is the first, most common and least recommended method

si
of loading data since it involves numerous code sections to read a single tuple from
the DataFrame. When there is no particular pattern to identify or a specific pattern in
the dataset, this method comes into play. er
2) np. load txt: One of the NumPy techniques for loading various forms of data, albeit it
is only supported when the data is in a specified format, i.e., pattern recognisable, as
opposed to the manual method of reading the dataset.
3) Using [Link]: This is yet another NumPy method for reading data, but it is far
v
superior to the previous one. The [Link]() method recognises the presence
of the column header on its own, whereas the previous one does not. It can also
ni

recognise the appropriate data type for each column.

4) Using PD.read_csv: This is the most recommended and extensively used method.
It only works with CSV data, but the support for multiple parameters makes it a gold
U

mine for data analysts working with diverse types of data (they should have a certain
format).
5) Using pickle: Last but not least, we will use a pickle to read the binary-formatted
ity

dataset.

3.6.4 Querying Data from Databases with Python

Using the SELECT query, you may retrieve/fetch data from a table in MySQL. This
m

query/statement is known as a result-set because it returns the contents of the specified

table in tabular form.

Syntax
)A

The syntax of the SELECT query is as follows:−

SELECT column1, column2, columnN FROM table_name;

Example
(c

Assume we’ve established a table in MySQL called cricketers_data.

CREATE TABLE cricketers_data(

Amity Directorate of Distance & Online Education

 154 Data Engineering

First_Name VARCHAR(255),
Notes

e
Last_Name VARCHAR(255),
Date_Of_Birth date,

in
Place_Of_Birth VARCHAR(255),
Country VARCHAR(255)

nl
);

And suppose we’ve added 5 records into it using INSERT commands as follows:−

O
insert into cricketers_data values(
‘Shikhar’, ‘Dhawan’, DATE(‘1981-12-05’), ‘Delhi’, ‘India’);
insert into cricketers_data values(

ty
‘Jonathan’, ‘Trott’, DATE(‘1981-04-22’), ‘CapeTown’, ‘SouthAfrica’);
insert into cricketers_data values(
‘Kumara’, ‘Sangakkara’, DATE(‘1977-10-27’), ‘Matale’, ‘Srilanka’);

si
insert into cricketers_data values(
‘Virat’, ‘Kohli’, DATE(‘1988-11-05’), ‘Delhi’, ‘India’);
er
insert into cricketers_data values(
Rohit’, ‘Sharma’, DATE(‘1987-04-30’), ‘Nagpur’, ‘India’);

The following query pulls the values FIRST_NAME and COUNTRY from the table.
v
mysql> select FIRST_NAME, Country from cricketers_data;
ni

+------------+-------------+
| FIRST_NAME | Country |
+------------+-------------+
U

| Shikhar | India |
| Jonathan | SouthAfrica |
| Kumara | Srilanka |
ity

| Virat | India |
| Rohit | India |
+------------+-------------+
m

5 rows in set (0.00 sec)

You may also get all the values for each record by putting * in front of the column
names, as in−
)A

mysql> SELECT * from cricketers_data;

+------------+------------+---------------+----------------+-------------+
| First_Name | Last_Name | Date_Of_Birth | Place_Of_Birth | Country |
(c

+------------+------------+---------------+----------------+-------------+
| Shikhar | Dhawan | 1981-12-05 | Delhi | India |

Amity Directorate of Distance & Online Education

 Data Engineering 155

| Jonathan | Trott | 1981-04-22 | CapeTown | SouthAfrica |

Notes

e
| Kumara | Sangakkara | 1977-10-27 | Matale | Srilanka |
| Virat | Kohli | 1988-11-05 | Delhi | India |

in
| Rohit | Sharma | 1987-04-30 | Nagpur | India |
+------------+------------+---------------+----------------+-------------+

nl
5 rows in set (0.00 sec)

Python data reading from a MYSQL table

READ Any database operation entails retrieving meaningful information from the

O
database. The mysql-connector-python provides the fetch() function for retrieving data
from MYSQL.
The [Link] class provides three methods namely fetchall(),

ty
fetchmany() and, fetchone() where,
●● The fetchall() function fetches all of the rows in a query’s result set and returns them
as a list of tuples. (If we run this after retrieving a few rows, it returns the remaining
ones).

si
●● The fetchone() function returns the next row in the result of a query as a tuple.
●● The fetchmany() function is similar to the fetchone() method, except it obtains the
er
next group of rows in a query’s result set rather than a single row.
Note − A result set is an object that is returned when you use a cursor object to
query a table.
v
rowcount − This property is read-only and returns the number of rows impacted by
an execute() method.
ni

Example
The next example retrieves all the rows of the EMPLOYEE table using the SELECT
query and from the resulting result set, we retrieve the first row using the fetchone()
U

method, followed by the remaining rows using the fetchall() method.

import [Link]
#establishing the connection
ity

conn = [Link](
user=’root’, password=’password’, host=’[Link]’, database=’mydb’)

#Creating a cursor object using the cursor() method

m

cursor = [Link]()
)A

#Retrieving single row

sql = ‘’’SELECT * from EMPLOYEE’’’

#Executing the query

(c

[Link](sql)

#Fetching 1st row from the table

Amity Directorate of Distance & Online Education

 156 Data Engineering

result = [Link]();
Notes

e
print(result)

in
#Fetching 1st row from the table
result = [Link]();
print(result)

nl
#Closing the connection

O
[Link]()

Output
(‘Krishna’, ‘Sharma’, 19, ‘M’, 2000.0)

ty
[(‘Raj’, ‘Kandukuri’, 20, ‘M’, 7000.0), (‘Ramya’, ‘Ramapriya’, 25, ‘M’, 5000.0)]
Using the fetchmany() function, the following example fetches the first two rows of
the EMPLOYEE database.

si
Example
import [Link] er
#establishing the connection
conn = [Link](
v
user=’root’, password=’password’, host=’[Link]’, database=’mydb’)
ni

#Creating a cursor object using the cursor() method

cursor = [Link]()
U

#Retrieving single row

sql = ‘’’SELECT * from EMPLOYEE’’’
ity

#Executing the query

[Link](sql)

#Fetching 1st row from the table

m

result = [Link](size =2);

print(result)
)A

#Closing the connection

[Link]()
Output
(c

[(‘Krishna’, ‘Sharma’, 19, ‘M’, 2000.0), (‘Raj’, ‘Kandukuri’, 20, ‘M’

Amity Directorate of Distance & Online Education

 Data Engineering 157

Summary
Notes

e
●● SQL is a language for working with databases. It can be used to create databases,
delete databases, get data rows, change and delete data rows and more.

in
●● Structured Query Language, or SQL, is a computer language used to store, change
and get information from a relational database.
●● SQL was created by IBM computer scientists in the 1970s. It became a standard

nl
for both the American National Standards Institute (ANSI) and the International
Organisation for Standardisation (ISO) in 1986 and 1987, respectively.
●● A database management system is a piece of software used to run databases. For

O
example, MySQL, Oracle and other business databases are very common and are
used in many different ways.
●● DBMS gives you a way to do things like create a database, store data in it, change
data, make a table in the database and a lot more.

ty
●● The SELECT command is what is utilised in order to get information from the
database. With the help of this command, users of a database are able to access
the particular information that they want from a functional repository.

si
●● GROUPING SET is a GROUP BY clause extension. The GROUPING SET
statement is an expansion of the GROUP BY clause.
●● The GROUP BY statement is used to summarise data when used with aggregate
er
functions like SUM, AVG, COUNT and so on. It organises the result set into groups
based on a single or more columns.
●● Pig is a high-level programming language used to create data transformation
v
programmes. It allows data analysis programmes to be expressed in the same
manner that humans communicate about their job.
ni

●● Mahout is a collection of machine-learning libraries that operate on Hadoop.

Many common methods, such as k-means clustering, naive Bayes classification,
logistic regression, support vector machines (SVM), random forests and so on, are
implemented.
U

Glossary
●● Data Definition: It is used to add, change, or remove definitions that describe how
the data in the database is organised.
ity

●● Data Updating: This is used to add, change, or get rid of real data in the database.
●● Data Retrieval: It is used to get data from the database so that applications can use
it in different ways.
m

●● User Administration: This is used to register and keep track of users, maintain data
integrity, enforce data security, deal with process control, keep track of performance
and recover information that was damaged by an unexpected failure.
●● Column_list: It specifies the column’s name that we want to obtain in a result set
)A

from the table. o Column_name: It indicates the column name that we want to
retrieve.
●● aggregate(column_name): This parameter specifies the column that the aggregate
function (such as SUM, COUNT, AVG, etc.) will be applied to.
(c

●● GROUP BY: This allows you to specify a column or list of columns as a single
group. You can do this by using the notation.

Amity Directorate of Distance & Online Education

 158 Data Engineering

●● GROUPING SET: This denotes the collection of columns that need to be grouped
Notes

e
with one another.

Check Your Understanding

in
1. The __________________ clause may be utilised to place a maximum restriction on
the number of tuples that will be returned by SQL.
a) LIMIT

nl
b) COUNT
c) DISTINCT

O
d) INSERT
2. ___________________ is a language that may be used to pick data from a database,
as well as insert, delete and update that data. Data stored in a relational database
may be retrieved from it and manipulated with its help.

ty
a) Transaction Control Language
b) Data Definition Language

si
c) Data Control Language
d) Data Manipulation Language
3. The ___________________ command is what’s utilised to make changes to the
er
records that are already in the table. This command modifies the data that is already
included within a table.
a) COUNT
v
b) CREATE
c) UPDATE
ni

d) DELETE
4. The _______________ function is used to determine the value of a character in the
ASCII code set.
U

a) ASCII()
b) CHAR_LENGTH()
ity

c) CHARACTER_LENGTH()
d) FIND_IN_SET()
5. The ___________________ method is used to combine two words or strings into a
single value.
m

a) CONCAT()
b) LTRIM()
c) MID()
)A

d) LOWER()
6. Inserting data into a database is accomplished with the help of the _________________
method.
a) POSITION()
(c

b) INSERT()
c) RTRIM()

Amity Directorate of Distance & Online Education

 Data Engineering 159

d) REVERSE()
Notes

e
7. The length of a word may be determined with the help of the _________________
function.

in
a) LOWER()
b) LENGTH()
c) LTRIM()

nl
d) MID()
8. _____________________ is a function that is used to reverse the contents of a string.

O
a) REVERSE()
b) GROUP BY
c) SPACE()

ty
d) RTRIM()
9. _______________ specifies the column’s name that we want to obtain in a result set
from the table. o Column_name: It indicates the column name that we want to retrieve.

si
a) SUBSTRING_INDEX()
b) GROUP BY
c) Column_list
er
d) UCASE()
10. _______________ parameter specifies the column that the aggregate function (such
as SUM, COUNT, AVG, etc.) will be applied to.
v
a) aggregate(column_name)
ni

b) Column_list
c) SUBSTR()
d) MID()
U

11. A ________________ in SQL is a programming construct that accepts zero or more

inputs and returns a result.
a) Built-in function
ity

b) Column_list
c) SPACE()
d) LOWER()
12. _______________ function is used to commit any pending transaction to the database.
m

a) commit()
b) rollback()
)A

c) count()
d) distinct()
13. ________________ function forces the database to revert to the beginning of any
outstanding transaction.
(c

a) Close()
b) Commit()

Amity Directorate of Distance & Online Education

 160 Data Engineering

c) rollback()
Notes

e
d) connection()
14. A ___________________ is a structured collection of data that is often stored and

in
accessible electronically via a computer system.
a) Transaction
b) Manipulation

nl
c) JAVA
d) Database

O
15. __________________ is the first, most common and least recommended method of
loading data since it involves numerous code sections to read a single tuple from the
DataFrame in python.
a) Closing the Connection

ty
b) Manually loading a file
c) Using pickle

si
d) np. load txt
16. ____________________ Returns a list of all (remaining) rows of a query result. It
should be noted that the cursor’s array size property might have an impact on the
er
execution of this operation. When there are no rows available, an empty list is returned.
a) fetchall()
b) cursor()
v
c) rollback()
d) commit()
ni

17. _________________ function uses the connection to return a new cursor object.
a) cursor()
U

b) close()
c) rollback()
d) commit()
ity

18. _________________ function displays current date and time are displayed.
a) GROUP BY
b) INSERT()
c) NOW( )
m

d) INSTR()
19. __________________ function displays current time is displayed.
)A

a) FIND_IN_STR()
b) CURTIME( )
c) DATE()
d) EXTRACT()
(c

20. __________________ are database objects that include a group of SQL statements
to accomplish a specified activity.

Amity Directorate of Distance & Online Education

 Data Engineering 161

a) Data Definition
Notes

e
b) Data Updating
c) Data Retrieval

in
d) SQL Server functions

Exercise

nl
1. What are the five different methods of loading Data in Python?
2. Explain the INSERT, UPDATE and DELETE Statements in SQL.
3. Explain the COUNT, DISTINCT and LIMIT of SQL.

O
4. What is an Aggregate Function in SQL?
5. Explain ALTER, DROP and TRUNCATE functions of a table.
6. What is the difference between Data Definition Language and Data Manipulation

ty
Language.
7. How to install DB2 on Windows?

si
Learning Activities
1. Explain Date and Time Built-in Functions in SQL.
2. Write a query to get the first day of the current year.
er
3. Write a query to display the first day of the month (in datetime format) three months
prior the month of August. .

Check Your Understanding – Answers

v
1. a) 2. d)
ni

3. c) 4. a)
5. a) 6. b)
7. b) 8. a)
U

9. c) 10. a)
11. a) 12. a)
13. c) 14. d)
ity

15. b) 16. a)
17. a) 18. c)
19. b) 20. d)
m

Further Readings and Bibliography

1. Joe Reis and Matt Housley. (2022). Fundamentals of Data Engineering: Plan
and Build Robust Data Systems. First Edition. O’Reilly Publications.
)A

2. Andreas Kretz. (2019). The Data Engineering Cookbook: Mastering the

Plumbing of Data Science.
3. Holden Karau andy Konwinski, Patrick Wendell and Matei Zaharia. (2015).
Learning Spark. First Edition. O’Reilly Publications.
(c

Amity Directorate of Distance & Online Education

 162 Data Engineering

Module -IV: Big Data, Hadoop and Spark Basics

Notes

e
Learning Objectives

in
At the end of this module, you will be able to:
●● Analyse impact of big data and scaling challenges

nl
●● Identify tools of big data and beyond the hype
●● Learn introduction to the Hadoop ecosystem
●● Analyse understanding Hadoop and MapReduce

O
●● Identify introducing HDFS, Hive, HBase and Spark
●● Learn other modules in the Hadoop ecosystem
●● Analyse working with HDFS and HBase

ty
●● Learn why use Apache spark
●● Analyse functional programming basics
●● Identify parallel programming with resilient distributed DataSets (RDDs)

si
●● Analyse scale-out and data parallelism in apache spark
●● Learn introduction to DataFrames and DataSets
er
●● Analyse catalyst and tungsten optimisations
●● Identify ETL with DataFrames
●● Identify real-world usage of SparkSQL
v
●● Learn apache spark architecture overview
●● Identify apache spark cluster modes
ni

●● Identify applications of running apache spark

●● Learn using apache spark on IBM cloud
●● Recognise the apache spark user interface
U

●● Identify monitoring jobs and debugging parallel jobs

●● Analyse managing memory and processor resources in spark
●● Identify setting apache spark configuration and running on Kubernetes
ity

Introduction
Big Data is a massive body of information that is continuously growing exponentially.
It is a data set that is too large and intricate for normal data management technologies
m

to effectively store or examine. Big data is a kind of information that is very huge in size.
Think about the volume of data that millions of other people producing the same thing
every hour and day must generate. Data in the terabytes and petabytes range. Big data
)A

is a term that describes this. It is occasionally required to collect and evaluate user-
generated data in order to provide better services. However, handling this amount of
data is straightforward; our main concern is finding tools that are incredibly effective in
handling and analysing this amount of data.
(c

4.1 Big Data and Introduction to Hadoop Ecosystem

The technologies that support the storage, upkeep and analysis of massive volumes

Amity Directorate of Distance & Online Education

 Data Engineering 163

of data are examined in this topic. There are four stages to each big data processing
Notes

e
process, each with its own set of tools. The following are these levels:
™™ Data Ingestion

in
™™ Data Storage
™™ Resource Management
™™ Data Processing and analysis

nl
™™ Data Access

O
ty
si
v er
Hadoop Eco-system
Source: [Link]
ni

Let’s define Hadoop before we discuss the following difficulties.

Describe Hadoop.
U

Let’s say you own a business where you must manage and store gigabytes of data
every day. You use a single storage unit that is powered by a relational database system
to do this. Take into account what may possibly go wrong with this setup.
●● Scalability - We are unable to grow the entire system to meet demand and we
ity

are unable to upgrade the existing hardware without experiencing disruption.

Additionally, it can be prohibitively expensive to install specialised CPU and memory.
●● Fault Tolerance - If the system is compromised, whether by a malicious assault or
a natural calamity, we risk losing all of the priceless data, which is worth millions of
m

dollars.
●● Convenience - It may be tough to bring a large volume of data to analyse and
uncover trends.
)A

●● Unstructured Data - A large portion of the data required by organisations is

underdeveloped. Traditional relational database management systems (RDBMS)
are theoretically incapable of handling unstructured data.
Hadoop, created by Doug Cutting and Mike Cafarella, is the answer to everything
described above. Hadoop’s primary premise is to split and distribute data to multiple
(c

nodes in a cluster and these nodes then process the data. The job tracker schedules jobs
on these nodes and returns the output when they are completed.

Amity Directorate of Distance & Online Education

 164 Data Engineering

Hadoop’s primary components are HDFS, YARN and Map Reduce. Other libraries
Notes

e
and software packages are installed alongside or on top of key components to assist the
full Big Data process and these components are interconnected.

in
1) Data Ingestion
Data intake makes up the first layer of a big data architecture. Social media is one of
the various sources from which the data is gathered. It is necessary to collect and convey

nl
sensors, IoT devices and SaaS platforms to a central warehouse or database. There are
three categories of data intake techniques.
●● Real-time: Used when data has to be processed immediately. A power system’s

O
data, for instance, has to be verified often.
●● Batch Ingestion: When data needs to be ingested at regular intervals, this is
advantageous.

ty
●● Lambda: This combines batch and real-time processing.
The three main data input technologies are Flume, Sqoop and Kafka.

Flume

si
Flume is a platform for data intake that enables the collection, consolidation and
transfer of huge amounts of data from one source to another. It is commonly necessary to
write data created from many sources to HDFS or Hbase. Flume serves as a middleman,
er
simplifying the process of writing data from sources to storage platforms like HDFS
and HBase. Both batch and real-time data collection are supported. It is trustworthy,
expandable, managed, fault-tolerant and flexible.
v
Sqoop
Apache Sqoop is a different data intake method that mostly use relational databases.
ni

From several sources, we might need to transfer structured data to HDFS, where Sqoop
is employed. Practically all SQL databases, including MySQL, PostgreSQL and SQLite,
are compatible with it. It carries out two fundamental tasks: importing and exporting data
U

from and to a variety of sources.

Kafka
Kafka is yet another free and open-source solution for building streaming
ity

applications and real-time data pipelines. As a platform that manages and mediates
communication between two applications, Kafka is widely utilised as a message broker
solution.

2) Data Storage
m

Big Data Architecture is built on top of this. The entire Hadoop system is essential
because it can efficiently store petabytes of data. The main data storage component of
Hadoop is HDFS. The present architecture may be enhanced by using other services like
)A

Hbase and Cassandra.

HDFS
HDFS (Hadoop Distributed File System) is a file system designed to run on
commodity systems. A Name node and numerous data nodes comprise a typical HDFS
(c

design. A cluster is a group of nodes that may be considered as a single computer and
each cluster can include 1000s of nodes.

Amity Directorate of Distance & Online Education

 Data Engineering 165

When HDFS receives data, it separates it into small chunks, generally 64 or 128 MB.
Notes

e
The system parameters define the size of the chunk. HDFS follows a principle known as
“rack awareness” while dividing and duplicating data. An assembly of 40–50 data nodes
is called a rack. Because each duplicate of the chunked data is stored on a different rack,

in
it is extremely fault-tolerant.
HDFS utilises a master-slave design for data processing. The data nodes are
thought of as secondary, but the master node is also known as the Name node. The

nl
file system namespace is managed by the master server, sometimes referred to as the
Name node, which also manages client access to files. The data nodes, which handle
read and write operations from file system clients, are where the node’s storage is

O
located. Additionally, they are in responsibility of responding to requests from Name
nodes by removing, replicating and generating blocks.
The full procedure is depicted in the diagram below.

ty
si
v er
ni
U

Source: [Link]
ity

The master-slave design has one crucial flaw: if the Name node or Master node is
hacked, the cluster function would come to a standstill.

HBase
After talking about HDFS, it’s time to explain HBase, a non-relational column-
m

oriented database management system that runs on top of HDFS. Similar to HDFS, it
uses slave nodes or region servers to store the portion of the table and handle read and
write operations. A master node manages clusters. It includes the innovative concept
)A

of classifying columns into column families. These column families can be changed as
needed to accommodate shifting application requirements. It offers random read-write
access to huge volumes of data and is ideal for researching real-time data.

Cassandra
(c

With the exception of image data, the non-relational database management system
Cassandra is best suited for semi-structured data, while it can also manage structured
and unstructured data. Contrary to HDFS, which might crash if the master server

Amity Directorate of Distance & Online Education

 166 Data Engineering

goes down, Cassandra features a masterless or peer-to-peer design, making it almost

Notes

e
impervious to failure. It is ideal for businesses with a worldwide user base since it has
the special ability to read/write data on nodes in a completely other location. It also has a
high failure tolerance.

in
3) Resource Management
Resource management is one of the key concepts in Big Data architecture. The best

nl
performance will be achieved if we utilise our resources to their fullest potential. Hadoop
uses YARN, another essential part of the Hadoop ecosystem, for resource management.

O
YARN
YARN, or Yet Another Resource Negotiator, is an acronym. In Hadoop 1.0, map-
reduce was in charge of processing and job tracking. The utilisation of resources,
however, turned out to be incredibly ineffective. Then came YARN, which took over the

ty
task of allocating resources and scheduling jobs from map-reduce. Between HDFS and
map-reduce is now where the YARN is located. There are four key parts of YARN.
●● Resource Manager: The master node’s main duties include distributing resources

si
and providing direction to Node Managers so they may do real-time processing.
While an application manager manages job submissions and is in charge of running
application masters in a cluster, a scheduler is in charge of scheduling jobs and
allocating resources.
er
●● Node Manager: A slave daemon is managed by a node manager. It is responsible
for carrying out specific tasks on each node. To show whether the nodes are active
v
or inactive, it periodically sends out heartbeats.
●● Application Masters: They negotiate with the Resource Manager for resources and
ni

collaborate with node managers to execute tasks.

●● Containers: The hardware resources that may be accessed in a cluster, such
as CPUs, RAMs, disc space, etc., are referred to as containers. Apps are able to
U

access certain amounts of resources thanks to it.

ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 167

Oozie
Notes

e
An open-source Java web application called Apache OOzie is used to schedule
operations across a distributed cluster. A broad variety of jobs, including Hive, Map

in
Reduce, Pig and others, are supported by Oozie, which combines many processes into a
single unit. Oozie jobs are divided into three categories.
●● Oozie Workflow jobs: Oozie Workflow jobs are Directed Acyclic Graphs (DAGs) that

nl
provide a list of tasks to be carried out.
●● Oozie Coordinator tasks: Tasks created by the Oozie Coordinator are recurrent
Oozie Workflow jobs that are started when there is enough time and data.

O
●● Oozie Bundle: This tool enables you to manage the lifecycle of several workflow
jobs and coordinators by bundling them together.

4) Data Processing and Analysis

ty
This might be seen as the Big Data architecture’s nervous system. Another key
component of Hadoop is Map Reduce, which is largely responsible for data processing.
We’ll also talk about different software libraries that help with data processing and
analysis.

si
Map Reduce
Map Reduce is in charge of processing massive amounts of data in a parallel
er
distributed fashion. It performs two distinct functions: Map and Reduce. Map, as the
name suggests, always progresses to Reduce. In the Map step, the data is processed
and turned into key-value pairs or tuples. The map job’s output is provided into the
reducer as inputs. The intermediate data is sorted and arranged before being passed to
v
the reducer and the reduction then aggregates the key-value pair to produce a smaller
set of outputs. The final data is then saved to HDFS.
ni
U
ity

Source: [Link]

Map Reduce, like HDFS, uses a master-slave system to complete tasks. Each Name
m

node contains a Job tracker, which splits and monitors the client-submitted jobs. The jobs
are subsequently divided across the data nodes. These data nodes hold task trackers
and provide a pulse to indicate that the node is still alive. Job tracker records the entire
process in this manner. In the event that a data node fails, the task tracker transfers the
)A

job to another node, making the system fault-tolerant.

(c

Amity Directorate of Distance & Online Education

 168 Data Engineering

Notes

e
in
nl
O
ty
Source: [Link]

si
Pig
Yahoo developed Pig to analyse massive amounts of data. This is also what map-
er
reduce achieves, however one fundamental problem with Map Reduce is that it needs a
lot of code to carry out the necessary operations. This is the main motivation for why Pig
was made. The two main parts of it are the Pig Latin and Pig engine.
Pig Latin is a sophisticated language that may be used for analytical work. 200 lines
v
of map-reduce code and 10 lines of Pig Latin code may both be used to do the same
task. The pig engine helps transform the pig codes into internal map-reduce operations.
ni

This results in a simplification of the overall process. SQL and Pig Latin are both
programming languages.

Spark
U

The fact that map-reduce needs a sequential, multi-step process to run a job and
that it must read cluster data in order to do the operation and send it back to nodes
in order to finish the task is one of the main challenges with the method. Map-reduce
ity

activities have a high delay as a result, making them inappropriate for real-time analytics.
Spark was developed to make up for this shortcoming. Apache Spark and map-
reduce vary primarily in their capacity to process data in memory and their ability to
reuse data across several concurrent processes. This makes it approximately 100 times
m

speedier than Hadoop map-reduce for processing big amounts of data.

The Spark framework includes Spark core, Spark SQL, MLlib, streaming and
Graphx.
)A

●● Spark Core: This is in charge of managing memory, scheduling jobs, distributing

them, monitoring them and recovering from errors. It also had a relationship with
storage devices. Several programming languages, including Java, Scala, Python
and R, may access it thanks to APIs.
●● MLlib: A collection of computer algorithms for grouping, regression and other
(c

applications.
●● Streaming: It enables you to use batch analytics code to run real-time analytics on

Amity Directorate of Distance & Online Education

 Data Engineering 169

real-time data that you ingest in mini-batches from sources like Kafka, Twitter and
Notes

e
Flume.
●● Spark SQL: A distributed querying engine that is up to 100 times faster than map-

in
reduce at executing highly effective queries. It comes pre-configured to support a
number of data sources, including Hive, Cassandra, HDFS and others.
●● Graphx is a distributed graph processing unit that scales exploratory analysis, graph
computing and ETL.

nl
O
ty
si
er
Source: [Link]

Spark is a self-contained ecosystem. With Spark SQL for data access, streaming
for batch and real-time data processing and other features, it offers an independent
v
management system for its cluster.
ni

5) Data Access
After the data has been ingested from multiple sources and stored in cluster nodes,
the following step is to retrieve the pertinent information for our needs. Numerous pieces
U

of software exist that enable us to efficiently access data as and when it is needed.

Hive
To manage enormous volumes of data, Hive is a data warehouse solution that runs
ity

on top of HDFS and Map Reduce. Because Hive’s query language is similar to SQL, it is
simple to use. Hive queries are internally transformed into map-reduce or spark jobs that
are executed on the distributed node cluster of Hadoop.

Impala
m

An open-source data warehouse querying tool for massive volumes of data is called
Apache Impala. Syntactically, it is similar to HQL, however it does searches much more
quickly than Hive. It does not use map-reduce, unlike Hive and instead features a built-in
)A

engine that caches intermediate results to speed up query processing. Integration with
HDFS, Hbase and Amazon S3 is straightforward. In many aspects, Impala and SQL are
similar and there is not a steep learning curve with Impala.

Hue
(c

An open-source web interface for Hadoop components is called Apache Hue from
Cloudera. It provides a straightforward user interface for managing HDFS files and
directories, interfacing with Hive data stores, controlling map-reduce processes and

Amity Directorate of Distance & Online Education

 170 Data Engineering

monitoring Oozie workflows. This is the tool you use to connect with various Hadoop
Notes

e
components if you detest command line interfaces.

Zookeeper

in
Another significant element of the Hadoop family responsible for cross-node
synchronisation and coordination is Apache Zookeeper. For Hadoop applications,
cross-cluster services could be necessary; introducing Zookeeper solves this problem.

nl
Applications build znodes within Zookeeper and by modifying their status there, they may
coordinate their actions throughout the distributed cluster. Then, Zookeeper can alert
other nodes about a specific node’s status change.

O
4.1.1 Impact of Big Data and Scaling Challenges
Impact of Big Data on Business

ty
What effects do data mining and big data have on international business? Big data
is used by businesses to improve customer service, which might result in higher revenue.
The majority of companies strive to enhance client experiences. Improved target
marketing, cost savings and operational efficiency are some further goals.

si
Businesses may store enormous volumes of data with big data technology while
saving money. Hadoop and cloud-based analytics are two examples of this technology.
They aid businesses in data analysis and improved decision-making. Data breaches
er
also demand more security, which technology applications could provide. Big data has
the potential to provide businesses social and financial advantages. As a result, several
government agencies have created regulations to promote the expansion of big data.
v
With the development of agile technology and a greater focus on sophisticated
analytics, big data analytics have evolved through time. A single technology cannot
ni

encompass big data analytics. In order to help organisations make the most of their
information, many technologies work together. Examples include in-memory analytics,
in-machine learning, artificial intelligence, quantum computing, Hadoop and predictive
U

analytics. These technological advancements are anticipated to increase demand for big
data analytics throughout the course of the projection period.
Prior until now, big data was mostly used by businesses with the financial means to
pay for the channels and tools needed to gather and evaluate data. Both large and small
ity

firms are increasingly using big data to offer advanced business insights. They thus drive
up the need for big data. Businesses across a range of sectors are thinking about how to
use big data. Its applications are designed to boost productivity, detect customer wants,
provide businesses an advantage over rivals and open doors for long-term economic
growth.
m

Challenges of Big Data

In a broader context, here are 10 big data issues that organisations should be aware
)A

of and some solutions for them.

1. Managing large volumes of data

By definition, big data refers to enormous volumes of data spread across several
platforms and systems. According to Szybillo, merging the incredibly large data sets
(c

derived from CRM and ERP systems, as well as other data sources, into a coherent and
intelligible big data architecture is the first challenge for enterprises.

Amity Directorate of Distance & Online Education

 Data Engineering 171

He said that by making little adjustments, it is easier to pinpoint insights once you
Notes

e
have a sense of the data being gathered. To do this, design an architecture that enables
progressive modifications. Making significant changes might lead to the emergence of
new challenges. Moving a big data initiative from boardroom discussions to business

in
insights involves a number of challenges for businesses. To shepherd a big data
initiative from boardroom discussions to business insights, enterprises face a number of
challenges.

nl
2. Finding and fixing data quality issues
Analytics algorithms and artificial intelligence applications built on big data may give

O
subpar results when data quality issues creep into big data platforms. These problems
can get worse and harder to audit as data management and analytics teams work to
collect more and different types of data. Bunddler, an online marketplace for employing
web shopping assistants who assist people in making purchases and arranging

ty
deliveries, observed these problems firsthand when it expanded to 500,000 clients. A key
growth engine for the company was the use of big data to develop highly customised
experiences, reveal chances for upselling and track new trends. Effective data quality
control was a key issue.

si
According to Bunddler CEO Pavel Kovalenko, “You must constantly monitor and
correct any data quality issues.” He asserts that errors and duplicate entries are frequent,
particularly when data comes from several sources. To ensure the accuracy of the
er
data they collect, Kovalenko’s team created an intelligent data identifier that can detect
duplicates with minute data variations and identify any potential errors. As a result, data
analysis’s business insights are now more accurate.
v
3. Dealing with data integration and preparation complexities
ni

The difficulty of collecting and storing vast amounts of data of all kinds is handled by
big data platforms, as is the quick retrieval of data needed for analytics. Rosaria Silipo, a
Ph.D. and lead data scientist at open source analytics platform provider Knime, asserts
that data collection may still be challenging.
U

The integrity of an enterprise’s collected data repositories depends on how

frequently they are updated. This need ongoing access to a variety of data sources in
addition to certain big data integration tools.
ity

Some companies, without taking into account how the heterogeneous data will be
integrated, use a data lake as a catch-all repository for vast volumes of big data collected
from several sources. For instance, different business domains offer data that is helpful
for combined analysis, but this data often has inconsistent underlying semantics that
must be addressed. According to Silipo, initiatives involving ad hoc integration may need
m

a significant amount of rework. For the highest return on investment on big data efforts, it
is frequently better to adopt a strategic approach to data integration.
)A

4. Scaling big data systems efficiently and cost effectively

If businesses don’t have a plan for what to do with their massive amounts of data,
they risk wasting a lot of money. George Kobakhidze, head of corporate solutions at the
technology and services firm ZL Tech, asserts that businesses need to understand that
big data analytics starts with data entry. Since data from before the COVID-19 pandemic
(c

is usually no longer reliable in today’s market, curating company data repositories also
requires constant retention techniques to cycle out outdated information.

Amity Directorate of Distance & Online Education

 172 Data Engineering

Data management teams should thus map out the types, forms and uses of data
Notes

e
before implementing big data solutions. This is easier said than done, as Travis Rehl,
vice president of product at cloud management platform vendor CloudCheckr, points out.

in
“Often, you start with one data model and expand out, but you quickly realise the
model doesn’t fit your new data points and you suddenly have technical debt to resolve,”
he said.

nl
Data reuse may be done effectively and affordably with the help of a generic data
lake and the right data structure. In a data lake, Parquet files, for instance, usually offer a
better performance-to-cost ratio than CSV dumps.

O
5. Evaluating and selecting big data technologies
Data management teams can choose from a variety of big data technologies and the
features of the different solutions usually overlap.

ty
Lenley Hensarling, chief strategy officer at Aerospike, advises that teams start by
evaluating current and potential demands for data from batch and streaming sources
such mainframes, cloud applications and third-party data services. He lists Apache Kafka,
Apache Pulsar, AWS Kinesis and Google Pub/Sub as enterprise-grade streaming options

si
to think about since they all permit seamless data movement between cloud, on-premises
and hybrid cloud systems. er
Teams should next start evaluating the complex data preparation abilities required
to feed AI, machine learning and other advanced analytics technologies. Making
arrangements for the location of data processing is also required. Teams must think
about how to run analytics and AI models on edge servers while also making it simple
v
to update the models when latency is an issue. These capabilities must be compared
against the cost of setting up and running the hardware and software that operate on-
ni

premises, in the cloud, or at the edge.

4.1.2 Tools of Big Data and Beyond the Hype

U

1. APACHE Hadoop
It is an open-source platform for storing and analysing massive volumes of data
that is built on Java. Its cluster design enables the system to effectively and concurrently
ity

process data. It can move data from one server to many computers, both organised and
unstructured. Additionally, Hadoop offers cross-platform interoperability to its clients. It is
now the best big data analytical tool and various tech behemoths like Amazon, Microsoft,
IBM and others frequently use it. Features of Apache Hadoop include:
●● It is free to use and provides a cost-effective storage option for organisations.
m

●● Provides easy access using HDFS (Hadoop Distributed File System).

●● Extremely adaptable and simple to build with MySQL and JSON.
)A

●● Highly scalable because to its ability to distribute massive amounts of data in tiny
chunks.
●● It runs on low-cost commodity hardware such as JBOD or a collection of discs.

2. Cassandra
(c

Open source, distributed NoSQL database called APACHE Cassandra is used to

retrieve massive amounts of data. One of the most widely used data analytics tools, it
has received praise from several IT businesses for its excellent scalability and availability
Amity Directorate of Distance & Online Education
 Data Engineering 173

without compromising speed and performance. It has almost zero downtime and can
Notes

e
handle petabytes of resources while performing thousands of operations per second.
Facebook constructed it in 2008 and it became well known in 2009. The traits of APACHE
Cassandra are as follows:

in
●● Flexibility in Data Storage: This feature supports all data kinds, including structured,
unstructured and semi-structured data and enables users to make adjustments as
necessary.

nl
●● Data Distribution System: By duplicating data across many data centres, data may
be distributed easily.
●● Fast Processing: Cassandra is designed to run on effective commodity technology

O
while offering quick data processing and storage.
●● Fault tolerance: If a node malfunctions, a replacement will be sent out right away.

3. Spark

ty
APACHE Spark is yet another tool for processing data and carrying out several
tasks at once. Using tools for distribution, it is also used to examine data across several
machines. Because it offers straightforward APIs for data extraction and can handle

si
many petabytes of data, it is well-liked among data analysts. Recently, Spark broke
Hadoop’s previous world record (71 minutes) by processing 100 terabytes of data in
under 23 minutes. Due of Spark’s suitability for ML and AI today, big IT businesses are
turning to it. Features of APACHE Spark include:
er
●● User-friendliness: Users can choose their preferred language to run. (Java, Python,
etc.).
v
●● Spark Streaming is capable of handling real-time streaming.
●● Flexible: It can run on Kubernetes, Mesos, or the cloud.
ni

4. Mongo DB
A document-oriented (NoSQL) database and a free, open-source platform with
U

the capacity to store massive quantities of data have gained attention since 2010. Key-
value pairs, which are considered as Mongo DB’s fundamental building blocks, make
up the documents that contain the data that are stored in collections and documents.
Its popularity among programmers is due to the support it offers for a wide variety of
ity

programming languages, such as Python, Jscript and Ruby.

Features of MongoDB include:

●● It is a schema-less database that is written in C++ and can hold many types of
documents.
m

●● Simplifies Stack: By using mongo, a user may save files rapidly without messing up
the stack.
●● Master-Slave Replication: It may contact the slave for backup and write/read data
)A

from the master.

5. Apache Storm
An effective, user-friendly data analytics tool called a storm is frequently used in
small organisations. The best feature of the storm is that it supports all languages and
(c

has no programming language limitations. It was developed to handle enormous amounts

of data in fault-tolerant, scalable ways. Due to its distributed real-time massive data

Amity Directorate of Distance & Online Education

 174 Data Engineering

processing capability, Storm is the industry leader in real-time data processing, which is
Notes

e
why many IT giants use APACHE Storm in their systems today. Examples include Twitter,
Zendesk, NaviSite and other well-known companies. Storm possesses the following
traits:

in
●● Processing of Data: Storm keeps processing data even when a node is
disconnected.

nl
●● Highly Scalable: It keeps up performance even as load increases.
●● Quick: One node of APACHE Storm can process up to 1 million 100-byte messages,
making it incredibly quick.

O
6. SAS
It is now one of the most effective techniques data analysts use to create statistical
models. Using SAS, a data scientist may extract, organise, update and mine data
from several sources in multiple versions. Users of SAS, also known as the Statistical

ty
Analytical System, may access data in any format, including Excel workbooks and SAS
tables. In addition, it offers the SAS Viya cloud platform for business analytics and it
has unveiled new tools and technologies to better understand AI and ML. SAS features

si
consist of:
●● Versatile Programming Language: It is suitable for non-programmers due to its easy
syntax and sizable library.
●●
er
Extensive Data Format: It can read data in any format and supports a broad range
of programming languages, including SQL.
●● Encryption: End-to-end security is made possible via the SAS/SECURE function.
v
4.1.3 Introduction to the Hadoop Ecosystem
ni

One of the most important advances in Big Data is the Hadoop ecosystem. It
provides data storage and analysis tools, as well as a platform on which other firms may
build their apps. This means you may use the Hadoop ecosystem to develop apps or just
U

as an infrastructure tool for storing and analysing data. Because of its adaptability, it is an
excellent choice for enterprise-level businesses with vast volumes of data that must be
analysed effectively.
ity

What Is Hadoop?
Hadoop is a software framework for storing and processing massive volumes of data
on a cluster of computers. Hadoop comprises two distinct parts –

1. HDFS (Hadoop Distributed File System)

m

It allows you to distribute massive volumes of data over numerous servers, making it
easier to retrieve via requests from your clients or apps.
)A

2. MapReduce
It enables your computer to process these requests, no matter how huge, promptly.

What Is the Hadoop Ecosystem?

The Hadoop Ecosystem is a set of tools, libraries and frameworks that aid in
(c

the development of applications that run on top of Apache Hadoop. Hadoop offers
huge parallelism with low latency and great throughput, making it ideal for large data
challenges.
Amity Directorate of Distance & Online Education
 Data Engineering 175

The Hadoop ecosystem definition expands on these capabilities by including

Notes

e
additional tools to make it simpler to utilise Hadoop in conjunction with other frameworks
such as Spark or Kafka for real-time processing or machine learning activities.

in
Introduction To Hadoop Ecosystem For Big Data
Hadoop is a collection of technologies that work together to handle and analyse
massive volumes of data. It is an open-source framework with a globally dispersed

nl
system design. Among these tools are:

Map Reduce

O
This utility breaks the input into little bits, distributes them throughout the cluster and
aggregates the output from all computers into a single file.

Pig

ty
This tool enables you to create scripts in the Pig Latin programming language that
may be used to query big datasets stored in the Hadoop Distributed File System (HDFS).

Hive

si
Users can use this tool to store data in tables similar to those found in SQL
databases. Nonetheless, it is kept as files on HDFS rather than in relational database
management systems (RDBMS).
er
Hadoop Ecosystem Architecture With Diagram
Here’s a Hadoop ecosystem diagram -
v
ni
U
ity
m

Source: Understanding Hadoop Ecosystem: Architecture, Components & Tools ([Link])

The Hadoop ecosystem design consists of four major components: data storage,
processing, access and administration.
)A

1. Data Storage
The location of all your raw data is the first step in describing the Hadoop environment.
It might be on your computer or on the cloud.
2. Data Processing
(c

The second phase of the Hadoop ecosystem in Big Data is data analysis and
transformation into something useful that can be utilised for further analysis.

Amity Directorate of Distance & Online Education

 176 Data Engineering

3. Data Access
Notes

e
In this third phase of the Hadoop ecosystem, you may query your data sets with tools
like Hive or Pig and execute operations like filtering out certain rows, sorting them by

in
specified columns or values inside them (such as location or birthday) and so on.
4. Data Management
Finally, the final step of the Hadoop ecosystem design is storing all of the work we

nl
have done on data sets in prior phases somewhere secure so we may return to it later
if necessary.

Hadoop Ecosystem Fundamental Tools

O
Many tools for data processing and analysis are included in Hadoop and its
ecosystem. Some of these tools collect data from a variety of sources, while others store
and analyse the data.

ty
Here’s a rundown of the key Hadoop ecosystem tools you should be aware of:
1. Oozie - Workflow Management
Oozie is a workflow management solution that allows users to track and manage

si
processes. It may be used to automate a wide range of operations, including data
processing, system management and troubleshooting.
2. Chukwa - Supervision
er
Chukwa is a distributed monitoring solution for high-performance computing clusters
that is open source. Data is collected from Hadoop Distributed File System (HDFS),
MapReduce and YARN applications using this tool. It provides a web interface for
v
viewing data collected by Chukwa agents operating on each cluster node.
3. Flume – Monitoring
ni

Flume is a free and open-source distributed log collecting system that stores log
events from web servers and application servers in HDFS or other systems.
U

4. Zookeeper – Management
It is a management tool that assists Hadoop clusters with configuration management,
data synchronisation and service discovery.
5. Hive – SQL
ity

Hive is a Hadoop data warehouse solution that enables users to query data using
Structured Query Language (SQL). It may also be used to build and alter tables and
views, as well as grant people rights.
6. Pig – Dataflow
m

Pig is a high-level programming language used to create data transformation

programmes. It allows data analysis programmes to be expressed in the same
manner that humans communicate about their job. Pig programmes are compiled into
)A

MapReduce jobs, which are executed on the Hadoop infrastructure.

7. Mahout - Machine Learning
Mahout is a collection of machine-learning libraries that operate on Hadoop.
Many common methods, such as k-means clustering, naive Bayes classification,
(c

logistic regression, support vector machines (SVM), random forests and so on, are
implemented.

Amity Directorate of Distance & Online Education

 Data Engineering 177

8. MapReduce - Cluster Management

Notes

e
It is a popular programming approach for processing and handling huge datasets. It is
divided into two stages:

in
™™ Map Phase: We chunk the input data and process it in parallel.
™™ Reduce phase: Each intermediate key-value pair group is handed to a reducer,
which computes the final output depending on the values in that group.

nl
9. HBASE - Column DB Storage
HBase (Hadoop Base) is a free and open-source database that runs on the HDFS file
system. It offers scalable NoSQL storage for enormous amounts of unstructured data.

O
4.1.4 Understanding Hadoop and MapReduce
Using straightforward programming principles, Hadoop is a framework that enables

ty
the distributed processing of enormous data volumes across clusters of computers.
Hadoop is, in a word, a framework for managing “Big Data.” Developed by Doug Cutting,
Hadoop. It was also designed by Mike Cafarella. A single server may be scaled up to
thousands of units, each with its own computation and storage capabilities. A piece of

si
open-source software is called Hadoop. The Hadoop Distributed File System (HDFS) and
a processing component that may employ the Map-Reduce programming paradigm make
up Apache Hadoop’s fundamental components. Hadoop breaks up files during a cluster
er
and distributes them among nodes. After that, nodes get packed code and process the
data concurrently.
Mapreduce: On computer clusters, the MapReduce programming approach is used
to manage and produce massive data volumes. It was initially released by Google. A
v
concept or method for large-scale parallelization is called map-reduce. The functional
programming tools map() and reduce() served as inspiration for it. The MapReduce
ni

programming is carried out in the following three steps:

●● Mapping: A mapper’s responsibility is to process input data. Each node applies the
map function on the local data.
U

●● Shuffle: In this case, the distribution of nodes is determined by the output keys. The
output keys are produced by the map function.
●● Reduce: Nodes are now processed into each group of output data in parallel,
ity

according to the key.

Based on Hadoop MapReduce
Definition Using straightforward programming, the MapReduce is a programming
Apache Hadoop software enables the paradigm that implements a
m

distributed processing of enormous data distributed method for processing

volumes across computer clusters. and producing massive data sets
on a cluster.
Meaning The toy elephant that Doug Cutting’s son The skills of mapping and reducing
)A

had served as the basis for “Hadoop.” Since in key-value pairs are what gave rise
“Hadoop” is simple to say, he gave this project to the phrase “MapReduce”.
that name.
Framework Hadoop provides other frameworks like A c o m p u t e r s y s t e m c a l l e d
MapReduce in addition to a storage framework M a p R e d u c e u s e s k e y - v a l u e
(c

for storing data and another framework for mappings to sort and process data.
creating name nodes and data nodes.

Amity Directorate of Distance & Online Education

 178 Data Engineering

Invention Hadoop was created by Mike Cafarella and MapReduce was created by Google.
Notes

e
Doug Cutting.
Features An open-source project is Hadoop. MapReduce provides fault tolerance.

in
The Hadoop cluster may grow very quickly. There is excellent availability using
MapReduce.
Concept A dependable, scalable and ready-to-use A programming technique called
environment for distributed computing is MapReduce is a project submodule

nl
offered by the Apache Hadoop eco-system. used to process sizable datasets
kept on HDFS (the Hadoop
distributed file system).

O
Language Hadoop includes all modules, therefore it J a v a i s u s e d t o c r e a t e t h e
could also use other programming or scripting MapReduce framework.
languages.
Pre- Hadoop is powered by HDFS (Hadoop MapReduce may be operated on

ty
requisites Distributed File System). HDFS/GFS/NDFS or any other
distributed system, for example -
MapR-FS.

si
4.2 Components of the Hadoop Ecosystem
Introduction er
Hadoop is a distributed storage and parallel computing solution for storing and
managing massive volumes of data. It is the programme that data analysts use the most
frequently to manage vast amounts of data and its market share is increasing. Hadoop
consists of three components:
v
1. Hadoop HDFS - Hadoop Distributed File System (HDFS) serves as the storage
ni

system.
2. Hadoop MapReduce - Hadoop MapReduce serves as the processing engine.
3. The Hadoop resource management component YARN is Yet Another Resource
U

Negotiator.

4.2.1 Introducing HDFS, Hive, HBase and Spark

●● HDFS (Hadoop Distributed File System)
ity
m

Source: [Link]

Data in the form of files is kept in the Hadoop storage component. Each file is
divided into 128MB (adjustable) chunks and stored on several cluster workstations. It
)A

uses a master-slave architecture with the Name Node and the Data Node as its two main
building blocks.
●● Each cluster has a single master node, also known as the name node. Its
responsibility is to locate each file’s block in the cluster.
(c

●● Each cluster has a number of data nodes, which are slave nodes that store
data blocks. Its responsibility is to retrieve data as and when required. Through
heartbeats, it keeps in constant touch with the Name node.
Amity Directorate of Distance & Online Education
 Data Engineering 179

●● Hive
Notes

e
in
Source: [Link]

nl
Hive is a distributed data warehousing system developed by Facebook. It makes
managing, reading and writing files on HDFS straightforward. Its own querying language
for this is called Hive Querying Language (HQL), which is somewhat comparable to SQL.

O
This makes it reasonably simple for programmers to create MapReduce functions with
plain HQL queries.
●● HBase

ty
Similar to Google’s huge table, the HBase data format enables quick random access
to enormous amounts of structured data. It makes use of the fault tolerance of the
Hadoop File System (HDFS). It is a part of the Hadoop ecosystem that enables ad hoc

si
read/write access to data stored in the Hadoop File System in real time.
Either directly or through HBase, data may be stored in HDFS. The data consumer
randomly reads/accesses data from HDFS using HBase. A database called HBase, which
er
supports both read and write access, is built on top of the Hadoop File System.
v
ni
U
ity

Source: [Link]

●● Spark

Apache Spark
m

High-performance computing uses Apache Spark, a lightning-fast cluster computing

solution. Based on Hadoop MapReduce, it enhances the architecture to make it more
useful for many types of computations, including interactive queries and stream
)A

processing. In-memory cluster computing, which improves an application’s processing

performance, is Spark’s primary feature.
In addition to batch applications, iterative algorithms, interactive queries and
streaming, Spark is designed to handle a wide range of workloads. It minimises the
(c

administrative burden of maintaining several tools by supporting all of these workloads on

a single system.

Amity Directorate of Distance & Online Education

 180 Data Engineering

Evolution of Apache Spark

Notes

e
At UC Berkeley’s AMPLab, Matei Zaharia founded Spark, a Hadoop subsidiary
project, in 2009. In 2010, it was made available under a BSD licence. It was given to

in
the Apache Software Foundation in 2013 and as of February 2014, it is now a top-level
Apache project.

Features of Apache Spark

nl
Apache Spark has the following functionality.
●● Speed: Spark enables a Hadoop cluster to run an application up to 100 times faster

O
in memory and 10 times faster on disc. By reducing the number of disc read/write
operations, this is achieved. Data from the middle stages of processing are stored in
memory.
●● Supports a wide range of languages: Spark comes with Python, Java and Scala

ty
APIs. You may thus make apps in a multitude of languages. 80 high-level operators
are provided by Spark for interactive querying.
●● Advanced analytics are possible with Spark, which offers more than simply “Map”

si
and “Reduce.” Graph approaches, machine learning (ML), streaming data, SQL
queries and ML are all supported.

4.2.2 Other Modules in the Hadoop Ecosystem

er
YARN
The main orchestrator for Hadoop data processing is YARN. It is in charge of
v
overseeing the cluster’s resources. This involves choosing how to distribute tasks,
determining if nodes are available to handle additional burden and ensuring the cluster’s
ni

overall effectiveness and stability. YARN serves to provide seamless coordination and
functioning across the ecosystem’s numerous parts.

MapReduce
U

Exploiting YARN’s features allows for the development of the fascinating MapReduce
application. The second part of the Hadoop ecosystem is called MapReduce and it
offers a programming style that allows distributed data processing over an entire cluster.
ity

It functions by using Mappers and Reducers, two distinct scripts or functions utilised in
the construction of a MapReduce software. Reducers are in charge of aggregating the
changed data whereas Mappers are capable of simultaneously and highly efficiently
transforming data throughout the whole computer cluster. Although the MapReduce
paradigm seems simple, its flexibility is important because the pairing of Mappers and
m

Reducers enables the resolution of challenging problems. We will go more into the
mechanics of MapReduce in a later section of this Hadoop topic, exploring its capabilities
and potential uses.
)A

Apache Pig
The subsequent technology in the Hadoop ecosystem is called Apache Pig. A high-
level scripting language built on top of MapReduce is all that it is. Pig is for you if you
prefer using a scripting language with SQL-like syntax over writing MapReduce code in
Java or Python. A high-level programming API is used to enable the creation of simple
(c

scripts. You may receive complex responses without writing any Java code. Pig Latin will
translate the script into a format that is suitable with MapReduce. In other words, you

Amity Directorate of Distance & Online Education

 Data Engineering 181

may write your MapReduce code in Pig Latin, which is akin to SQL, rather than in Java.
Notes

e
As a result, you won’t need to run MapReduce tasks. Instead, functions for MapReduce
may be executed by writing simple Pig Latin code.

in
Apache Ambari
The next Hadoop ecosystem component, Apache Ambari, sits on top of everything
and offers a view of your cluster. It functions primarily as an open-source administrative

nl
tool for managing and tracking initiatives. It gives you access to information about your
cluster’s operating processes, active systems and resource use. Ambari gives you
a view on the present state of your cluster in terms of the apps that are using it as a

O
consequence. It is a management tool that will manage the health and monitoring of
several Hadoop [Link] in big data analytics

Mesos

ty
Though technically not a part of Hadoop, Mesos is included into the ecosystem as
a replacement for YARN. It is a resource negotiator, just as YARN. The same problem
is addressed differently by Mesos and YARN. The schedulers used by Mesos and
YARN are the main difference between them. When a job request is made in Mesos,

si
the Mesos master receives it, reviews the available resources and responds with offers.
These suggestions could be accepted or disregarded. Mesos provides a different way to
manage the resources in your cluster.
er
Spark, Apache
Spark is the most fascinating technology in this Hadoop context. It executes
queries on your data right beneath Mesos and at the same level as MapReduce.
v
It is basically a real-time data processing engine meant to provide analytics that are
speedier and easier to use than MapReduce. Spark is actively being developed and is
ni

extremely speedy. Due to the fact that it processes data in memory, it is a very powerful
technology. Spark may be used to reliably and effectively process data on a Hadoop
cluster. It is capable of handling SQL queries, machine learning across a whole data
U

cluster, streaming data and more.

Tez
Tez, which follows Spark in the Hadoop ecosystem and uses some of the same
ity

methods as Spark, is connected to Spark. As it creates a more effective strategy

to execute your queries, it illustrates how MapReduce functions. Tez has the effect of
enhancing Hive’s performance when used with it. Hive is built on top of MapReduce, but
because Tez is faster than MapReduce, it may also be built on top of Tez. They are both
different approaches to query optimisation.
m

4.2.3 Working with HDFS and HBase

●● HBase
)A

On top of the Hadoop file system, the column-oriented distributed database HBase
was created. It is a horizontally scalable open-source project.
Similar to Google’s huge table, the HBase data format enables quick random access
to enormous amounts of structured data. It makes use of the fault tolerance of the
(c

Hadoop File System (HDFS). It is a part of the Hadoop ecosystem that enables ad hoc
read/write access to data stored in the Hadoop File System in real time.

Amity Directorate of Distance & Online Education

 182 Data Engineering

Either directly or through HBase, data may be stored in HDFS. The data consumer
Notes

e
randomly reads/accesses data from HDFS using HBase. A database called HBase, which
supports both read and write access, is built on top of the Hadoop File System.

in
nl
O
ty
si
Source: [Link]

Storage Mechanism in HBase er

The tables in the column-oriented database HBase are arranged in rows. In the table
structure, only column families—key value pairs—are defined. A table has numerous
different column families and each one can have any number of columns. The disc stores
subsequent column data in the same sequence. A timestamp appears in the value of
v
each table cell. In an HBase, basically:
ni

™™ Rows are arranged in a table.

™™ A row is a collection of related column families.
™™ A collection of columns is known as a column family.
U

™™ Key-value pair sets are referred to as columns.

Features of HBase
™™ HBase can be scaled linearly.
ity

™™ It encourages automatic error.

™™ Consistent read and write performance is provided.
™™ It functions as both a source and a destination for Hadoop.
™™ Its client-side Java API is straightforward.
m

™™ Data replication across clusters is supported.

Where to Use HBase

)A

™™ Apache HBase is used to provide Big Data with random, real-time read/write
access.
™™ It hosts very big tables on commodity hardware clusters.
™™ Apache HBase is a non-relational database that was inspired by Google’s
(c

Bigtable. Bigtable runs on Google File System, whereas Apache HBase runs on
Hadoop and HDFS.

Amity Directorate of Distance & Online Education

 Data Engineering 183

Applications of HBase
Notes

e
™™ It is utilised anytime it is necessary to develop large applications.
™™ HBase is utilised anytime quick random access to accessible data is required.

in
™™ HBase is used internally by companies such as Facebook, Twitter, Yahoo and
Adobe.
●● HDFS

nl
How to use HDFS?
How do you utilise HDFS? In any case, HDFS runs on a commodity hardware cluster

O
that includes multiple extra data nodes in addition to a primary NameNode. These nodes
are clustered together inside the data centre. The blocks are then sent to the several
DataNodes for storage after being partitioned. In order to reduce the possibility of data
loss, blocks are regularly replicated between nodes. In the case that data is lost, it acts

ty
as a backup system.
Now let’s look at NameNodes. The cluster node that is aware of the data’s contents,
the block to which it belongs, the size of the block and its intended location is known as

si
the NameNode. Additionally, NameNodes are used to control file access, including who is
allowed to write, read, add, delete and duplicate data across data notes.
When there is a data surge, the cluster may also be modified in real-time according
er
to server capacity. As required, nodes can be added or withdrawn. We will now discuss
DataNodes. In order to decide whether to begin and complete a job, NameNodes and
DataNodes are in continual communication. The NameNode is keenly aware of the
condition of each DataNode as a result of this ongoing communication.
v
The namemode has the ability to rapidly transfer responsibility for a failing DataNode
to another functional node in the same datablock. Similar to this, DataNodes may
ni

communicate with one another, enabling them to take part in routine file operations.
The NameNode is essential to system maintenance since it is aware of DataNode
performance. Datablocks that have been copied over many datanotes are accessed by
U

the NameNode. You must first install and set up a Hadoop cluster before using HDFS.
For beginners, this may be a single node setup, whereas for large, scattered clusters, it
may be a cluster configuration.
ity

Features:
™™ Data storage that is distributed.
™™ Blocks shorten search time.
™™ Since the same block is present at numerous datanodes, the data is extremely
m

accessible.
™™ Even if numerous datanodes fail, we can still complete our task, making it very
dependable.
)A

™™ Excellent fault tolerance.

Limitations: While HDFS offers numerous functionalities, there are several instances
where it falls short.
●● Low latency data access: Applications that require low-latency data access, i.e. in
(c

the millisecond range, will not operate well with HDFS since HDFS is designed with
high-throughput data in mind, even at the expense of latency.

Amity Directorate of Distance & Online Education

 184 Data Engineering

●● Small file problem: Having a lot of small files means a lot of seeks and moving from
Notes

e
one datanode to another to obtain each small file; this is a particularly inefficient
data access pattern.

in
4.3 Introduction to Apache Spark
Introduction

nl
A general-purpose distributed cluster computing framework available for free is
called Apache Spark. Spark offers a programming interface with implicit data parallelism
and fault tolerance for whole clusters. A unified analytics engine for analysing lots of data

O
is called Apache SparkTM.

ty
si
v er
ni

Apache Spark – The unified analytics engine

U

Source: [Link]

Some of the most important elements of using Apache Spark are listed below.
●● It processes larger datasets about 100 times faster than conventional data
ity

processing methods. Spark’s in-memory processing engine enables it to be 100

times faster for data maintained in RAM and around 10 times faster for data saved
on disc when compared to MapReduce.
●● It comes with libraries that can immediately handle SQL, Graph and Streaming
m

APIs. Regular SQL queries may be used by users to query and analyse data and
the Spark data frame can be used to query data frames.
●● Since no new programming language is needed, using Spark is straightforward for
)A

programmers to begin. Although data pipelines may be constructed in Python, R, or

Scala, Scala is the primary language used in Spark. For generating Python Spark
data frames, the PySpark module is quite useful. The parts that follow will teach us
more about PySpark.

Understanding the Apache Spark Ecosystem Components

(c

Let’s now have a look at the various Spark components.

Amity Directorate of Distance & Online Education

 Data Engineering 185

Notes

e
in
nl
O
Components of Spark Core
Source: [Link]

According to the diagram above, Spark is built on top of its primary engine, the

ty
“Apache Spark Core.” This is the general-purpose execution engine that powers all of
Spark’s additional features. The Spark Core engine interacts with every other component,
including Spark SQL, Spark Streaming, MLlib and GraphX.

si
●● Spark SQL - The Spark processing engine’s widely used functionality. Users
may perform data analysis on massive datasets using the conventional SQL
language. Additionally, it allows us to run native Hive queries on existing Hadoop
er
configurations. Data transformation queries may also be extracted and carried out
using Spark SQL.
●● Spark Streaming - Using this special Spark capability, you can do analytics on top
of constantly updating streaming data. By applying certain procedures to the data,
v
it facilitates the study of enormous volumes of data as they come in. It continuously
uses the Spark Core engine to ingest data into a small cluster and execute RDD on
ni

the data.
●● MLlib - One of the most powerful things that most Spark users wish to use is MLlib,
or machine learning libraries. On top of the Spark Core engine, machine learning
U

algorithms are executed using MLlib. Faster than Apache Mahout’s previous
iteration, it trains data models using distributed in-memory data structures.
●● GraphX: On top of the Spark Core engine, the distributed graph data processing
ity

engine GraphX was created. By creating a Resilient Distributed Property Graph, it

enhances the Spark RDD’s capabilities. Vertices and edges connect the nodes and
their properties in this fundamental framework.

4.3.1 Why Use Apache Spark

m

High-performance computing uses Apache Spark, a lightning-fast cluster computing

solution. Based on Hadoop MapReduce, it enhances the architecture to make it more
useful for many types of computations, including interactive queries and stream
)A

processing. In-memory cluster computing, which improves an application’s processing

performance, is Spark’s primary feature.
In addition to batch applications, iterative algorithms, interactive queries and
streaming, Spark is designed to handle a wide range of workloads. It minimises the
(c

administrative burden of maintaining several tools by supporting all of these workloads on

a single system.

Amity Directorate of Distance & Online Education

 186 Data Engineering

Usage of Spark
Notes

e
●● Data integration: System-generated data is not consistently enough to be merged
for analysis. Techniques for extracting, transforming and loading (ETL) data from

in
systems can be utilised to get consistent data. Spark is used in this ETL procedure
to cut down on time and costs.
●● Stream processing: Managing data created in real-time, such log files, is always

nl
challenging. Spark can manage data streams and thwarts any fraudulent activity.
●● Machine learning: Machine learning techniques become more practical and
accurate as data amount rises. Spark makes it easy to work on machine learning
algorithms since it has the ability to store data in memory and execute repeated

O
queries quickly.
●● Interactive analytics: Spark has a short turnaround time for replies. So, rather than
running pre-defined queries, we could engage the data.

ty
Why would you want to use Spark?
Spark has some big pros:
™™ Quick data analysis, transformation and querying on large data sets.

si
™™ Spark offers far less disc reading and writing than MapReduce, as well as multi-
threaded tasks within Java Virtual Machine (JVM) processes (per Wikipedia:
threads share the resources of a single or multiple cores).
er
™™ Excellent for iterative algorithms, which base subsequent estimates on a
sequence of guesses.
™™ As was already said, simple APIs significantly improve development, readability
v
and maintenance.
™™ Exceptionally fast, especially for interactive queries. (Without restructuring,
ni

standard Hadoop Hive searches are 100x faster!)

™™ Supports integrations with other well-known companies and several languages.
™™ Aids in bringing coherence and simplicity to complex data streams.
U

4.3.2 Functional Programming Basics

The goal of the programming paradigm known as functional programming is to
ity

connect everything using just mathematical functions. It follows declarative programming

principles. In contrast to an imperative approach, which concentrates on “how to solve,”
it places a greater emphasis on “what to solve.” More phrases than statements are
used. Variables are assigned using statements, but values are created by evaluating
expressions. The distinctive characteristics of these functions are described in further
m

detail below.

Functional Programming is based on Lambda Calculus

The Lambda calculus framework was developed by Alonzo Church to study
)A

computations involving functions. It is referred to as the smallest programming language

in the world. It outlines what being calculable entails. Everything that can be computed
using lambda calculus is a computeable. It functions computationally in a manner
similar to a Turing machine. It offers a theoretical framework for outlining and assessing
functions. It serves as the basis for almost all current functional programming languages.
(c

Fact: Alonzo Church taught Alan Turing, who later created the Turing machine and
popularised the imperative programming style.
Amity Directorate of Distance & Online Education
 Data Engineering 187

Programming Languages that support functional programming: Haskell, JavaScript,

Notes

e
Python, Scala, Erlang, Lisp, ML, Clojure, OCaml, Common Lisp, Racket.

Concepts of functional programming:

in
1) Pure functions have two major characteristics. There are two main properties
of pure functions. First of all, they always deliver the same answer for the same
reasons, independent of other factors. Second, they don’t alter any arguments,

nl
local or global variables, or input or output streams since they have no side
effects.
2) Immutability is a later quality. A subsequent characteristic is immutability.

O
The sole result is the value that the pure function returned. They already
exist. Functional programming programmes are easy to debug because pure
functions have no hidden I/O or side effects. Additionally, creating concurrent/
parallel applications is made simpler by pure functions. When code is written

ty
in this manner, a clever compiler may take numerous actions, including
parallelizing the instructions, deferring result evaluation until it is necessary
and memorising the results because they are constant so long as the input is
constant..

si
3) Recursion: In functional languages, for” and “while” loops are absent from
functional languages. In functional languages, iteration is implemented through
er
recursion. Recursive functions repeat the process until the base case is
reached.
4) Referential transparency: variables in functional programmes that have
been declared do not change their values while they are running. Functional
v
programming do not employ assignment statements. We add new variables
whenever we need to save a value. All side effects are removed since any
ni

variable may be modified to reflect its real value at any time during execution.
The status of every variable remains constant at all times.
5) Functions are First-Class and can be Higher-Order: First-class variables are
U

handled as first-class functions. Variables of the first type can be provided as

parameters to functions, returned from functions, or kept in data structures.
Higher order functions are those that can accept other functions as inputs and
return functions.
ity

6) Variables are Immutable: Variables in functional programming are immutable,

meaning that once they have been initialised, they cannot be modified. The
ability to introduce new variables while preventing the editing of existing ones is
particularly helpful for keeping state throughout a programme. Once a variable
m

is created and its value is set, we can be sure that it won’t ever change.

4.3.3 Parallel Programming with Resilient Distributed DataSets

(RDDs)
)A

Spark’s main logical data units are RDDs. They are a dispersed group of items
stored in the memory or on the discs of various cluster machines. Several logical
divisions of a single RDD can be created, which can subsequently be stored and
processed on different cluster computers. By definition, RDDs are read-only and
(c

immutable. An original RDD cannot be altered, but you may create new RDDs by
performing coarse-grain operations, such transformations, on an existing RDD.

Amity Directorate of Distance & Online Education

 188 Data Engineering

Notes

e
in
nl
O
Source: [Link]
Distributed%20Datasets%20(RDDs),-RDDs%20are%20the&text=They%20are%20a%20distributed%20
collection,different%20machines%20of%20a%20cluster.

ty
Spark allows for the storage and reuse of RDDs for later updates, which is highly
advantageous for users. RDDs are thought to assess inadvertently, delaying evaluation
until it is absolutely essential. Time is saved and productivity is raised.

si
Features of an RDD in Spark
Here are a few Spark RDD functionalities:
●●
er
Resilience: In the case of a failure, RDDs automatically recover lost data by tracking
data lineage information. It also goes by the name of fault tolerance.
●● Distributed: An RDD’s data is dispersed among many nodes. It is spread among a
v
number of cluster nodes.
●● Lazy evaluation: Despite your data selection, an RDD is not loaded with it.
ni

Transformations are really computed when you run an operation, like count or
collect, or save the output to a file system.
U
ity
m
)A

Source: [Link]
Distributed%20Datasets%20(RDDs),-RDDs%20are%20the&text=They%20are%20a%20distributed%20
(c

collection,different%20machines%20of%20a%20cluster.

●● Immutability: An RDD’s data is read-only; it cannot be changed after it has been

stored. However, you may modify existing RDDs to create brand-new RDDs.
Amity Directorate of Distance & Online Education
 Data Engineering 189

●● In-memory computation: To enable quicker access, an RDD stores any immediately

Notes

e
produced data in memory (RAM) rather than on disc.
●● Partitioning: Any RDD that already exists may be partitioned to create logical parts

in
that can be changed. Applying adjustments to current divisions can achieve this.

5 Reasons on When to use RDDs

●● You need to have low-level access to, control of and manipulation of your dataset.

nl
●● Your data is unstructured and includes streams of text or audio;
●● You like to change data using functional programming structures as opposed to
expressions from a particular domain.

O
●● When processing or gaining access to data attributes by name or column, you are
not concerned with imposing a schema, such as columnar format; and
●● Some of the efficiency and optimisation advantages that DataFrames and Datasets

ty
offer for structured and semi-structured data may be skipped.

Advantages of RDD
RDD benefits Spark in a number of ways. We’ve covered a few of the most important

si
ones in this post:
●● RDD helps to increase Spark’s execution speed. er
●● RDDs, which are the basic building block of parallelism, support data consistency.
●● RDDs provide the execution and saving of individual tasks.
●● Because they could be utilised again, they are tenacious.
v
Limitation of RDD
●● RDDs do not support input optimisation.
ni

●● One of the most significant disadvantages of RDDs is that the execution process
does not begin immediately.
U

●● Once an RDD is formed, it cannot be changed.

●● RDD does not have adequate storage memory.
●● RDDs do not have run-time type safety.
ity

4.3.4 Scale-out and Data Parallelism in Apache Spark

Apache Spark is a sophisticated distributed processing platform for scaling out
data processing operations to huge clusters of processors. Spark enables scalability
by combining scale-out and data parallelism. Scale-out refers to the ability to add more
m

computers to a Spark cluster in order to expand the available computing power. This is
accomplished by spreading data and tasks across cluster nodes. The computing power
of the cluster grows proportionately as additional nodes are added.
)A

The capacity to break a data processing operation into smaller tasks that may be
completed in parallel is referred to as data parallelism. This is accomplished by dividing
the data into smaller chunks and allocating each chunk to a distinct job. The activities are
then completed concurrently, which considerably reduces total processing time.
Spark’s scale-out and data parallelism capabilities make it a very powerful tool for
(c

processing massive datasets. Spark can be scalable to tackle even the most demanding
data processing jobs by adding more machines to the cluster or splitting the data into
smaller chunks.
Amity Directorate of Distance & Online Education
 190 Data Engineering

●● Increased processing power: As more machines are added to the cluster, the
Notes

e
cluster’s processing power grows correspondingly. As a result, Spark can be scaled
to address even the most difficult data processing jobs.
●● Improved performance: Data parallelism in Spark may dramatically enhance the

in
performance of data processing activities. Because the activities are conducted in
parallel, the overall processing time is reduced.
●● Scalability: The Spark framework is extremely scalable. By adding more machines

nl
to the cluster or separating the data into smaller parts, it may be expanded to tackle
even the most demanding data processing jobs.
Here are some of the difficulties associated with employing scale-out and data

O
parallelism in Apache Spark:
●● Data partitioning: Data must be partitioned in an effective manner for parallel
processing. This can be difficult with huge datasets.

ty
●● Resource management: Spark must be able to properly manage the cluster’s
resources. This comprises assigning jobs to nodes, monitoring resource utilisation
and dealing with failures.
●● Communication overhead: Executing activities in parallel incurs some

si
communication overhead. For huge datasets, this can be a bottleneck.
Overall, Apache Spark is an effective framework for handling huge datasets. Its
scale-out and data parallelism capabilities make it a highly scalable and performant
er
solution. However, there are several drawbacks to adopting Spark, including data splitting
and resource management.

4.4 Spark DataFrames and SparkSQL

v
Introduction
ni

Dataframe
A distributed data collection with named columns is known as a dataframe. It has
sophisticated optimisation techniques and is theoretically equivalent to relational tables.
U

Hive tables, Structured Data files, external databases and pre-existing RDDs are just a
few of the sources from which a DataFrame might be created. For modern Big Data and
data science applications, this API was developed with inspiration from Pandas in Python
and DataFrame in R programming.
ity

SparkSQL
A Spark module for processing structured data is Spark SQL. The Spark SQL
interfaces provide more details about the structure of the data and the calculation than
m

the default Spark RDD API does. Internally, Spark SQL uses this additional knowledge
to do more optimisations. The Dataset API and SQL are two ways to communicate with
Spark SQL. Regardless of the API or language used to define the computation, the same
execution engine is employed to compute the result. This unity allows developers to
)A

swiftly transition between many APIs depending on which one offers the most natural way
to describe a certain change.

4.4.1 Introduction to DataFrames and DataSets

●● Datasets
(c

A data set is a collection of ordered data. Data is a collection of information that has
been obtained by observations, measurements, study, or analysis, as we all know. It

Amity Directorate of Distance & Online Education

 Data Engineering 191

could include data, figures, names, or even basic descriptions of things. Tables, charts
Notes

e
and graphs may all be used to organise the data for our research. Data mining is a tool
used by data scientists to assist in the analysis of gathered data.

in
A dataset is a group of numbers or values related to a certain subject. An example of
a dataset might be a student’s test results for a certain class. Datasets can be presented
as a table, a list of numbers in any order, or by enclosing them in curly brackets.
However, while dealing with data sets, you don’t always know what the data stands for

nl
and you don’t necessarily need to grasp what the data represents to solve the problem.
Generally, the data sets are labelled so that you understand what the data represents.

O
Types of Datasets
In statistics, many types of data sets are available for various types of information.
These are what they are:
™™ Numerical data sets

ty
™™ Bivariate data sets
™™ Multivariate data sets

si
™™ Categorical data sets
™™ Correlation data sets

Numerical Datasets
er
The numerical data set consists of information that is presented quantitatively rather
than verbally. The term “quantitative data” is sometimes used to describe numerical data.
The collection of all numerical and quantitative data is known as the numerical data set.
v
Since the numerical data is always presented as numbers, we may perform arithmetic
operations on it.
ni

™™ An individual’s height and weight

™™ A report on a patient’s RBC count
U

™™ A book’s amount of pages

Bivariate Datasets
A data set with two variables is referred to as bivariate. It is focused on the
ity

relationship between the two variables. A bivariate dataset frequently has two different
related data types. for example, to determine the age and proportion of students in a
class. Two factors to take into account are score and age. Sales of ice cream in relation
to the day’s temperature. The two factors used in this are ice cream and temperature.
(Note: A dataset is referred to as a univariate dataset if it just contains one set of
m

data, like temperature.)

Multivariate Datasets
)A

a collection of data with many different variables. A dataset is said to as multivariate

if it contains three or more different types of data (variables). The multivariate dataset,
in other words, consists of individual measurements that were collected as functions of
three or more parameters. For instance, we need to use a lot of variables to distinguish
between the dimensions of a rectangular box’s length, width, height and volume.
(c

Categorical Datasets
Categorical data sets represent the characteristics of an individual or an entity.

Amity Directorate of Distance & Online Education

 192 Data Engineering

The qualitative variable, also known as the categorical variable, which makes up the
Notes

e
categorical dataset, can only have two values. It is hence referred to as a dichotomous
variable. Categorical data or variables with more than two possible values are said to be
polytomous. Qualitative and categorical variables are typically assumed to be polytomous

in
unless specifically stated differently.

Example:

nl
™™ A person’s gender (male or female)
™™ Marital status (married/unmarried)

Correlation Datasets

O
Data sets for correlation are collections of values that relate to one another in
some way. In this instance, it is revealed that the values are interdependent. A statistical
link between two items or variables is how correlation is frequently defined. You could

ty
occasionally be asked to predict how two variables will interact. Understanding
correlation’s operation is crucial. Correlations can be of three different types. These are
what they are:

si
™™ Positive correlation - when two variables move in the same direction (either up
or down).
™™ Negative correlation - The movement of two variables in opposing directions.
er
(One variable is higher than the other and vice versa.)
™™ No or little correlation - There is no link between two variables.
As an illustration, a tall person is perceived as being heavier than a small person.
v
The factors of height and weight are interrelated in this situation.

Mean, Median, Mode and Range of Datasets

ni

The mean, median, mode and range are some of the fundamental concepts in
statistics. In other words, calculating the mean, median and mode is one of the three
methods for handling data sets. However, we must first prepare our data set by rewriting
U

it from least to greatest in ascending order before we can compute these three dataset
metrics. The mean of a dataset is the average of all the table’s observations. It is the
proportion between the total number of data collecting items and the total number of
observations. The mean formula can be found in;
ity

Mean = Sum of Observations / Total Number of Elements in Data Set

When a dataset is sorted in both ascending and descending order, the median
represents the collection’s middle value. The variable, number, or value that appears
the most frequently in a dataset is considered to be its mode. The space between the
m

greatest and lowest values in a dataset is known as its range.

Maximum Value - Minimum Value = Range
)A

●● Dataframe
What is a DataFrame?
A DataFrame is a programming abstraction in the Spark SQL module. Because
the data is stored in rows and columns of different datatypes, DataFrames resemble
relational database tables or excel spreadsheets with headers.
(c

Amity Directorate of Distance & Online Education

 Data Engineering 193

Notes

e
in
nl
Source: [Link]
is%20a%20DataFrame%3F,and%20columns%20of%20different%20datatypes.

O
Both complex user-defined functions and common data manipulation techniques
like sorting, joining, grouping and so on are used to complete processing. For distributed
data, schemas are employed to structure the information. The column name, datatype
and nullable properties are all parts of a DataFrame column. If nullable is set to true for a

ty
column, null properties are permitted.

si
er
Source: [Link]
v
is%20a%20DataFrame%3F,and%20columns%20of%20different%20datatypes.

How Does a DataFrame Work?

ni

The DataFrame API is a part of the Spark SQL module. The API interfaces with
languages like Java, Python and Scala and provides straightforward data processing
within the Spark SQL framework. Despite certain parallels between Python Pandas and R
U

data frames, Spark achieves something different. This API has various optimisations and
is made to deal with enormous volumes of data for data science and machine learning.
Spark DataFrames are Catalyst-optimized and may be deployed across several clusters.
ity

The Catalyst optimizer creates the optimum parallel processing plan from queries
(including SQL commands run on DataFrames).
m
)A
(c

Source: [Link]
is%20a%20DataFrame%3F,and%20columns%20of%20different%20datatypes.

Amity Directorate of Distance & Online Education

 194 Data Engineering

If you’ve worked with data frames in Python or R, the Spark DataFrame code
Notes

e
will appear familiar. On the other side, if you utilise Spark RDDs (Resilient Distributed
Datasets), knowing the data structure provides options for optimisation.

in
Spark’s architects created DataFrames to address big data concerns in the most
efficient way possible. With familiar but optimised APIs, developers can leverage the
potential of distributed computing.

nl
Features of Spark DataFrames
Spark DataFrame includes several useful features, including:
●● Hive, CSV, XML, JSON, RDDs, Cassandra, Parquet and more data formats are

O
supported.
●● Integration with many Big Data tools is supported.
●● Processing kilobytes of data on smaller devices and petabytes on clusters.

ty
●● Catalyst optimizer for fast data processing in different languages.
●● Structured data processing using a data schematic perspective.
●● Enhanced memory management to avoid overload and increase performance over

si
RDDs.
●● Java, R, Python and Spark APIs.
er
4.4.2 Catalyst and Tungsten Optimisations
Catalyst and Tungsten are two significant Apache Spark optimisation frameworks
that assist to increase the speed of Spark applications.
v
●● Catalyst is a query optimizer that employs a number of approaches to increase the
performance of Spark queries. Among these methods are:
ni

™™ Predicate pushdown, which directs predicates to data sources to prevent

wasteful data processing.
™™ Column pruning, which removes columns that are no longer required for the
U

query.
™™ Constant folding, which evaluates constant expressions at compile time to avoid
computation.
ity

™™ Plan reordering, which reorders query plan steps to increase speed.

●● Tungsten is a Spark execution engine that focuses on increasing memory and CPU
efficiency. Tungsten accomplishes this by:
™™ Making use of off-heap memory, which may be accessed more quickly than
m

heap memory.
™™ Creating optimized code that the CPU can run more efficiently.
™™ Making use of vectorized operations, which may conduct operations on
)A

numerous data items at once.

Tungsten and Catalyst collaborate to increase the performance of Spark
applications. Tungsten creates optimised code for the physical query plan whereas
Catalyst optimises the logical query plan. This set of optimisations has the potential to
greatly increase the speed of Spark applications.
(c

The following are some of the advantages of employing Catalyst and Tungsten
optimisations:

Amity Directorate of Distance & Online Education

 Data Engineering 195

●● Better performance: Catalyst and Tungsten may dramatically increase the speed
Notes

e
of Spark applications by lowering the amount of data that must be processed and
running queries more effectively.

in
●● Reduced memory usage: Tungsten employs off-heap memory, which may be
accessed more efficiently than heap memory. This can aid in reducing the amount of
memory used by Spark applications.
●● Improved scalability: Tungsten can create optimised code for a number of hardware

nl
platforms, which improves scalability. As Spark applications are deployed to bigger
clusters, this can assist to increase their scalability.
Overall, Catalyst and Tungsten are two essential optimisation frameworks that may

O
assist Spark applications enhance their speed, memory utilisation and scalability.

4.4.3 ETL with DataFrames

ty
ETL, which stands for Extract, Transform and Load, is a method of transferring data
from one area to another. ETL may be used in the context of DataFrames to:
●● Extract data from a number of sources, such as CSV files, databases and APIs; and

si
●● Transform the data to make it more consistent, accurate and usable. Cleaning the
data, deleting duplicates and performing statistical modifications are all examples of
this. er
●● Transfer the data to a data warehouse or another location for further study.
Here’s an example of how to do ETL in Python with DataFrames:

Python
v
import pandas as pd
ni

# Extract the data from a CSV file

df = pd.read_csv(“[Link]”)
U

# Transform the data

df = [Link]()
df = [Link](columns={“Column1”: “New_Column1”})
ity

# Load the data into a database

df.to_sql(“my_table”, “my_database”)
This code will first read the data from the CSV file “[Link]” and generate
m

a DataFrame from it. Then it will remove any rows with missing data and rename the
“Column1” column to “New_Column1”. Finally, the DataFrame will be loaded into the
“my_table” table in the “my_database” database.
)A

Here are a few advantages of utilising DataFrames for ETL:

™™ DataFrames are a versatile and effective tool for working with data.
™™ They are simple to use and may be altered in a number of ways.
™™ A variety of libraries are available to assist you in performing ETL using
(c

DataFrames.
DataFrames are an excellent approach to execute ETL when working with data in

Amity Directorate of Distance & Online Education

 196 Data Engineering

Python. They are powerful, adaptable and simple to use. A variety of libraries are now
Notes

e
available to assist you in automating the ETL process.

4.4.4 Real-World Usage of SparkSQL

in
SparkSQL is a strong query and analysis tool for huge datasets. It has a wide range
of real-world applications, including:

nl
●● Data warehouses and data marts: SparkSQL may be used to create data
warehouses and data marts. This enables organisations to centrally store and
analyse massive volumes of data.

O
●● Business intelligence (BI): SparkSQL may be used to create BI applications. By
giving insights into their data, these applications may assist organisations in making
better decisions.
●● Machine learning: Machine learning models may be built using SparkSQL. These

ty
models can be used to forecast future events or to categorise data.
●● Streaming data: SparkSQL can handle streaming data. This enables businesses
to analyse data as it is created, which may be valuable for applications like fraud
detection and anomaly detection.

si
Here are some concrete instances of SparkSQL’s use in real-world applications:
●● Netflix: Netflix analyses streaming data from its consumers using SparkSQL.
er
This information is used to recommend films and television series to users and to
improve the overall user experience.
●● Amazon: SparkSQL is used by Amazon to analyse data from their e-commerce
platform. This information is then utilised to improve product suggestions, pricing
v
and inventory management.
●● Uber: Uber analyses data from their ride-sharing platform using SparkSQL. This
ni

data is utilised to improve the platform’s efficiency as well as to give insights into
rider and driver behaviour.
These are just a few instances of how SparkSQL is utilised in the real world.
U

SparkSQL is becoming an increasingly significant tool for organisations that need to

analyse massive datasets as the volume and velocity of data continues to rise.
The following are some of the advantages of utilising SparkSQL:
ity

●● Speed: SparkSQL is a lightning-fast data-querying and analysis tool. This is due to

the fact that it processes data in parallel using a distributed computing environment.
●● Adaptability: SparkSQL can query data from a broad range of sources, including
structured, semi-structured and unstructured data.
m

●● Scalability: SparkSQL can handle very huge datasets. This makes it an excellent
alternative for organisations that require huge data analysis.
SparkSQL is a wonderful alternative to explore if you need a strong and versatile tool
for searching and analysing huge databases.
)A

4.5 Apache Spark Development and Runtime Environment

Introduction
(c

●● Apache Spark
Apache Spark is a powerful and versatile cluster computing technology. It has high-
level Java, Scala, Python and R APIs, as well as an optimised engine that supports
Amity Directorate of Distance & Online Education
 Data Engineering 197

broad execution graphs. It also includes a comprehensive collection of higher-level tools,

Notes

e
such as Spark SQL for SQL and structured data processing, MLlib for machine learning,
GraphX for graph processing and Spark Streaming.

in
Apache Spark development is the process of generating and sustaining Apache
Spark-based applications. This can include a wide range of tasks, such as:
™™ Spark application design and architecture

nl
™™ Coding Spark in Java, Scala, Python, or R
™™ Debugging and testing Spark applications
™™ Production deployment of Spark apps

O
™™ Managing and monitoring Spark applications
The following are some of the advantages of being an Apache Spark developer:
™™ Because Apache Spark is a well-known and frequently used platform, there is a

ty
great need for Spark engineers.
™™ Spark is a strong platform for solving a wide range of big data challenges.
™™ Because Spark is a well-documented and maintained technology, there are

si
several resources available to assist you in learning about it.
If you want to work with big data, being an Apache Spark engineer is an excellent
er
place to start. Spark is a versatile and powerful platform that may be used to tackle a
wide range of challenges. A lucrative career as a Spark developer is possible if you have
the necessary abilities.
●● Runtime Environment
v
A runtime environment is a software environment that contains the tools and libraries
ni

required for a programme to run. Apache offers several runtime environments, including:
™™ Apache Tomcat: A web server capable of hosting Java web applications.
™™ Apache Hadoop: A massive data processing framework and distributed file
U

system.
™™ Apache Spark: A general-purpose, rapid cluster computing solution.
™™ Apache Mesos: A cluster management solution for Apache Hadoop, Apache
ity

Spark and other distributed applications.

The runtime environment that you require will be determined by the programme
that you are executing. For example, if you want to execute a Java web application, you
must utilise Apache Tomcat. You may need to use Apache Hadoop, Apache Spark, or
Apache Mesos if you are operating a distributed application. Follow the procedures in
m

the documentation for the individual runtime environment that you are using to add an
Apache runtime environment to your machine.
You can execute apps that are compatible with the Apache runtime environment
)A

after you have installed it to your system. For example, if you have installed Apache
Tomcat on your PC, you may execute Tomcat-compatible Java web applications.
The following are some of the advantages of utilising Apache runtime environments:
™™ They provide a stable and dependable environment in which to operate
(c

programmes.
™™ They make application deployment and management simple.

Amity Directorate of Distance & Online Education

 198 Data Engineering

™™ They provide a variety of capabilities that can assist increase application speed
Notes

e
and scalability.
If you are creating or deploying applications that must operate on a distributed

in
platform, an Apache runtime environment is a terrific approach to simplify the process
and increase application performance.

4.5.1 Apache Spark Architecture Overview

nl
Spark Architecture
The Apache Spark base architectural diagram is shown in the graphic below:

O
ty
si
v er
Source: [Link]

The real programme of an application is called by the driver programme of the

ni

Apache Spark architecture, which also creates a SparkContext. All of the core operations
are included in SparkContext. The DAG Scheduler, Task Scheduler, Backend Scheduler
and Block Manager are further components of the Spark Driver that turn user-written
U

code into jobs that are executed on the cluster.

The Cluster Manager is in charge of monitoring the cluster’s numerous jobs while
they are being carried out. The Cluster Manager and the Spark Driver work together to
coordinate the execution of several additional tasks. Resources for work are distributed
ity

by the cluster manager. Once the task has been broken up into smaller tasks and given
to worker nodes, SparkDriver will control the execution.
Multiple worker nodes may process an RDD created in the SparkContext and the
processed data may then be cached. Information about tasks is sent from the Cluster
m

Manager to the Spark Context, which queues it on worker nodes. The executor is in
charge of handling these duties. Executors share the Spark Application’s lifespan. The
performance of the system can be enhanced by increasing the workforce. This allows us
)A

to divide the task into more manageable portions.

4.5.2 Apache Spark Cluster Modes

The main programme (sometimes referred to as the driver programme), which
is where Spark applications are run on a cluster, controls these independent groups of
(c

processes via the SparkContext object.

The SparkContext may connect to a number of cluster managers, which distribute

Amity Directorate of Distance & Online Education

 Data Engineering 199

resources across applications, in order to function on a cluster (including Spark’s own

Notes

e
standalone cluster manager, Mesos, YARN and Kubernetes). When Spark is linked,
it receives executors—processes that carry out calculations and store data for your
application—on cluster nodes. Your application code (defined by JAR or Python files

in
submitted to SparkContext) is subsequently transmitted to the executors. SparkContext
assigns tasks to the executors for execution before finishing.

nl
O
ty
Source: [Link]

There are several crucial elements of this architecture to take into account, including:

si
●● Each application contains a set of executor processes that run throughout the
duration of the programme and carry out duties in several threads. This has the
er
benefit of isolating apps from one another in terms of execution (tasks from different
applications run in distinct JVMs) and task scheduling (each driver schedules
its own tasks). However, it implies that data sharing between different Spark
applications (instances of SparkContext) cannot happen without writing to an
v
external storage system.
●● The administration of the underlying cluster is unimportant to Spark. As long as the
ni

cluster management system can acquire executor processes and allow them to
communicate with one another, it may be easily operated on a cluster that supports
other applications (such as Mesos/YARN/Kubernetes).
U

●● The driver programme must continuously wait for and accept connections from its
executors (for an illustration, see the network setup section’s [Link]). As a
result, the worker nodes in the network must be able to communicate with the driver
software.
ity

●● The driver should be installed close to the worker nodes, ideally on the same local
area network, as it schedules jobs on the cluster. Open an RPC to the driver and
have it submit operations from close by if you wish to send requests to the cluster
remotely rather than operating a driver far from the worker nodes.
m

Types of Cluster Managers

The system now supports the following cluster managers:
)A

●● Standalone, a sparse cluster manager included with Spark to make cluster setup
easier.
●● Apache Mesos, a general-purpose cluster manager that can manage applications
and execute Hadoop MapReduce. (Deprecated)
●● Hadoop YARN, which manages resources for Hadoop 2 and 3.
(c

●● Kubernetes, an open-source programme for automating the deployment, scaling

and management of containerized applications.

Amity Directorate of Distance & Online Education

 200 Data Engineering

4.5.3 Applications of Running Apache Spark

Notes

e
Applications of Apache Spark
Since its creation in 2009 and conversion to open-source technology, Apache Spark

in
has completely revolutionised the Big Data industry. With more than 200 contributors, it
has developed into one of the largest open-source communities. Its success is mostly
due to its ability to examine massive volumes of data more quickly than ever before.

nl
O
ty
si
Source: [Link]
er
The technology known as Spark is employed in practically every business. Here are
some of the most widely used Apache Spark applications:
●● Machine learning: The scalable Machine Learning Library, or MLlib, in Apache Spark
is capable of doing complex analytics like clustering, classification, dimensionality
v
reduction and other tasks. Spark is an intelligent technology because it excels at key
analytics tasks including sentiment analysis, customer segmentation and predictive
ni

analysis.
●● Fog computing: With the advent of big data concepts, IoT has emerged as a key
area for the creation of more cutting-edge technology. This technology handles
U

enormous volumes of data arriving from several sources and is based on the idea
of connecting digital equipment with the help of small sensors. Parallel processing
is required by Spark, which is impossible with cloud computing. As a result, Spark
Streaming is used in Fog computing, which decentralises data and storage, as a
ity

solution to this problem.

●● Event detection: In order to safeguard their systems, businesses may keep track
of strange and unusual activities with the aid of the Spark Streaming function.
Institutions including financial, security and health organisations employ triggers to
m

identify potential dangers.

●● Interactive analysis: Apache Spark’s capability to provide interactive analysis is
one of its most outstanding features. Apache Spark processes data more fast than
)A

MapReduce, which enables batch processing, enabling it to execute exploratory

queries devoid of sampling.

4.5.4 Using Apache Spark on IBM Cloud

More and more clients are using Apache Spark to handle all the labor-intensive tasks
(c

associated with the four steps of the AI ladder—collect, organise, analyse and infuse.
IBM Cloud Pak for Data is a native cloud solution that enables you to use your data
quickly and effectively.
Amity Directorate of Distance & Online Education
 Data Engineering 201

IBM Cloud Pak for Data

Notes

e
Your business has data. A lot of information is available. Utilise your data to generate
pertinent insights that will help you avoid obstacles and accomplish your goals.

in
It’s useless, though, if you can’t access or trust your data. By connecting to your
data, controlling it, locating it and analysing it, IBM Cloud Pak for Data helps you to
accomplish both. Additionally, IBM Cloud Pak for Data enables seamless communication

nl
between all of your data users with a unified interface that supports a wide range of
services that are designed to work together. On your IBM Cloud Pak for Data cluster, a
variety of workloads may be run using Analytics Engine powered by Apache Spark.

O
™™ Spark applications that run Spark SQL
™™ Data transformation jobs
™™ Data science jobs

ty
™™ Machine learning jobs

si
v er
ni
U

Source: [Link]
e0e5d68b5e46
ity

Analytics Engine powered by Apache Spark

A unique Spark cluster is created for each task that you submit. It is possible to
specify the size of the executor, the size of the Spark driver and the number of executors
m

for the job. As a consequence, your output will be dependable and constant.
Your Spark Jobs may be run on any Red Hat OpenShift cluster, whether it is on-
premises or in the cloud, thanks to IBM Cloud Pak for Data. atop the sky If you have an
)A

OpenShift deployment on IBM Cloud, AWS, Microsoft Azure, or Google Cloud, you may
install Cloud Pak for Data on your cluster. To prevent accidentally releasing your data, it
makes logical to keep the applications that access it behind your firewall if the majority of
your company data is maintained inside.
We use a brand-new approach that, in contrast to conventional Hadoop/Spark
(c

clusters, enables each activity to be handled by a specifically tailored, quickly appearing

Spark cluster on RedHat [Link] cluster is promptly cleaned up when a job is

Amity Directorate of Distance & Online Education

 202 Data Engineering

completed so that resources may be utilised by next operations. Additionally, the service
Notes

e
offers APIs for evaluating and troubleshooting your Spark applications’ performance.

4.6 Monitoring and Tuning Apache Spark

in
Introduction
Monitoring and adjusting Apache Spark is critical to ensuring that your applications

nl
perform smoothly and efficiently. You may monitor and modify your Spark applications
using a variety of tools and methodologies, including:
●● Spark Web UI: The Spark Web UI is a graphical interface for monitoring your Spark

O
applications. A number of metrics are available, including the number of tasks
running, the quantity of data processed and the time it takes to finish jobs.
●● Spark History Server: The Spark History Server saves data about your Spark

ty
applications, including metrics gathered by the Spark Web UI. The Spark History
Server allows you to see historical data about your Spark applications.
●● Spark Metrics API: The Spark Metrics API allows you to collect metrics from your
Spark applications programmatically. The Spark Metrics API may be used to gather

si
custom metrics or metrics that are not accessible in the Spark Web UI or the Spark
History Server. er
●● Spark Tuning Guide: The Spark Tuning Guide gives a complete review of the
aspects that might impact Spark application performance. The topic contains
suggestions for optimising Spark applications for a range of contexts.
Here are some crucial metrics to keep an eye on while customising your Spark
v
applications:
●● Executor cores: The number of cores assigned to each executor.
ni

●● Executioner memory: The amount of memory assigned to each executor.

●● Shuffle partitions: The number of shuffle partitions that your programme uses.
U

●● Shuffle size: The quantity of data that your programme shuffles.

●● Network bandwidth: The amount of network bandwidth that your application
consumes.
●● IO latency: The time it takes for your programme to conduct I/O operations.
ity

Once you’ve found the metrics influencing the performance of your Spark
applications, you can make changes to your configuration settings using the Spark
Tuning Guide. You may increase the performance and efficiency of your Spark apps
by tweaking them. Here are some more monitoring and tweaking recommendations for
m

Apache Spark:
●● To monitor your Spark applications, use the Spark Web UI and the Spark History
Server. These tools give a graphical interface via which you may see metrics and
)A

troubleshoot issues.
●● Collect custom metrics with the Spark Metrics API. This API collects metrics that
aren’t available in the Spark Web UI or the Spark History Server.
●● Make changes to your setup settings using the Spark Tuning Guide. The topic offers
suggestions for optimising Spark applications for a range of contexts.
(c

●● Regularly monitor your Spark apps. This will assist you in identifying performance
issues early and making necessary modifications.

Amity Directorate of Distance & Online Education

 Data Engineering 203

4.6.1 The Apache Spark User Interface

Notes

e
The Apache Spark User Interface (UI) is a web-based tool that allows you to monitor
and manage Spark applications visually. Any web browser that can connect to the Spark

in
master node may access the Spark UI. The Spark UI is divided into tabs that display
several perspectives of the Spark application. Some of the more essential tabs are as
follows:

nl
●● Jobs: This page displays a list of all the jobs sent to the Spark cluster. You can see
the status of each job as well as the tasks that are currently running for that job.
●● Stages: This page displays a list of all the stages completed for the current task.

O
You can see the stage’s status as well as the tasks that are currently running for that
stage.
●● Tasks: This page displays a list of all the tasks presently active in the Spark cluster.
You can see the status of each task as well as the data that each job is processing.

ty
●● Storage: This tab displays the data that is stored in the Spark cluster. You can see
the files saved in the cluster as well as the partitions utilised to store each file.
●● Environment: This tab contains information about the environment in which

si
the Spark application is executing. You may inspect the Spark cluster’s setup
parameters as well as the hardware resources accessible to the cluster.
The Spark User Interface (UI) is a useful tool for monitoring and managing Spark
er
applications. It may be used to debug issues, improve performance and gain insight into
how the Spark application is operating. Here are some more hints for utilising the Spark
UI:
v
●● Monitor the status of your Spark apps with the Spark UI. This will assist you in
identifying problems early and taking necessary action.
ni

●● To troubleshoot issues, use the Spark UI. If you are experiencing issues with your
Spark application, you may utilise the Spark UI to inspect the logs and pinpoint the
source of the issue.
U

●● To optimise performance, use the Spark UI. You may use the Spark UI to check
stats for your Spark apps and make configuration changes to optimise performance.
●● Use the Spark UI to see how your Spark apps are doing. The Spark UI may provide
you with useful information about how your Spark apps use resources and perform.
ity

4.6.2 Monitoring Jobs and Debugging Parallel Jobs

Monitoring and troubleshooting concurrent processes in Apache Spark may be
difficult, but there are a variety of tools and strategies available to assist you. Here are a
m

few of the most prevalent approaches:

●● Make use of the Spark Web UI. The Spark Web UI provides a graphical interface
for monitoring your Spark operations. A number of metrics are available, including
)A

the number of tasks running, the quantity of data processed and the time it takes
to finish jobs. You may also utilise the Spark Web UI to troubleshoot issues by
checking task logs.
●● Make use of the Spark History Server. The Spark History Server keeps data about
your Spark operations, including metrics gathered by the Spark Web UI. You may
(c

access past data about your Spark tasks using the Spark History Server. This might
be useful for troubleshooting occasional difficulties.

Amity Directorate of Distance & Online Education

 204 Data Engineering

●● Make use of the Spark Metrics API. The Spark Metrics API allows you to collect
Notes

e
metrics from your Spark tasks programmatically. The Spark Metrics API may be
used to gather custom metrics or metrics that are not accessible in the Spark Web
UI or the Spark History Server. This might be useful for diagnosing difficulties unique

in
to your application.
●● Make use of the Spark Debug Tools. The Spark Debug Tools provide a suite of
tools for debugging Spark tasks. A debugger, a profiler and a visualizer are among

nl
the tools available. You can walk through your code line by line with the debugger,
measure the performance of your code with the profiler and visualise the execution
of your code with the visualizer.

O
Here are some more recommendations for monitoring and troubleshooting parallel
tasks in Apache Spark:
●● To monitor your tasks, use the Spark Web UI and the Spark History Server. These

ty
tools give a graphical interface via which you may see metrics and troubleshoot
issues.
●● Collect custom metrics with the Spark Metrics API. This API collects metrics that
aren’t available in the Spark Web UI or the Spark History Server.

si
●● To debug your tasks, use the Spark Debug Tools. These tools can assist you in
identifying and correcting errors in your code.
er
●● Keep a close eye on your jobs on a frequent basis. This will assist you in identifying
performance issues early and making necessary modifications.

4.6.3 Managing Memory and Processor Resources in Spark

v
Spark is a strong distributed computing platform for processing massive volumes of
data. However, managing the memory and CPU resources utilised by Spark applications
ni

is critical. This will assist you in ensuring that your apps function efficiently and do not run
out of resources. In Spark, memory and CPU resources may be managed in a variety of
ways. Here are a few of the most prevalent approaches:
U

●● Configuration of the [Link] property: This attribute indicates how

much memory is assigned to each executor. The default setting is 1GB. If your
programme need additional memory, you can raise this number.
●● Configuring the [Link] of cores: This attribute indicates how
ity

many cores are assigned to each executor. The default setting is one core. If your
application requires additional cores, you can raise this number.
●● Customising Spark apps with the SparkConf object: The SparkConf object provides
a variety of methods for customising Spark applications. You can use these ways to
m

configure the memory and CPU resources that your programmes consume.
●● Using the Spark UI: The Spark UI offers a graphical interface for monitoring the
memory and CPU resources used by your Spark applications. You may use this
)A

information to make necessary changes to the setup settings.

It should be noted that the amount of memory and CPU resources utilised by Spark
applications varies depending on the application. An application that does a lot of data
processing, for example, will require more memory than an application that does a lot
of network I/O. It is also worth noting that the amount of memory and CPU resources
(c

Amity Directorate of Distance & Online Education

 Data Engineering 205

available on the cluster will have an impact on the speed of your Spark applications. If the
Notes

e
cluster lacks sufficient memory or CPU resources, your applications may operate slowly
or even fail.

in
You may assist to guarantee that your applications operate effectively and do not run
out of resources by carefully controlling the memory and processor resources that Spark
applications require.

nl
Here are some more suggestions for managing memory and CPU resources in
Spark:
●● Monitor the memory and CPU resources consumed by your Spark apps using the

O
Spark UI. This will assist you in identifying problems early and taking necessary
action.
●● Consult the Spark manual to learn more about the configuration options available for
managing memory and CPU resources.

ty
●● Experiment with various setup parameters to see which ones work best for your
unique application.
●● Make use of a cluster manager to assist you manage the resources available to

si
Spark applications.

4.6.4 Setting Apache Spark Configuration and Running on

Kubernetes
er
Kubernetes (sometimes spelt Kube or k8s) is an open-source container orchestration
system created by Google that was released in 2014 and is maintained by the Cloud
Native Computing Foundation. Kubernetes is a container orchestration system that is
v
used to automate the deployment, scaling and administration of containerized workloads,
most often Docker containers.
ni

The next procedures will differ based on your present architecture and cloud provider
(or on-premise configuration). However, these are the essential things you need to set up
to get started with Spark on Kubernetes on your own:
U

●● Set up a Kubernetes cluster

●● Determine your preferred node pools based on the demands of your workload.
●● Increase security to meet your networking needs (we recommend making the
ity

Kubernetes cluster private).

Set up a docker registry for your Spark docker images and begin developing your
own.)
●● Set up the Spark-operator
m

●● Download and install the Kubernetes cluster autoscaler.

●● Save the Spark driver logs and Spark event logs to a persistent storage location.
)A

●● Install the Spark history server (Helm Chart) or, alternatively, utilise our open-source
solution Delight (blog post, github).
●● Configure the node and Spark metrics collection (CPU, Memory, I/O, Discs).
(c

Amity Directorate of Distance & Online Education

 206 Data Engineering

How it works
Notes

e
in
nl
O
ty
si
Source: [Link]

Spark-submit may be used to submit a Spark application to a Kubernetes cluster

directly. The submission process is as follows:
●●
er
Spark produces a Spark driver that runs in a Kubernetes pod.
●● The driver generates executors that operate within Kubernetes pods, connects to
them and runs application code.
v
●● When the application is finished, the executor pods are terminated and cleaned
up, but the driver pod keeps logs and remains in the “completed” state in the
ni

Kubernetes API until it is garbage collected or manually cleaned up.

It should be noted that the driver pod uses no computational or memory resources in
its finished state.
U

Kubernetes manages the scheduling of driver and executor pods. Fabric8 is used to
communicate with the Kubernetes API. The driver and executor pods can be scheduled
on a subset of available nodes using a node selector and the configuration attribute for
ity

it. In a future edition, more complex scheduling cues such as node/pod affinities will be
available.

Summary
●● Big Data is a massive collection of data that is expanding exponentially over time. It
m

is a data set that is so huge and complicated that typical data management systems
cannot store or analyse it efficiently.
●● Flume is a data ingestion platform that allows you to collect, consolidate and move
)A

massive volumes of data from one source to another.

●● Cassandra has a masterless or peer-to-peer architecture, making it nearly immune
to failure, unlike HDFS, which can fail if the master server fails. It offers the unique
capacity to read/write data on nodes in a completely different region.
●● Kubernetes (sometimes spelt Kube or k8s) is an open-source container
(c

orchestration system created by Google that was released in 2014 and is

maintained by the Cloud Native Computing Foundation.

Amity Directorate of Distance & Online Education

 Data Engineering 207

●● Kubernetes manages the scheduling of driver and executor pods. Fabric8 is

Notes

e
used to communicate with the Kubernetes API. The driver and executor pods
can be scheduled on a subset of available nodes using a node selector and the
configuration attribute for it.

in
●● Apache Spark has a scalable Machine Learning Library known as MLlib, which can
do sophisticated analytics such as clustering, classification, dimensionality reduction
and so on.

nl
●● Spark is an intelligent technology because of major analytics occupations such as
predictive analysis, consumer segmentation, sentiment analysis and so on.
●● Apache Spark is a powerful and versatile cluster computing technology. It has high-

O
level Java, Scala, Python and R APIs, as well as an optimised engine that supports
broad execution graphs.
●● SparkSQL is a lightning-fast data-querying and analysis tool. This is due to the fact

ty
that it processes data in parallel using a distributed computing environment.

Glossary
●● HDFS (Hadoop Distributed File System) is a file system designed to run on

si
commodity systems. A Name node and numerous data nodes comprise a typical
HDFS design. A cluster is a group of nodes that may be considered of as a single
computer and each cluster can include 1000s of nodes.
●●
er
YARN is an acronym that stands for Yet Another Resource Negotiator. Map-reduce
was in charge of processing and job tracking in Hadoop 1.0. However, resource use
proved to be exceedingly inefficient.
v
●● Resource Manager: The master node is primarily in charge of allocating resources
and guiding Node Managers to do real-time processing. A scheduler is in charge
ni

of scheduling jobs and distributing resources, whereas an application manager

handles job submissions and is accountable for operating application masters in a
cluster.
●● Node Manager: A node manager is in charge of running a slave daemon. It is in
U

charge of performing assigned jobs on each node. It sends out heartbeats on a

regular basis to indicate whether or not the nodes are active.
●● Application Masters: It bargains for resources from the Resource Manager and
ity

works with node managers to complete tasks.

●● Containers: Containers are the hardware resources that are accessible in a cluster,
such as CPUs, RAMs, disc space and so on. It allows apps to access particular
quantities of resources.
m

●● Oozie Workflow jobs: These are Directed Acyclic Graphs (DAGs) that define a set of
activities to be performed.
●● Oozie Coordinator tasks: These are recurring Oozie Workflow jobs that are triggered
)A

by the availability of time and data.

●● Oozie Bundle: It allows you to package several coordinators and workflow jobs and
control their lifespan.

Check Your Understanding

(c

1. _________________ is a file system designed to run on commodity systems. A Name

node and numerous data nodes comprise a typical HDFS design.

Amity Directorate of Distance & Online Education

 208 Data Engineering

a) Hadoop Distributed File System

Notes

e
b) Knime
c) Mongo DB

in
d) Apache Storm
2. A _________________ is a group of nodes that may be considered of as a single
computer and each cluster can include 1000s of nodes.

nl
a) MapReduce
b) Machine Learning

O
c) Cluster
d) Dataflow
3. __________________ is a non-relational database management system that is best

ty
suited for semi-structured data, while it can also handle structured and unstructured
data with the exception of picture data.
a) Cassandra

si
b) Zookeeper
c) APACHE Hadoop
d) Mongo DB
4.
er
What is the full form of YARN?
a) Yet Another Reduce Negotiator
b) Yet Access Resource Negotiator
v
c) Yet Apache Resource Negotiator
ni

d) Yet Another Resource Negotiator

5. ________________ is in charge of processing massive amounts of data in a parallel
distributed fashion. It performs two distinct functions: Map and Reduce.
U

a) Map Reduce
b) Thrift
c) Mahout
ity

d) Apache Avro
6. What is the full form of HDFS?
a) Hadoop Distributed File System
b) Hadoop Distributed Field System
m

c) Hadoop Division File System

d) Hadoop Distributed File Subunit
)A

7. __________________ is a Hadoop data warehouse solution that enables users to

query data using Structured Query Language (SQL). It may also be used to build and
alter tables and views, as well as grant people rights.
a) Spark SQL
(c

b) Impala
c) Hue
d) Hive
Amity Directorate of Distance & Online Education
 Data Engineering 209

8. The _____________________ is the name node and there is only one in each cluster.
Notes

e
Its job is to figure out where each file’s block is in the cluster.
a) Master node

in
b) Data node
c) Data Storage
d) Data Access

nl
9. A ________________ is a slave node that stores data blocks and there are several
per cluster. Its job is to retrieve data as and when it is needed. It maintains regular
contact with the Name node through heartbeats.

O
a) Data node
b) Master node
c) Unstructured data

ty
d) Structured Data
10. _________________ is a distributed general-purpose cluster computing framework
that is free source. Spark provides a programming interface for complete clusters with

si
implicit data parallelism and fault tolerance.”
a) Apache Spark er
b) Flume
c) Sqoop
d) Kafka
v
11. ________________ are similar to relational database tables or excel spreadsheets
with headers in that the data is stored in rows and columns of various datatypes.
ni

a) DataFrames
b) Spark Core
c) MLlib
U

d) Graphx
12. What is the full form of BI?
ity

a) Business intelligence
b) Business Internet
c) Buying Intelligence
d) Better Internet
m

13. __________________ is used by Amazon to analyse data from their e-commerce

platform. This information is then utilised to improve product suggestions, pricing and
inventory management.
)A

a) SparkSQL
b) Graphx
c) Spark Core
d) MapReduce
(c

14. A _________________ is a software environment that contains the tools and libraries
required for a programme to run.

Amity Directorate of Distance & Online Education

 210 Data Engineering

a) Runtime environment
Notes

e
b) Mapping
c) Reduce

in
d) Shuffle
15. The ________________ is a graphical interface for monitoring your Spark applications.
A number of metrics are available, including the number of tasks running, the quantity

nl
of data processed and the time it takes to finish jobs.
a) Hadoop HDFS

O
b) Shuffle
c) Spark Web UI
d) Hadoop MapReduce

ty
16. _________________ saves data about your Spark applications, including metrics
gathered by the Spark Web UI. The Spark History Server allows you to see historical
data about your Spark applications.
a) Spark History Server

si
b) Spark Streaming
c) GraphX er
d) Loading
17. ________________ allows you to collect metrics from your Spark applications
programmatically. The Spark Metrics API may be used to gather custom metrics or
v
metrics that are not accessible in the Spark Web UI or the Spark History Server.
a) Apache Pig
ni

b) Apache Ambari
c) Spark Metrics API
U

d) Mesos
18. __________________ gives a complete review of the aspects that might impact
Spark application performance. The topic contains suggestions for optimising Spark
applications for a range of contexts.
ity

a) Scala
b) Spark Tuning Guide
c) JavaScript
d) Python
m

19. ___________________ is a container orchestration system that is used to automate

the deployment, scaling and administration of containerized workloads, most often
Docker containers.
)A

a) Kubernetes
b) Clojure
c) Erlang
(c

d) OCaml

Amity Directorate of Distance & Online Education

 Data Engineering 211

Exercise
Notes

e
1. Explain the Features of Spark DataFrames
2. What is Apache Spark?

in
3. What is HDFS, Hive, HBase and Spark?
4. What is the difference between Hadoop and MapReduce?

nl
5. What is a DataFrame and how does it works?
6. What are the types of Datasets
7. Write a short note on Resilient Distributed DataSets.

O
8. Explain the modules of Hadoop Ecosystem.

Learning Activities
1. What is Hadoop Ecosystem and what tools would you use for data processing?

ty
2. Write a case study on how companies-built Data Lake on AWS, resulting an increase
in company’s revenue.

si
3. Write a case study on successful implementation of ELT in various industries.

Check Your Understanding – Answers

1. a) 2. c)
er
3. a) 4. d)
5. a) 6. a)
v
7. d) 8. a)
9. a) 10. a)
ni

11. a) 12. a)
13. a) 14. a)
U

15. a) 16. c)
17. a) 18. c)
19. b) 20. a)
ity

Further Readings and Bibliography

1. Joe Reis and Matt Housley. (2022). Fundamentals of Data Engineering: Plan
and Build Robust Data Systems. First Edition. O’Reilly Publications.
2. Andreas Kretz. (2019). The Data Engineering Cookbook: Mastering the
m

Plumbing of Data Science.

3. Holden Karau andy Konwinski, Patrick Wendell and Matei Zaharia. (2015).
Learning Spark. First Edition. O’Reilly Publications.
)A
(c

Amity Directorate of Distance & Online Education

 212 Data Engineering

Module - V: Apache Spark for Data Engineering and

Notes

e
Machine Learning

in
Learning Objectives
At the end of this module, you will be able to:

nl
●● Learn introduction to apache spark
●● Analyse working with spark DataFrames and DataSets
●● Recognise SparkSQL for data analysis and manipulation

O
●● Analyse ETL with spark and data wrangling
●● Analyse introduction to spark MLlib
●● Identify machine learning with spark

ty
●● Explain building and training ML models with spark
●● Describe evaluating model performance in spark
●● Analyse implementation of data engineering and machine learning project using

si
spark
●● Identify applying spark for data processing and analysis
er
●● Analyse utilising spark ML for building predictive models

Introduction
Apache Spark is a distributed computing system that is designed for broad use
v
and is capable of managing very large datasets. It was initially created at the University
of California, Berkeley and has since grown to become one of the most well-known
ni

frameworks for processing enormous amounts of data in the business world. Spark
was developed to be compatible with a number of different data sources, including
the Hadoop Distributed File System (HDFS), Apache Cassandra, Apache HBase and
U

Amazon S3.

5.1 Spark for Data Engineering

ity

The following is a list of the primary advantages of using Spark for data engineering:
● Speed
By combining in-memory processing and data partitioning techniques, Spark can
analyse large datasets at rapid rates.
m

● Scalability
Spark can handle massive datasets without losing speed since it can scale horizontally
over a cluster of computers.
)A

● Ease of Use
Spark provides a clear and user-friendly interface for creating data pipelines, making
it simple for developers to implement sophisticated data processing operations.
● Flexibility
(c

Spark provides a diverse set of data sources and data processing operations, allowing
users to build customised data pipelines to match their specific needs.

Amity Directorate of Distance & Online Education

 Data Engineering 213

5.1.1 Introduction to Apache Spark

Notes

e
Apache Spark is a lightning-fast computing solution for clusters that was created
with high-performance computing in mind. It is built on Hadoop MapReduce and extends

in
the MapReduce architecture to make it possible to utilise it in a more effective manner
for other kinds of calculations, such as interactive queries and stream processing. This
was accomplished by adding new features. The most important aspect of Spark is its in-
memory cluster computing capability, which accelerates the overall processing speed of

nl
an application.
Spark is designed to cope with many different kinds of work, such as streaming data,
batch applications, iterative algorithms and interactive searches. In addition to being able

O
to support all of these workloads inside a single system, it also reduces the amount of
administrative labour that is required to manage separate tools.

Evolution of Apache Spark

ty
Spark is a subproject of Hadoop that was developed in 2009 by Matei Zaharia at
the AMPLab at UC Berkeley. In 2010, it was made available to the public under a BSD
licence. It was presented to the Apache Software Foundation as a gift in 2013 and as of

si
February 2014, it has been elevated to the status of a top-level Apache project.

Features of Apache Spark er

The following capabilities are available in Apache Spark.
●● Speed: With Spark, it is feasible to run an application in a Hadoop cluster up to one
hundred times faster in memory and ten times faster on disc than without Spark.
This is done by reducing the total number of read/write operations performed on the
v
disc. The data that is being processed at an intermediate stage is kept in memory.
ni

●● Supports many programming languages, including Python, Scala and Java through
its application programming interfaces (APIs). As a direct consequence of this, you
will have the ability to develop apps in a range of languages. Spark has eighty high-
level operators, which may be used for interactive querying.
U

●● Advanced Analytics: Spark offers more than simply the ‘Map’ and ‘Reduce’
operations to its users. SQL queries, machine learning (ML), streaming data and
graph approaches are all supported and available to users.
ity

Spark Built on Hadoop

The figure that follows demonstrates three different approaches to constructing
Spark using Hadoop components.
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 214 Data Engineering

As stated below, there are three approaches to deploy Spark.

Notes

e
●● Standalone − The term “Spark Standalone deployment” refers to the fact that Spark
is installed on top of HDFS (Hadoop Distributed File System) and that space has

in
been expressly designated for HDFS. Spark and MapReduce will run in parallel in
order to complete all Spark tasks that are waiting on the cluster.
●● Hadoop Yarn Hadoop Yarn deployment essentially signifies that spark is able to

nl
function on Yarn without the requirement of any pre-installation or root access. It
is to everyone’s advantage to add Spark to the Hadoop ecosystem as well as the
Hadoop stack. Additional components are able to operate above the stack.

O
●● Spark in MapReduce (SIMR) — In addition to standalone deployment, Spark in
MapReduce (SIMR) is employed to launch spark tasks. Users who have SIMR
installed do not need to have administrator credentials in order to run Spark or make
use of its shell.

ty
5.1.2 Working with Spark DataFrames and DataSets
Spark Release

si
●● DataFrames - DataFrames were introduced in Spark 1.3.
●● DataSet - Dataset was introduced in the Spark 1.6 version.
er
Data Representation
●● DataFrame - It is a distributed collection of data organised into named columns. It is
essentially equivalent to a relational database table.
v
●● DataSet - It is a DataFrame API extension that provides the capability of the RDD
API’s type-safe, object-oriented programming interface as well as the performance
ni

benefits of the Catalyst query optimizer and off heap storing method.

Data Formats
U

●● DataFrame — It only works with organised and semi-structured data. It arranges the
data in the specified column. Spark can handle schema using DataFrames.
●● DataSet - It also handles organised and unstructured data effectively. It represents
data as JVM row objects or a group of row objects. This is expressed in tabular form
ity

by encoders.

Data Sources API

●● DataFrame - Data source API enables data processing in a variety of forms (AVRO,
m

CSV, JSON and storage systems HDFS, HIVE tables and MySQL). It can read and
write from the aforementioned data sources.
●● DataSet - The Spark Dataset API also supports data from other sources.
)A

Immutability and Interoperability

●● DataFrame - A domain object cannot be regenerated after transforming into a
DataFrame. For example, if you build testDF from testRDD, you won’t be able to
restore the test class’s original RDD.
(c

●● DataSet - It solves DataFrame’s constraint to renew the RDD from Dataframe. You
may use Datasets to transform your current RDDs and DataFrames into Datasets.

Amity Directorate of Distance & Online Education

 Data Engineering 215

Compile-time type safety

Notes

e
●● DataFrame - If you try to access a field that does not exist in the table, Dataframe
APIs do not provide compile-time errors. It only identifies attribute errors during

in
runtime.
●● DataSet - It guarantees type safety at build time.

Optimization

nl
●● DataFrame - The catalytic optimizer is used for optimisation. In four stages,
dataframes employ the catalyst tree transformation framework: a) Resolving
references by analysing a logical scheme. b) Optimisation of logical plans. d)

O
Physical preparation. d) Code generation to compile query portions to Java
bytecode.
●● Dataset - It incorporates the Dataframe Catalyst optimizer idea for query plan

ty
optimisation.

Serialization
●● DataFrame - As Spark knows the schema, it can serialise the data into off-heap

si
storage (in memory) in binary format and then execute numerous operations
directly on this off heap memory. There is no need to encode the data using java
serialisation. It includes a Tungsten physical execution backend that actively
er
controls memory and creates bytecode dynamically for expression evaluation.
●● DataSet - When it comes to serialising data, Spark’s Dataset API offers the idea of
an encoder, which manages the conversion of JVM objects to tabular form. It saves
tabular data in the spark internal Tungsten binary format. Datasets allow you to
v
conduct operations on serialised data while also optimising memory use. It enables
on-demand access to certain attributes without destroying the entire object.
ni

Garbage Collection
●● DataFrame - Avoids the overhead of trash collection when creating distinct objects
U

for each entry in the dataset.

●● DataSet: As serialisation occurs via Tungsten, there is no need for the trash
collector to remove the object. Off heap data serialisation is used in this case.
ity

Efficiency/Memory use
●● DataFrame - Off heap memory is used for serialisation, which minimises overhead.
It creates byte code dynamically so that numerous actions on serialised data can be
done. Deserialization is not required for minor tasks.
m

●● DataSet - It allows you to execute an operation on serialised data while also

optimising memory use. As a result, it enables on-demand access to certain
attributes without deserializing the entire object.
)A

5.1.3 SparkSQL for Data Analysis and Manipulation

●● Data Analysis
Spark SQL is the most efficient data processing technology that is now in use in
businesses all around the world. Even though Spark has a high price tag because to the
(c

large amount of random access memory (RAM) that it needs to do in-memory computing,
it continues to be a favourite option among data scientists and big data engineers. In the
Spark Ecosystem, we have the following components available to us:

Amity Directorate of Distance & Online Education

 216 Data Engineering

1. MLlib is Spark’s scalable machine learning library. It contains algorithms for a variety
Notes

e
of workloads, including regression, clustering and classification. This post will provide
you with some useful information that will assist you in getting started with the
construction of machine learning pipelines using Spark’s MLlib: What are the Steps in

in
the Process of Creating Machine Learning Pipelines in PySpark?
2. Spark Streaming: At this rate and at this volume, we are producing data at a level
that has never been seen before. How can we make sure that our machine learning

nl
pipeline continues to provide results even after fresh data has been developed and
gathered? Gain an understanding of how to use PySpark to utilise a machine learning
model to create predictions based on streaming data.

O
3. GraphX is an application programming interface (API) developed by Spark for graphs.
GraphX is a network graph engine that offers parallel graph computation.
4. Spark SQL: A distributed platform for structured data processing Spark SQL is an

ty
acronym for Structured Query Language.
It is common knowledge that relational databases maintain a link between the many
variables and tables that make up the database and create it in such a manner that it
is able to execute complex query operations. In Spark, the relational processing and

si
functional programming paradigms come together beautifully in the form of Spark SQL. As
a consequence of its support for several data sources and ability to execute SQL queries,
the product is a particularly effective tool for evaluating structured data on a large scale.
er
How exactly does Spark SQL carry out the execution of a query?
In its most basic form, what role does Spark SQL play? In this part of the guide, let’s
get a grasp on the procedure.
v
●● Analysis: Before doing anything else, Spark SQL examines the object being queried
ni

to establish the relationship that needs to be computed. It is produced with the

assistance of an abstract syntax tree (AST), which first verifies the appropriate
utilisation of the query’s components and then produces a logical plan to execute
the query.
U

●● Logical Optimisation: The logical plan is then optimised using rule-based

optimisation in this stage. It employs methods such as:
™™ If the query contains a where clause, data is filtered early.
ity

™™ Making use of available indexes in tables to increase speed and

™™ Ensuring that diverse data sources are connected in the most efficient
sequence possible.
●● Physical Planning: Using the logical plan, one or more physical plans are created
m

in this stage. Spark SQL then chooses the plan that will perform the query most
efficiently, i.e. with the fewest computing resources.
●● Code Generation: Spark SQL creates code as the final step. It entails creating Java
)A

byte code that will run on each system. Catalyst employs a Scala language feature
known as “Quasiquotes” to facilitate code creation.
(c

Source: [Link]
data/#:~:text=Spark%20SQL%20supports%20almost%20every,while%20the%20query%20is%20running.

Amity Directorate of Distance & Online Education

 Data Engineering 217

●● Data Manipulation
Notes

e
Spark SQL is an Apache Spark module that offers a SQL interface for dealing with
structured data. Spark SQL may be used to accomplish a wide range of data processing

in
tasks, such as:
™™ Creating DataFrames: Spark SQL can generate DataFrames from a number of
sources such as CSV files, JSON files and Hive tables.

nl
™™ DataFrame Querying: Spark SQL may be used to query DataFrames using SQL
syntax.
™™ Transforming DataFrames: Spark SQL may be used to transform DataFrames

O
by performing aggregations, joins and projections.
™™ DataFrame Saving: Spark SQL can save DataFrames in a number of formats,
including as CSV files, JSON files and Hive tables.

ty
The following are some of the advantages of utilising Spark SQL for data
manipulation:
™™ Simplicity of usage: Spark SQL is simple to use, even for beginners.

si
™™ Speed: Spark SQL may be quite fast, especially for huge datasets.
™™ Adaptability: Spark SQL may be utilised to perform a wide range of data
processing activities.
er
Spark SQL may be used for the following specialised tasks:
™™ Cleaning data: Spark SQL can clean data by eliminating mistakes, duplicates
and outliers.
v
™™ Data transformation: Spark SQL may be used to change the format, structure,
or content of data.
ni

™™ Data analysis: Spark SQL can execute statistical and machine learning
techniques to analyse data.
U

™™ Data visualisation: Charts, graphs and other visualisations may be used with
Spark SQL to visualise data.
Spark SQL is a powerful tool that may be used to automate numerous data
manipulation activities. This can save time and effort while also helping to enhance data
ity

quality.
Here are some of the examples of how Spark SQL can be used for data
manipulation:
™™ Create a DataFrame from a CSV file:
m

df = [Link](“my_data.csv”)
™™ Query a DataFrame:
)A

[Link](“name”, “age”).show()
™™ Transform a DataFrame:
df = [Link](“new_column”, df[“age”] * 2)
™™ Save a DataFrame to a CSV file:
(c

[Link](“my_output.csv”)

Amity Directorate of Distance & Online Education

 218 Data Engineering

5.1.4 ETL with Spark and Data Wrangling

Notes

e
Data Wrangling
Data wrangling is a broad term that encompasses the variety of processes involved

in
in preparing data for analysis. It is also defined as the process of cleaning and unifying
disorganised and complicated datasets in order to improve data accessibility and
analysis.

nl
Data wrangling is becoming increasingly important as data volumes increase and
spread throughout organisations. As a result, it became critical to organise massive
amounts of data in order to perform better data analysis and insights.

O
The first stage in the data wrangling process is to gain access to the data. After
gaining access to the data, the following step is to alter it. Transforming jobs include
changing the data’s structure, granularity, correctness, timeliness and scope. The final

ty
stage in data manipulation is publication. Publishing entails providing the altered version
of the data so that it may be utilised for visualisation later on.

ETL

si
ETL is an acronym that stands for Extract, Transform and Load. The ETL process
involves extracting data from disparate source systems, combining the data, transforming
the data based on business rules and loading the data into target systems. ETL is
er
commonly used to load data warehouses or data marts. Because data warehouse tables
are often in the Star schema (Dimensional model), ETL must load data in accordance
with data integrity and referential integrity criteria.
v
ETL processes data in stages and across various physical levels (Operational
Datastore, Stage database, Datawarehouse and Data marts). ETL is also commonly
ni

used to load processed data into relational database tables or flat files.

5.2 Spark ML for Machine Learning

U

Introduction
Spark is a faster cluster computing platform that acts as an open-source, distributed,
unified analytics engine for real-time data processing. Spark is a unified analytics engine.
ity

Spark is well-known for the in-memory computing features that it possesses, which
both accelerate the rate at which it processes data and enable it to manage enormous
amounts of data. Hadoop is a framework that reads data from disc and generates key-
value pairs. Apache Spark is an enhanced version of Hadoop since it employs map-
m

reduce for processing, whereas Hadoop reads data from disc and forms key-value pairs.
It is a highly time-consuming procedure if we take data from disc, process it and then
write it back to disc. On the other hand, Spark executes everything in main memory,
)A

which decreases the amount of time needed for computation and enables operations to
be carried out extremely rapidly.
Scala, a more advanced version of Java that can be used on the JVM, serves as
the foundation for Spark. Spark offers high-level application programming interfaces
(APIs), which make it possible for us to build and use it in any language, including as
(c

Java, Python, Scala, R and others. Pyspark is an extension for the Python programming
language that handles spark.

Amity Directorate of Distance & Online Education

 Data Engineering 219

5.2.1 Introduction to Spark MLlib

Notes

e
What is Spark MLLIB?
The full name of Spark’s Machine Learning Library is MLLIB and its acronym is

in
MLLIB. This library’s objective is to make real-world machine learning more scalable
and straightforward to implement. It provides tools for the implementation of all machine
learning methods, such as regression and classification tools, dimensionality reduction

nl
tools, transformation tools, feature extraction tools, pipelines (tuning), save and load
processes and linear algebra and statistics utility programmes.
When we speak of spark MLLIB, we are referring to an application programming

O
interface (API) that is based on dataframes. Beginning with spark 2, the RDD-based
API entered the maintenance phase and as of right now, the primary ML API is a spark
dataframe-based API. ml.0. Spark offers a wide variety of machine learning tools, which
may be used to carry out a variety of tasks and conduct a number of different activities.

ty
1. Machine Learning Algorithms - This topic covers the implementation of Regression,
Classification, Clustering and Collaborative Filtering and it contains the tools and
techniques necessary to do so.

si
2. Featurization: tools for the extraction, modification and reduction of dimensionality of
features, as well as the selection of features.
3. Pipelines, which are tools for constructing, evaluating and improving machine learning
pipelines.
er
4. Persistence refers to the capacity to save and load different pipelines, models and
algorithms.
v
5. Utilities: Linear algebra, statistical analysis and the manipulation of data
ni

Data Types in Spark MLLIB

A wide number of data types are supported by Spark. Both locally stored vectors
and matrices, as well as distributed matrices, are supported by the Spark MLLIB library.
U

Because of this, when it is paired with one or more RDDs, it is able to accommodate a
broad variety of data types.
1. Local Vector: Dense and sparse local vectors are the two varieties of MLLIB’s local
vectors that are available to users. A dense or sparse local vector that is associated
ity

with a label or response is referred to as a labelled point. For example, the label for a
binary classification should either be the number 0 (for negative) or the number 1 (for
positive).
2. Local Matrix: The local matrix is a single machine that stores values of type integer in
m

rows, columns and matrices all by themselves.

3. Distributed matrix: Its values are of the double type, while its row and column indices
are of the long type. It is kept and dispersed in one or more RDDs at the same time.
)A

Machine learning Pipelines

When we discuss machine learning pipelines, one of the most important things to
understand is the numerous steps that are involved, such as the estimator, the evaluator,
the transformer and so on. Pipelines for machine learning offer standardised, high-level
(c

APIs that be constructed using data frames. It is utilised in the process of building and
tuning machine learning pipelines for the actual world. The majority of the time, it is
utilised for structured data.
Amity Directorate of Distance & Online Education
 220 Data Engineering

●● Dataframe: It is shorthand for a spark dataframe. For the purposes of machine

Notes

e
learning, SQL is used as a dataset. It is able to hold a variety of data types,
including text, feature vectors, labels and other types of data as well.

in
●● Transformer: An algorithm that transforms one dataframe into another, a transformer
takes one dataframe and creates another.
●● Estimator - An estimator is a method for fitting data to a dataframe that results in a
transformer.

nl
●● Pipeline: A pipeline is a method of machine learning that integrates a number of
transformers and estimators in a single process.

O
●● Evaluator - It will evaluate the results produced by the model.

5.2.2 Machine Learning with Spark

Spark MLlib is an abbreviation for the Spark Machine Learning library. Machine

ty
Learning is simple to use and scalable with PySpark. It is applicable to distributed
systems. Spark Machine Learning may be used to analyse data. Because of the PySpark
MLlib, you may utilise numerous strategies using Machine Learning algorithms such as
regression, classification and so on.

si
PySpark MLlib parameters
The following are some of the most important PySpark MLlib parameters:
er
●● Ratings: This choice allows you to construct an RDD consisting of ratings, rows, or
tuples.
●● Rank: This feature not only displays the total number of computed attributes but also
v
ranks those features.
●● Lambda: A regularisation parameter, lambda is referred to as the lambda parameter.
ni

●● Blocks: In order to parallelize computations, blocks are utilised. By default, this is set
to the value -1.
U

Performing Linear Regression on a Real-world Dataset

Let’s obtain a better grasp of Machine Learning by building a full-fledged code to run
linear regression on a dataset consisting of the top 5 Fortune 500 organisations in 2017.
This will help us gain a better understanding of Machine Learning.
ity

Loading Data
As previously stated, you will utilise a DataFrame built straight from a CSV file. The
instructions to load data into a DataFrame and view the loaded data are as follows.
m

●● Input:
o In [1]:
from pyspark import SparkConf, SparkContext
)A

from [Link] import SQLContext

Sc = SparkContext()
sqlContext = SQLContext(sc)
(c

o In [2]:
company_df = [Link](‘[Link]’).

Amity Directorate of Distance & Online Education

 Data Engineering 221

options(header=’true’, inferschema=’true’).load(‘C:/Users/intellipaat/Downloads/
Notes

e
spark-2.3.2-bin-hadoop2.7/[Link]’)
company_df.take(1)

in
When presenting the DataFrame’s data, you may choose the number of rows to
display. Only the first row is shown here.
●● Output:

nl
o Out[2]:
[Row (Rank=1, Title= ‘Walmart’, Website= ‘http:/[Link]’,
Employees-2300000, Sector= ‘retailing’)]

O
Data Exploration
You may use the following commands to examine the data type of each column in a
DataFrame and to output the DataFrame’s structure in a tree format:

ty
●● Input:
o In[3]:

si
company_df.cache()
company_df.printSchema()
●● Output:
er
o Out [3]:
DataFrame[Rank: int, Title: string, Website: string, Employees: Int, Sector: string]
v
root
|-- Rank: integer (nullable = true)
ni

|-- Title: string (nullable = true)

|-- Website: string (nullable = true)
U

|-- Employees: integer (nullable = true)

|-- Sector: string (nullable = true)

Performing Descriptive Analysis

ity

●● Input:
o In [4]:
company_df.describe().toPandas().transpose()
●● Output:
m

o Out [4]:

0 1 2 3 4
)A

Summary count mean stddev min max

Rank 5 3.0 1.581138830084 1 5
Title 5 None None Apple Walmart
Website 5 None None [Link] [Link]
(c

Employees 5 584880.0 966714.2168190142 68000 2300000

Sector 5 None None Energy Wholesalers

Amity Directorate of Distance & Online Education

 222 Data Engineering

Finding the Correlation Between Independent Variables

Notes

e
A scatter matrix can be used to determine whether any of the variables, i.e., fields,
have correlations or dependencies. In Machine Learning, plotting a scatter matrix is one

in
of the best approaches to find any linear relationships.
You may use the following code to plot a scatter matrix on your DataFrame:
●● Input:

nl
o In [5]:
import pandas as pdnumeric_features = [t[0] for t in company_df.dtypes if t[1] ==
‘int’ or t[1] == ‘double’]

O
sampled_data = company_df.select(numeric_features).sample(False, 0.8).
toPandas()
axs = pd.scatter_matrix(sampled_data, figsize=(10, 10))

ty
n = len(sampled_data.columns)
for i in range(n):

si
v = axs[i, 0]
[Link].set_rotation(0)
[Link].set_ha(‘right’)
v.set_yticks(())
er
h = axs[n-1, i]
[Link].set_rotation(90)
v
h.set_xticks(())
ni

●● Output:
o Out [5]:
U
ity
m
)A
(c

Source: [Link]

Amity Directorate of Distance & Online Education

 Data Engineering 223

You may conclude from this that the “Rank” and “Employees” columns in the dataset
Notes

e
have a link. Let’s look a bit more closely at the association between these two columns.

Correlation Between Independent Variables

in
●● Input:
o In [6]:

nl
import six
for i in comapny_df.columns:
if not( isinstance(company_df.select(i).take(1)[0][0], six.string_types)):

O
print( “Correlation to Employees for “, i, company_df.[Link](‘Employees’,i))
●● Output:
o Out [6]:

ty
Correlation to Employees for Rank −0.778372714650932
Correlation to Employees 1.0

si
The correlation value varies from -1 to 1 and the closer it is to ‘1’, the more positive
connection may be established between the variables. If the value is near to -1, it
indicates that the fields have a high negative association. Now you may examine your
er
data to determine whether there is a correlation and if so, if there is a strong positive or
negative correlation.

Preparing Data
v
●● Input:
o In [7]:
ni

from [Link] import VectorAssembler

vectorAssembler = VectorAssembler(inputCols = [‘Rank’, ‘Employees’], outputCol
U

= ‘features’)
tcompany_df = [Link](company_df)
tcompany_df = tcompany_df.select([‘features’, ‘Employees’])
ity

tcompany_df.show(3)
●● Output:
o Out [7]:
m
)A

●● Input:
o In [8]:
(c

splits = tcompany_df.randomSplit([0.7, 0.3])

train_df = splits[0]
Amity Directorate of Distance & Online Education
 224 Data Engineering

test_df = splits[1]
Notes

e
Linear Regression
●● Input:

in
o In [10]:
from [Link] import LinearRegression

nl
lr = LinearRegression(featuresCol = ‘features’, labelCol=’Emplyees’, maxIter=10,
regParam=0.3, elasticNetParam=0.8)
lr_model = [Link](train_df)

O
print(“Coefficients: “ + str(lr_model.coefficients))
print(“Intercept: “ + str(lr_model.intercept))
●● Output:

ty
o Out [10]:
Coefficients: [-32251.88812374517, 0.9255193858709874]
Intercept: 140317.88600801243

si
You may finally come to the conclusion that ‘Employees’ is the most meaningful field
or feature in the given dataset, which can be used to estimate firm rankings in the future
after performing linear regression on the dataset. This may be done by using the dataset
er
to do linear regression. ‘Ranks’ has a linear connection with ‘workers,’ which means
that the number of employees in a particular year in the companies that are included in
our dataset has a direct impact on their Rank. This is implied by the linear nature of the
v
relationship between the two terms.
The Role of Machine Learning in the Workplace Computer systems that could learn
ni

to predict and improve themselves based on the data that they were given were thought
to be a pipe dream until quite recently. However, this is now feasible because to advances
in machine learning. Large businesses are increasingly turning to the subject of artificial
U

intelligence known as machine learning in an effort to find ways to streamline their

operations and increase productivity. Expertise in machine learning is currently in great
demand and there is a lot of competition for it. Professionals in the field of machine learning
are seeing increases in their salaries. The following is a list of some of the companies and
ity

organisations where various applications of machine learning are being used:

●● PayPal: In order to identify potentially fraudulent actions, PayPal utilises machine
learning.
●● IBM: IBM has patented a machine learning method that can help determine when
m

it is safe to hand over control of a self-driving car to a human driver from a vehicle
control processor. This technology is intended to be used in autonomous vehicles.
●● On Google’s Search Engine, Machine Learning is utilised to first collect data from
)A

users, which is then used to improve search results.

●● Walmart: In order to achieve higher levels of productivity, Walmart implements
machine learning.
●● Amazon: In order to generate and apply customised product suggestions, Amazon
uses machine learning.
(c

●● Facebook uses machine learning to weed out content of low quality to improve the
user experience.

Amity Directorate of Distance & Online Education

 Data Engineering 225

5.2.3 Building and Training ML Models with Spark

Notes

e
Apache Spark is a robust large data processing platform for building and training
machine learning models. Spark has several properties that make it ideal for machine

in
learning, including:
●● Speed: Because Spark is a fast framework, it is perfect for machine learning jobs
that demand huge datasets.

nl
●● Scalability: Spark can handle extremely huge datasets.
●● Usability: Spark is simple to use, especially for beginners.
●● Adaptability: Spark may be used for a wide range of machine learning workloads.

O
Here are some of the stages required in developing and training machine learning
models with Spark:
1. Load the data: The first step is to load the data for training your model. Spark can read

ty
data from CSV files, JSON files and Hive tables, among other sources.
2. Prepare the data: Once the data has been imported, it must be prepared for training.
This may entail cleaning the data, eliminating outliers and translating the data into a

si
format that the machine learning system can understand.
3. Select a machine learning algorithm: You may train your model using a number of
machine learning methods. The algorithm you select will be determined by the sort of
er
problem you are attempting to solve.
4. Train the model: After selecting a machine learning algorithm, you must train the
model using data. Depending on the amount of the dataset and the complexity of the
algorithm, this procedure may take some time.
v
5. Evaluate the model: Once the model has been trained, it must be evaluated to
ni

determine how well it works. This may be accomplished through the use of a holdout
dataset or by cross-validation.
6. Deploy the model: Once you’ve assessed the model and are pleased with its
U

performance, you may put it into production. This might entail saving the model to a
file or uploading it to a cloud service.

5.2.4 Evaluating Model Performance in Spark

ity

Model Evaluation
Evaluate model performance by probability cutoff
performance_df = [Link]([(0,0,0)], [‘cutoff’, ‘AUPR’, ‘AUC’])
m

for cutoff in range(5, 95, 5):

cutoff = (cutoff * 0.01)
)A

print(‘Testing cutoff = ‘, str(format(cutoff, ‘.2f’)))

lrpredictions_prob_temp = [Link](‘prediction_test’,
when(col(‘probability’) >= cutoff, 1).otherwise(0).cast(DoubleType()))
(c

aupr_temp = BinaryClassificationMetrics(lrpredictions_prob_temp[‘label’, ‘prediction_

test’].rdd).areaUnderPR

Amity Directorate of Distance & Online Education

 226 Data Engineering

auc_temp = BinaryClassificationMetrics(lrpredictions_prob_temp[‘label’, ‘prediction_

Notes

e
test’].rdd).areaUnderROC
print(‘\tAUPR:’, aupr_temp,’\tAUC:’, auc_temp)

in
performance_df_row = [Link]([(cutoff,aupr_temp,auc_temp)],
[‘cutoff’, ‘AUPR’, ‘AUC’])
performance_df = performance_df.union(performance_df_row)

nl
display(performance_df)
Evaluate multiclass classification models

O
from [Link] import MulticlassClassificationEvaluator
from [Link] import MulticlassMetrics

ty
# Evaluate best model
print(‘Accuracy:’, [Link](lrpredictions))
lrmetrics = MulticlassMetrics(lrpredictions[‘label’,’prediction’].rdd)

si
print(‘Confusion Matrix:\n’, [Link]())
print(‘F1 Score:’, [Link](1.0,1.0))
er
Evaluate binary classification models
for model in [“lrpredictions”, “dtpredictions”, “rfpredictions”, “nbpredictions”,
“gbpredictions”]:
v
df = globals()[model]
ni

tp = df[([Link] == 1) & ([Link] == 1)].count()

tn = df[([Link] == 0) & ([Link] == 0)].count()
fp = df[([Link] == 0) & ([Link] == 1)].count()
U

fn = df[([Link] == 1) & ([Link] == 0)].count()

a = ((tp + tn)/[Link]())
ity

if(tp + fn == 0.0):
r = 0.0
p = float(tp) / (tp + fp)
m

elif(tp + fp == 0.0):
r = float(tp) / (tp + fn)
p = 0.0
)A

else:
r = float(tp) / (tp + fn)
p = float(tp) / (tp + fp)
(c

if(p + r == 0):

Amity Directorate of Distance & Online Education

 Data Engineering 227

f1 = 0
Notes

e
else:
f1 = 2 * ((p * r)/(p + r))

in
print(“Model:”, model)
print(“True Positives:”, tp)

nl
print(“True Negatives:”, tn)
print(“False Positives:”, fp)

O
print(“False Negatives:”, fn)
print(“Total:”, [Link]())
print(“Accuracy:”, a)

ty
print(“Recall:”, r)
print(“Precision: “, p)

si
print(“F1 score:”, f1)
print(‘AUC:’, BinaryClassificationMetrics(df[‘label’,’prediction’].rdd).areaUnderROC)
print(“\n”)
er
5.3 Project
v
Introduction
A project is defined as a sequence of tasks that must be completed to attain a
ni

certain outcome. The term Project refers to “any temporary endeavor with a definite
beginning and end”. Depending on its complexity, it can be managed by a single person
or hundreds.
U

5.3.1 Implementation of Data Engineering and Machine Learning

Project Using Spark
Apache Spark requires users to go through a number of stages in order to
ity

successfully implement a Data Engineering and Machine Learning project. These stages
include data intake and preprocessing, model training and assessment and model
evaluation. The following is an in-depth walkthrough of the implementation process:

Step 1: Data Ingestion

m

Determine where the data is coming from and what format it is in (CSV, JSON,
Parquet, etc.). When loading the data into Spark DataFrames, you may make use of
Spark’s DataFrame API or Spark SQL.
)A

Step 2: Data Exploration and Preprocessing

Conduct an exploratory data analysis, often known as EDA, in order to gain an
understanding of the structure and distribution of the data. Take care of any missing
values, duplicates, or outliers that you find. Carry out feature engineering in order to
(c

generate features that are pertinent to the machine learning model.

Step 3: Data Transformation and Feature Engineering

Amity Directorate of Distance & Online Education

 228 Data Engineering

To alter and manipulate the data, you may make use of Spark’s built-in functions
Notes

e
or UDFs, which stands for user-defined functions. Encode categorical variables by
utilising encoding methods such as one-hot encoding or string indexer. Use either the
StandardScaler or the MinMaxScaler to scale the numerical characteristics.

in
Step 4: Train-Test Split
The preprocessed data should be divided into a training dataset and a testing

nl
dataset.

Step 5: Model Selection and Training

O
Select the proper algorithm for machine learning (such as Logistic Regression,
Decision Trees, Random Forest, etc.). Using Spark’s MLlib or ML packages, define the
model [Link] data used for training the machine learning model should be used.

Step 6: Model Evaluation

ty
Evaluate the performance of the model on the testing data by using relevant
measures (such as accuracy, F1 score and other similar metrics). Adjusting the model’s
hyperparameters can help enhance its overall performance.

si
Step 7: Model Deployment
Keep the trained model safe for use or deployment at a later time. When deploying
er
the model into a production environment, you may utilise MLflow or one of the many
alternative deployment frameworks available.

Step 8: Batch or Real-time Prediction

v
Make predictions based on newly collected data using the trained model, either
in batch or real-time circumstances. Keep an eye on how well the model is doing in
ni

production and get more training if it’s required.

Step 9: Performance Optimization (Optional)

U

The Spark task should be optimised for optimal performance by tweaking Spark
setups and partitioning data, respectively. When growing, you might want to think about
using distributed computing platforms such as Apache Hadoop YARN or Kubernetes.
ity

Step 10: Documentation and Reporting

It is important to keep detailed documentation of each stage of the project, including
the data pretreatment, feature engineering, model selection and assessment processes.
Make some papers or graphic representations to present the findings and conclusions of
m

the project.
A number of data processing stages, as well as training and assessment of models,
are required for the implementation of a Data Engineering and Machine Learning project
)A

utilising Apache Spark. since of Apache Spark’s distributed computing capabilities,

it is a popular candidate for use in Big Data and machine learning applications since it
can effectively manage massive amounts of data. You will be able to effectively create
and deploy data-driven machine learning models using Spark in a variety of real-world
applications if you follow this guide’s step-by-step instructions.
(c

5.3.2 Applying Spark for Data Processing and Analysis

Apache Spark is a strong distributed computing platform that is frequently used in
Amity Directorate of Distance & Online Education
 Data Engineering 229

big data and data-intensive applications for data processing and analysis. It provides
Notes

e
a scalable and effective framework for dealing with big amounts of data as well as
executing different data manipulation and analysis operations. The following is a full
explanation of how Spark may be used for data processing and analysis:

in
Distributed Data Processing
Spark’s distributed computing paradigm enables it to distribute and analyse data in

nl
parallel over a cluster of computers. This distributed technique allows Spark to handle
enormous datasets that are too vast for a single system. It separates data into smaller
partitions that are handled independently on various cluster nodes.

O
In-Memory Data Processing
In-memory computation is used by Spark to keep intermediate data in RAM,
eliminating the requirement for frequent data read/write operations to disc. In-memory

ty
data processing greatly accelerates data analysis operations, particularly iterative
algorithms and repetitive computations.

Data Transformation and Manipulation

si
Spark supports a wide range of data transformation and manipulation operations,
such as filtering, mapping, aggregating, combining and sorting. Users can utilise these
processes to preprocess and clean data, execute data wrangling activities and prepare
data for analysis.
er
Batch Processing
Spark is ideal for batch processing jobs in which data is handled in fixed-size chunks
v
or batches. It is frequently used in Extract, Transform and Load (ETL) procedures, data
warehousing and data cleaning.
ni

Real-time Stream Processing

Spark Streaming is a Spark addon that enables real-time stream processing. It
U

receives data from a variety of sources, such as Kafka, Flume, or HDFS and processes
it in micro-batches or real-time windows. This enables data analysis and insights to be
obtained in near real time.
ity

Machine Learning with MLlib

Spark’s MLlib is a machine learning library that provides a variety of techniques
for developing and deploying machine learning models. Classification, regression,
clustering, collaborative filtering and other supervised and unsupervised learning tasks
m

are supported by MLlib.

Graph Processing with GraphX

GraphX is a Spark graph processing package that allows you to analyse and
)A

manipulate large-scale graphs and social networks. It includes graph algorithms and
parallel primitives for processing graph-based data effectively.

Integration with Big Data Ecosystem

Spark works in tandem with other big data ecosystem components such as Hadoop
(c

Distributed File System (HDFS), Hive, HBase and Cassandra. This connection enables
users to analyse and analyse data using current data and infrastructure.

Amity Directorate of Distance & Online Education

 230 Data Engineering

Scalability and Fault Tolerance

Notes

e
The distributed computing approach used by Spark guarantees scalability and fault
tolerance. It recovers automatically from node failures by recalculating missing data

in
partitions, guaranteeing that data processing operations may continue uninterrupted.

Data Visualization and Reporting

To generate meaningful visualisations and reports from processed data, Spark may

nl
be used in combination with data visualisation frameworks such as Matplotlib or Bokeh.
Data analysts and business users can get insights and make data-driven decisions as a
result of this.

O
Overall, the use of Spark for data processing and analysis is extensive and
diversified. Its distributed computing, in-memory processing, real-time stream processing,
machine learning and graph processing capabilities make it a powerful tool for a wide

ty
range of big data and analytics jobs. Spark is a popular solution for organisations working
with large-scale data processing and analysis difficulties due to its efficiency, scalability
and ease of use.

si
5.3.3 Utilising Spark ML for Building Predictive Models
One of the biggest real estate information markets in the U.S. is Zillow, which is also
a prime example of effective machine learning (ML). Zillow Research employs ML models
er
to calculate house values and forecast market changes by examining hundreds of data
points for each property. The median sales prices of homes in an area are predicted
using Apache Spark ML Random Forest Regression.
v
Classification and Regression
Two subcategories of supervised machine learning techniques are classification and
ni

regression. Predictive analytics, also known as supervised machine learning, employs

a model that can identify patterns in labelled data to predict the labels on fresh data. A
dataset having labels (also known as the intended outcome) and features (also known as
U

attributes) is used by classification and regression algorithms to learn how to label fresh
data based on those features.
ity
m
)A

Source: [Link]
learning/
(c

Classification determines the category to which a product or service belongs, such

as the legitimacy of a credit card transaction. A continuous numerical value, such as a
home price, can be predicted by regression.

Amity Directorate of Distance & Online Education

 Data Engineering 231

Regression
Notes

e
Regression calculates the link between a label, which is a dependent variable
on the intended result and one or more attributes, which are independent variables.

in
Regression can be used to evaluate how strongly a relationship exists between a label
and a set of feature variables, estimate how much a change in one or more of the feature
variables affects the label and forecast future trends. Let’s examine a housing price linear
regression example using historical home prices and house characteristics (such as

nl
square footage, the number of bedrooms and location):
●● What do we want to forecast?

O
The house price is the label.
●● What data characteristics can you use to make predictions?
These are the features: in order to create a regression model, you must extract
the relevant aspects that are most strongly correlated with the label and most strongly

ty
influence the prediction. The size of the house will be used in the example that follows.

si
v er
ni

Source: [Link]
learning/
U

With the equation: Y = intercept + (coefficient * X) + error, linear regression

models the connection between the Y “Label” and the X “Feature,” in this example the
association between the house price and size. The coefficient calculates how much a
ity

characteristic affects the price, in this example how much the size of the house affects
the price.
m
)A
(c

Source: [Link]
learning/

Amity Directorate of Distance & Online Education

 232 Data Engineering

Multiple linear regression models the association between two or more “Features”
Notes

e
and a “Label.” The multiple linear regression function might like the following, for
instance, if we were to model the association between price and house size, bedroom
count and bathroom count:

in
Yi = β0 + β1X1 + β2X2 + · · · + βp Xp + Ɛ
Price = intercept + (coefficient1 size) + (coefficient2 bedrooms) + (coefficient3 *

nl
bathrooms) + error.
The coefficients calculate each feature’s influence on pricing.

Decision Trees

O
By assessing a set of rules that use an if-then-else pattern, decision trees produce
a model that predicts the label. The “true” or “false” responses to the if-then-else feature
questions are the branches in the tree leading to the child nodes.

ty
A decision tree model calculates the bare minimum of true/false questions required
to determine the likelihood of a successful decision. Regression to forecast a continuous
numeric value, classification to predict a category, or probability of a category may all be

si
done using decision trees. An example of a condensed decision tree to forecast home
prices is shown below:
●● Q1: If the size of the house > 2000sqft
er
™™ T:Q2: If the number of bedrooms > 3
 T:Q3: If the number of bathrooms is > 3
 T: Price=$400,000
v
 F: Price=$200,000
ni
U
ity

Source: [Link]
learning/

Random Forests
m

To create a better model, ensemble learning techniques mix different machine

learning methods. Random forest is a common classification and regression ensemble
learning approach. During the training step, the algorithm constructs a model composed
of numerous decision trees based on distinct subsets of data. Predictions are formed by
)A

merging the results of all the trees, which minimises variation and enhances predictive
accuracy. The label is projected to be the class suggested by the majority of trees
in random forest classification. The label in random forest regression is the mean
regression prediction of the individual trees.
(c

Amity Directorate of Distance & Online Education

 Data Engineering 233

Notes

e
in
nl
O
Source: [Link]
learning/

Spark provides the following algorithms for regression:

ty
™™ Linear regression
™™ Generalized linear regression
™™ Decision tree regression

si
™™ Random forest regression
™™ Gradient-boosted tree regression
™™ XGBoost regression
™™ Survival regression
er
™™ Isotonic regression
v
Machine Learning Workflows
Machine learning is an iterative process that includes the following steps:
ni

●● Extracting, Transforming and Loading (ETL) historical data and analysing it to

extract relevant characteristics and labels.
●● Building a model through training, testing and assessing the outputs of ML
U

algorithms.
●● Making forecasts using the model in production with new data.
●● Model monitoring and data-driven model updates.
ity

Using Spark ML Pipelines

To be utilised by an ML method, the features and label must be placed in a feature
vector, which is a vector of integers reflecting the value of each feature. To develop the best
model, feature vectors are used to train, test and evaluate the outcomes of an ML algorithm.
m
)A
(c

Source: [Link]
learning/

Amity Directorate of Distance & Online Education

 234 Data Engineering

Reference Learning Spark

Notes

e
Spark ML provides a consistent collection of high-level APIs for constructing
ML pipelines or workflows on top of DataFrames. ML pipelines developed on top of

in
DataFrames combine the scalability of partitioned data processing with the simplicity of
data manipulation provided by SQL.

nl
O
ty
si
Source: [Link]
learning/

We make use of a Spark ML Pipeline to process the data and extract the features,
er
an estimator to build the model and an evaluator to determine how accurate the model is.
●● Transformer: An algorithm that transforms one DataFrame into another, a
Transformer is also known as “Transformer.” It is planned to make use of a
v
Transformer in order to produce a DataFrame that has a features vector column.
●● Estimator: An Estimator is an algorithm that can be used to a DataFrame in order to
ni

generate a transformer. This may be done by fitting the Estimator to the DataFrame.
We are going to train a model with an estimator and then we are going to give a
model Transformer that has the ability to add a predictions column to a DataFrame
U

that already has a features vector column.

●● Pipeline: An ML process may be defined by a collection of Transformers and
Estimators that are connected via a pipeline.
●● Evaluator: An Evaluator examines the validity of a trained Model based on the labels
ity

and predictions contained in a DataFrame column.

Summary
●● Apache Spark is a general-purpose distributed computing system that can handle
massive datasets. It was developed at the University of California, Berkeley and has
m

since become one of the most renowned huge data processing frameworks in the
industry.
●● Spark is designed to work with several data sources, including Hadoop Distributed
)A

File System (HDFS), Apache Cassandra, Apache HBase and Amazon S3.
●● ETL is an acronym that stands for Extract, Transform and Load. The ETL process
involves extracting data from disparate source systems, combining the data,
transforming the data based on business rules and loading the data into target
systems.
(c

●● Spark is an open-source, distributed, unified analytics engine that serves as a

quicker cluster computing platform for real-time data processing.

Amity Directorate of Distance & Online Education

 Data Engineering 235

●● Spark is well-known for its in-memory compute capabilities, which enhances data
Notes

e
processing speed and allows it to manage massive volumes of data.

Glossary

in
●● Data wrangling is a broad term that encompasses the variety of processes
involved in preparing data for analysis. It is also defined as the process of cleaning
and unifying disorganised and complicated datasets in order to improve data

nl
accessibility and analysis.
●● ETL is an acronym that stands for Extract, Transform and Load. The ETL process
involves extracting data from disparate source systems, combining the data,

O
transforming the data based on business rules and loading the data into target
systems.
●● Spark is an open-source, distributed, unified analytics engine that serves as a
quicker cluster computing platform for real-time data processing.

ty
●● Hadoop is a framework that uses map-reduce for processing, which reads data from
disc and forms key-value pairs.
●● MLLIB is an abbreviation for Machine Learning Library in Spark. The goal of this

si
library is to make real machine learning scalable and simple to apply.
●● GraphX: A Spark API for graphs, GraphX is a network graph engine that provides
parallel graph computing.
●●
er
Spark SQL: Spark SQL is a distributed platform for structured data processing.

Check Your Understanding

v
1. What is the full form of ML?
a) Matter Learning b) Monitor Learning
ni

c) Machine Library d) Machine learning

2. _______________ is defined as the process of cleaning and unifying disorganised
and complicated datasets in order to improve data accessibility and analysis.
U

a) Data wrangling b) Data Storage

c) Data Warehousing d) Data Structure
3. The ____________________ involves extracting data from disparate source systems,
ity

combining the data, transforming the data based on business rules and loading the
data into target systems.
a) Data Process b) Relational Process
c) ETL process d) Batch Processing
m

4. _______________ is an open-source, distributed, unified analytics engine that serves

as a quicker cluster computing platform for real-time data processing.
a) JSON file b) Hive
)A

c) Amazon d) Spark
5. _____________________ is a more advanced version of Hadoop because Hadoop
is a framework that uses map-reduce for processing, which reads data from disc and
forms key-value pairs.
(c

a) PySpark b) Apache Spark

c) Dataframe d) A

Amity Directorate of Distance & Online Education

 236 Data Engineering

6. MLLIB is an abbreviation for __________________. The goal of this library is to make

Notes real machine learning scalable and simple to apply.

e
a) Mapped Learning Library in Spark

in
b) Machine Learning Library in Spark
c) Map Learning Library in Spark
d) Master Learning Library in Spark

nl
7. ___________________ includes tools and techniques for the implementation of
Regression, Classification, Clustering and Collaborative Filtering.
a) GraphX b) Spark History Server

O
c) Machine Learning Algorithms d) Spark Streaming
8. A ____________________ model calculates the bare minimum of true/false questions
required to determine the likelihood of a successful decision.

ty
a) Decision tree b) Flume
c) Scoop d) Kafka

si
9. What is the full form of ETL?
a) Extracting, Transforming and Loading
b) Extracting, Transact and Loading
c)
er
Extracting, Transforming and Learning
d) Extracting, Transfer and Loading
10. A ___________________ is an algorithm that converts one DataFrame to another. A
v
Transformer will be used to generate a DataFrame with a features vector column.
ni

a) Estimator b) Pipeline
c) Transformer d) DataFrame
11. An _________________ is an algorithm that may be fitted to a DataFrame in order
U

to create a transformer. We’ll train a model with an estimator and provide a model
Transformer that can add a predictions column to a DataFrame containing a features
vector column.
a) Estimator b) Dataset
ity

c) Catalyst d) MapReduce
12. A __________________ connects numerous Transformers and Estimators to define
an ML process.
a) Pipeline b) MapReduce
m

c) Hadoop d) Spark
13. An __________________ evaluates a trained Model’s correctness on label and
)A

prediction DataFrame columns.

a) Featurization b) Evaluator
c) Data Manipulation d) Code Generation
14. _______________ developed on top of DataFrames combine the scalability of
(c

partitioned data processing with the simplicity of data manipulation provided by SQL.
a) Regression b) Classification

Amity Directorate of Distance & Online Education

 Data Engineering 237

c) ML pipelines d) Logical Optimisation

Notes

e
15. A dataset having labels, also known as the _________________ and features (also
known as attributes) is used by classification and regression algorithms to learn how

in
to label fresh data based on those features.
a) Intended outcome b) Advanced Analytics
c) Scala d) Python

nl
16. ________________ is a single machine that stores integer type row, column and
double type values.
a) Distributed matrix b) Local matrix

O
c) Local Vector d) Machine Learning Algorithms
17. _________________ has double-type values and long-type row and column indices.
It is stored and distributed in one or more RDDs.

ty
a) Distributed matrix b) Local matrix
c) Local Vector d) Logical Optimisation
18. What is the full form of MLLIB?

si
a) Managed Learning Library in Spark
b) Machine Learning Library in Spark
c) Machine Language Library in Spark
er
d) Machine Learning Library in SQL
19. Spark SQL creates code as the final step. It entails creating Java byte code that
v
will run on each system. Catalyst employs a Scala language feature known as
___________________ to facilitate code creation.
ni

a) Quasiquotes b) MLlib
c) GraphX d) Spark SQL
U

20. What is the full form of AST?

a) Abstract sparx tree b) Abstract syntax tree
c) Application syntax tree d) Analysis syntax tree
ity

Exercise
1. Write a short note on Apache Spark.
2. Explain Data Analysis and Data Manipulation.
3. What is Data Wrangling.
m

4. How to build and train ML Models with Spark?

5. How to a apply Spark for Data Processing and Analysis?
)A

Learning Activities
1. How will you evaluate model performance in Spark?

Check Your Understanding - Answers

(c

1. d) 2. a)
3. c) 4. d)

Amity Directorate of Distance & Online Education

 238 Data Engineering

5. b) 6. b)
Notes

e
7. c) 8. a)
9. a) 10. c)

in
11. a) 12. a)
13. b) 14. c)

nl
15. a) 16. b)
17. a) 18. b)
19. a) 20. b)

O
Further Readings and Bibliography
1. Joe Reis and Matt Housley. (2022). Fundamentals of Data Engineering: Plan
and Build Robust Data Systems. First Edition. O’Reilly Publications.

ty
2. Andreas Kretz. (2019). The Data Engineering Cookbook: Mastering the
Plumbing of Data Science.
3. Holden Karau andy Konwinski, Patrick Wendell and Matei Zaharia. (2015).

si
Learning Spark. First Edition. O’Reilly Publications.
v er
ni
U
ity
m
)A
(c

Amity Directorate of Distance & Online Education