RTIT (RECENT TREND IN IT)

Chapter 1. Introduction to Recent Trends

1.1 ARTIFICIAL INTELLIGENCE: -

 Artificial Intelligence is a technology that replicates human intelligence in a machine. If a

machine can learn, think, and execute just as a human brain would do, in short, the
intelligence that is exhibited by a machine is known as artificial intelligence.
 Artificial Intelligence helps a machine or system in thinking, rationalizing, learning,
reasoning, problem-solving, planning, decision making and language processing. Hence,
when a machine can take decisions and perform tasks on its own, it is called an intelligent
machine.

Advantages of AI: -

1. Reduction in Human Error: Humans make mistakes from time to time, but computers don't
if they programmed properly. With AI, the decisions are taken from the previously gathered
information applying a certain set of algorithms.
2. Available 24×7: An average human will work for 6-8 hours a day excluding the breaks. But
by using AI we can make machines work 24×7 without any breaks.
3. Digital Assistance: Highly advanced organizations use digital assistants to interact with users
which save the need of human resources. The digital assistant is also used in many websites to
provide things that the user wants.
4. Faster Decisions: While making a decision humans will analyse many factors both
emotionally and practically, but AI-powered machines work on what is programmed and
deliver the results in a faster way.

Disadvantages of AI: -

1. Making Human Idle: AI is making humans idle with its applications automating most of the
work. Humans tend to get addicted to these inventions which can cause problems for future
generations.
2. Unemployment: As AI is replacing most of the repetitive tasks and other work with robots,
human interference is becoming less, which will cause a major problem in the employment
standards.
3. High Costs of Creation: As AI is updating every day the hardware and software need to get
updated with time to meet the latest requirements. Machines need repairing and maintenance
which need plenty of costs.
4. No Emotions: There is no doubt that machines are much better when it comes to working
efficiently but they cannot replace the human connection that makes the team. Machines
cannot develop a bond with humans, which is an essential attribute when it comes to Team
Management.
1.2 DATA WAREHOUSE: -
 Data Warehouse is a concept which supports decision support systems where a large amount
of data is merged. A data warehouse is a repository which is at the top of multiple databases.
It can be defined as a process for collecting and managing data from varied sources to provide
meaningful business insights.
 A data warehouse is used to analyse the data, to generate reports based on that analysis.
 The data used for analysis or reporting is heterogeneous data collected from multiple sources.
The main purpose of data warehousing is to combine such data for the purpose of analysing
and reporting. The data is then used for the strategic planning and decision making for the
organizations.

Advantages of Data Warehouse: -

 Data Warehouse allows business users to quickly access critical data from some sources all in
one place.
 Data Warehouse stores a large amount of historical data that helps users to analyse different
time periods and trends to make future predictions.
 Data Warehouse helps to reduce total turnaround time for analysis and reporting.
 Data Warehouse provides consistent information on various cross-functional activities.
 Data Warehouse helps to integrate many sources of data to reduce stress on the production
system.

Disadvantages of Data Warehouse: -

 Not an ideal option for unstructured data.

 Difficult to make changes in data types and ranges, data source schema, indexes,
and queries.
 The data warehouse may seem easy, but it is too complex for the average users.

Applications of Data Warehouse: -

1. Transportation Industry: In the transportation industry, data warehouses record customer

data enabling traders to experiment with target marketing where the marketing campaigns are
designed by keeping customer requirements in mind.
2. Services Sector: Data Warehouses find themselves to be of use in the service sector for
maintenance of financial records, revenue patterns, customer profiling, resource management,
and human resources.
3. Manufacturing and Distribution Industry: A manufacturing organization has to take
several make-or-buy decisions which can influence the future of the sector, which is why they
utilize high-end OLAP tools as a part of data warehouses to predict market changes, analyse
current business trends, detect warning conditions, view marketing developments, and
ultimately take better decisions.
4. Banking: Data warehouse used in the banking industry for analysing consumer data, market
trends, government regulations and reports. It is more importantly used for financial decision
making.
1.3 DATA MINING: -
 Large amount of data is available in different industries and organizations. The availability of
this huge data is of no use unless it is converted into valuable information. data mining which
is the separation of useful information from the huge amount of data that is available.
 Data mining is defined as: "Data mining, also known as Knowledge Discovery in Data
(KDD), is the process of uncovering patterns and other valuable information from large data
sets". It is mining knowledge from data process of extracting hidden information from a large
data set.
 Data Mining deals with discovery of hidden knowledge, unexpected patterns, and new rules
from large data sets. Data mining uses data from various data sources and that data need to be
integrated pre-processed before data mining can be done.

Advantages of Data Mining: -

 The data mining helps financial institutions and banks to identify probable defaulters and
hence will help them whether to issue credit card, loan etc. or not.
 The data mining-based methods are cost effective and efficient compared to other statistical
data applications.
 It has been used in many different areas or domains viz. bioinformatics, medicine, genetics,
education, agriculture, law enforcement, e-marketing, electrical power engineering etc.

Disadvantages of Data Mining: -

 The information obtained based on data mining by companies can be misused against a group
of people.
 The data mining techniques are not 100% accurate and may cause serious consequences in
certain conditions.
 The selection of the right data mining tool is a tedious and cumbersome task as one needs to
obtain knowledge of algorithms, features etc. of various available tools.

Applications of Data Mining: -

1. Education: Educational data mining deals with developing the methods to discover the
knowledge from the education field. It is used to find out project student’s areas of interests,
future learning capacities and other aspects.
2. Health and Medicine: Data mining can effectively be used in health care systems. During
Covid-19 pandemic, the predictions of the Covid-19 waves and the volume of patients was
done using data mining.
3. Market Analysis: Market analysis is based on a particular pattern of purchase followed by
customers.
4. Fraud Detection: A fraud detection system helps in finding out the pattern of fraud, its
potential attackers/criminal detection and possible solutions using different data mining
algorithms.
1.4 SPARK: -
Spark is a general-purpose distributed data processing engine that is suitable for use in a wide
range of circumstances. Spark is an accessible, intense, powerful, and proficient Big Data tool for
handling different enormous information challenges. It is an open source, wide range data
processing engine. That reveals development API’s, which also qualifies data workers to
accomplish streaming, machine learning or SQL workloads which demand repeated access to data
sets. it is designed in such a way that it integrates with all the Big Data tools. Like spark can
access any Hadoop data source, also can run on Hadoop clusters. Spark supports the Java, Scala,
Python, and R languages.

Applications of Spark:
1. Machine Learning: Machine learning approaches become more feasible and increasingly
accurate due to enhancement in the volume of data. As spark is capable of storing data in
memory and can run repeated queries quickly, it makes it easy to work on machine learning
algorithms.
2. Data Integration: The data generated by systems are not consistent enough to combine for
analysis. To fetch consistent data from systems we can use processes like Extract, Transform,
and Load (ETL). Spark is used to reduce the cost and time required for this ETL process.
3. Processing Streaming Data: It is always difficult to handle the real-time generated data such
as log files. Spark is capable enough to operate streams of data and refuses potentially
fraudulent operations.
4. Interactive Analytics: Spark is able to generate the respond rapidly. So, instead of running
pre-defined queries, we can handle the data interactively.

Chapter 2. Artificial Intelligence

1. State Space Search: -
A state space represents a problem in terms of states and operators that change states. A
representation of the states the system can be in a board game, the board represents the current
state of the game. A set of operators that can change one state into another state. An initial state.
A set of final states; some of these may be desirable, others undesirable. This set is often
represented implicitly by a program that detects terminal states.

2. State Space Representation of Water Jug Problem

Definition: -
 In this problem, we use two jugs called four liter and three liters, four holds a maximum
of four liters of water and three a maximum of three liters of water. There is a pump that
can be used to fill the jugs with water.
State Space Representation:
The state space is a set of prearranged pairs giving the number of liters of water in the pair of
jugs at any time, i.e., (x, y) where,
x = 0, 1, 2, 3 or 4.
y = 0, 1, 2 or 3.
Where,

x - Represents the number of liters of water in the 4-liter jug.

y - Represents the number of liters of water in the 3-liter jug.
The start state is (0, 0) and the goal state is (2, n), where n may be any, but it is limited to
three holding from 0 to 3 liters of water or empty.

Step 1: We have two jugs; one is 4 liters and the other is 3 liters.

Step 2: Fill a 3 liter jug with water.

Step 3: Pour water from a 3 liter jug into a 4 liter jug.

Step 4: Fill again 3 Liter jug with water.

Step 5: Pour 3 liter jug into 4 liter jug till it is filled.

Step 6: Pour all water from a 4 liter jug on the ground.

Step 7: Pour the water from a 3 liter jug into a 4 liter jug.

3. State Space Representation of Tower of Hanoi

Definition:
Tower of Hanoi is a mathematical game puzzle where we have three pillars and n
numbers of disks.

Rules of game:

This game has following rules:

 Only one disk will move at a time.

 The larger disk should always be on the bottom and the smaller disk on top of it
(Even during intermediate move).
 Move only the uppermost disk.
 All disks move to the destination pillar from the source pillar.

Solution:
Disk 1 moved from A to C
Disk 2 moved from A to B
Disk 1 moved from C to B
Disk 3 moved from A to C
Disk 1 moved from B to A
Disk 2 moved from B to C, Disk 1 moved from A to C
4. State Space Representation of Missionary Cannibal Problem
Problem Statement:
Three missionaries and three cannibals are present at one side of a river and need to cross the
river. There is only one boat available. At any point of time, the number of cannibals should not
outnumber the number of missionaries at that bank. It is also known that only two persons can
occupy the boat available at a time.
Production Rules:
Rule 1: (0, M): One missionary sailing the boat from bank-1 to bank-2
Rule 2: (M, 0): One missionary sailing the boat from bank-2 to bank-1
Rule 3: (M, M): Two missionaries sailing the boat from bank-1 to bank-2
Rule 4: (M, M): Two missionaries sailing the boat from bank-2 to bank-1
Rule 5: (M, C): One missionary and one Cannibal sailing the boat from bank-1 to bank-2
Rule 6: (C, M): One missionary and one Cannibal sailing the boat from bank-2 to bank-1
Rule 7: (C, C): Two Cannibals sailing the boat from bank-1 to bank-2
Rule 8: (C, C): Two Cannibals sailing the boat from bank-2 to bank-1
Rule 9: (0, C): One Cannibal sailing the boat from bank-1 to bank-2
Rule 10: (C, 0): One Cannibal sailing the boat from bank-2 to bank-1

5. Control Strategy:
 Control strategies are also called as Search strategies. These are adopted for applying
the rules and searching the problem solution in search space.
 To solve the problem, we must use a good control strategy which ultimately takes us to
the goal state.
 The control strategy is responsible for obtaining the solution of the problem. if the
wrong control strategy is applied, it may be possible that a solution is never obtained,
even if it exists.
Features of good control strategies:
 A good control strategy is that it causes motion
 A good control strategy should be systematic.

6. SEARCH TECHNIQUES:
1. BLIND SEARCH TECHNIQUES: - Uninformed search or Blind search is the search
methodology having no additional information about states beyond that provided in the
problem definitions. In this search total search space is looked for solutions. blind search
algorithm uses only the initial state, search operators and test for a solution
The different types of search algorithms are as follows:
1. Breadth First Search
2. Depth First Search
 1.1 Breadth First Search: - Breadth First searches are performed by exploring all nodes at a
given depth before proceeding to the next level. This means that all immediate children of
nodes are explored before any children's children are considered.
Construct a tree with the initial state as its root. Generate all its successors by applying all the
rules that are appropriate. for each leaf node, generate all its successors by applying
appropriate rules. The tree at this point is Continue this process until some rule produces a
goal state. This process is called Breadth First search.

o Algorithm: Breadth-first Search

1. Create a variable called NODE-LIST and set it to the initial state.
2. Until goal state is found or NODE-LIST is empty:
a. Remove the first element from NODE-LIST and call it E. If NODE-LIST was
empty, quit.
b. For each way that each rule can match the state described in E do:
[Link] the rule to generate a new state.
[Link] the new state is a goal state, quit and return to this state.
[Link], add the new state to the end of NODE-LIST.

1.2 Depth First Search: -

Depth first searches are performed by going downward into a tree as early as possible.
Consider a single branch of the tree until it produces a solution or until a decision to terminate
the path is made. It makes sense to terminate a path if it reaches a dead end, produces a
previous state or becomes longer than some limit, in such cases backtracking occurs. To
overcome such backtracking is known as Depth First Search.

Algorithm of Depth First Search:

1. If the initial state is a goal state, quit and return success.
2. Otherwise, do the following until success or failure is signalled:
a) Generate a successor, E, of the initial state. If there are no more successors, signal
failure.
b) Call depth first search with E as the initial state.
c) If success is returned, signal success. Otherwise continue in this loop.
 1.3 Bidirectional Search: -
Bidirectional search algorithm runs two simultaneous searches, one from initial state called as
forward-search and other from goal node called as backward-search, to find the goal node.
Bidirectional search replaces one single search graph with two small subgraphs in which one
starts the search from an initial vertex and other starts from goal vertex. The search stops
when these two graphs intersect each other. Bidirectional search can use search techniques
such as BFS, DFS, DLS, etc.

Advantages:

1. Bidirectional search is fast.

2. Bidirectional search requires less memory.

Disadvantages:

1. Implementation of the bidirectional search tree is difficult.

2. In bidirectional search, one should know the goal state in advance.

1.4 Uniform Cost Search: - Uniform Cost Search is a searching algorithm used for traversing a
weighted tree or graph. This algorithm comes into play when a different cost is available for
each edge. The primary goal of the Uniform Cost Search is to find a path to the goal node
which has the lowest cumulative cost. Uniform Cost search expands nodes according to their
path costs from the root node. Uniform Cost Search algorithm is implemented by the priority
queue. It gives maximum priority to the lowest cumulative cost. Uniform Cost Search is
equivalent to the BFS algorithm if the path cost of all edges is the same.

Algorithm:

1. Insert root node into the queue.

2. Repeat till queue is not empty:
(a) Remove the next element with the highest priority from the queue.
(b) If the node is a destination node, then print the cost and the path and
exit. Else insert all the children of removed elements into the queue
with their cumulative cost as their priorities.
 2. HEURISTIC SEARCH TECHNIQUES: -These are the search techniques where additional
information about the problem is provided in order to guide the search in a specific direction.
When we need to solve hard problems, it often becomes necessary to compromise with
requirements of mobility and systematicity. We need to construct a control structure that is no
longer guaranteed to find the best answer but almost always find the good answer.
following search techniques.
o Generate-and-test
o Simple Hill Climbing
o Best First Search
o Constraint Satisfaction
o Means End Analysis
o A* and AO*

2.1 Simple Hill Climbing: - Hill Climbing is a form of heuristic search algorithm which is used
in solving optimization related problems in Artificial Intelligence domain. Simple Hill
Climbing is the simplest form of the Hill Climbing Algorithm. It is so
called because of the way the nodes are selected for expansion. In the search path at
each point, the successor node that appears to lead most quickly to the top of the hill
is selected for exploration. Hill Climbing is a variation of the DFS [Generate-and-
Test] algorithm in which feedback is used to decide in which direction to move in search
space.

Different regions in the State Space Diagram:

i. Local Maximum:
It is a state which is better than its neighbouring state however there exists a state which is
better than it (Global Maximum). This state is better because here the value of the objective
function is higher than its neighbours.
ii. Global Maximum:
It is the best possible state in the state space diagram. This is because in this state, the
objective function has the highest value.
iii. Plateau/flat local maximum:
It is a flat area of the search space in which all neighboring states [nodes] have the same
value. Actually, plateau is an area of the state space landscape where the evaluation function
is flat.
iv. Ridge:
It is a region which is higher than its neighbours but itself has a slope. It is a special kind of
local maximum.
 v. Current State:
The region of state space diagram where we are currently present during the search.
vi. Shoulder: It is a plateau that has an uphill edge.

2.2 Best-First Search: -

Best-First Search is a way of combining the advantages of both Depth and Breadth First
Search. We will call a graph an OR - graph, since each of its branches represents alternative
problem-solving path. The Best-First Search selects the most promising of the node we have
generated so far. This can be achieved by applying appropriate Heuristic function to each of
them. At any point in the search process best first moves forward from the most promising of
all the nodes generated so far.

Heuristic function:

f(n) = h(n)

where,

h(n) - estimated straight line distance from node n to goal.

Best-First Search Algorithm:

Step 1: Place the starting node into the OPEN list.

Step 2: If the OPEN list is empty, Stop and return failure.

Step 3: Remove the node n, from the OPEN list which has the lowest value of h(n),

and places it in the CLOSED list.

Step 4: Expand the node n, and generate the successors of node n.

Step 5: Check each successor of node n, and find whether any node is a goal node or not. If
any successor node is a goal node, then return success terminate the search, else proceed to
Step 6.
Step 6: For each successor node, the algorithm checks for evaluation function f(n), and then
checks if the node has been in either OPEN or CLOSED list. If the node has not been in both
lists, then add it to the OPEN list.
Step 7: Return to Step 2.

2.3 Generate-and-Test: -
The generate-and-test is the simplest approach among all the algorithms. Usually, it helps in
finding out the solution but not always. Generate-and-Test Search is a heuristic search
technique based on Depth-First Search with Backtracking which guarantees to find a solution
if done systematically and there exists a solution. In this technique, all the solutions are
generated and tested for the best solution. It ensures that the best solution is checked against
all possible generated solutions.
This algorithm works in two modules:
a) Generator Module: It creates the possible solution.
b) Tester Module: It tests or evaluates each of the proposed solution either accepting or
rejecting the solution.
 2.4 Means End Analysis (MEA): -
 The purpose of Means End Analysis is to identify a procedure that causes a transition
from the current state to a goal state or at least to an intermediate state that is closer to
the goal state.
 Means end analysis is a technique used to solve problems in AI which combines
forward and backward strategies to solve complex problems. Using these mixed
strategies, complex problems can be solved first, followed by smaller ones.
 the system evaluates the differences between the current state and the goal state. It then
decides the best action to be undertaken to reach the End goal.
2.5 A* and AO*:
1. A*:
 The A* Algorithm is a specialization of best-first search. It provides general
guidelines with which to estimate goal distances for general search graphs.
 the A* algorithm generates all successor nodes and computes an estimate of the
distance cost from the start node to a goal node through each of the successors.
 The form of the heuristic function for A* is,
f(n) = g(n) + h(n)
 A* Algorithm:
Step 1: Place the starting node in the OPEN list.
Step 2: Check if the OPEN list is empty or not, if the list is empty then return
failure and stop.
Step 3: Select the node from the OPEN list which has the smallest value of
evaluation function (g + h), if node n is goal node, then return success and stop,
otherwise.
Step 4: Expand node n and generate all of its successors, and put n into the
closed list. For each successor n', check whether n' is already in the OPEN or
CLOSED list, if not then compute the evaluation function for n' and place it into
the Open list.
Step 5: Else if node n' is already in OPEN and CLOSED, then it should be
attached to the back pointer which reflects the lowest g(n') value.
 Find the most cost-effective path to reach the final state from initial state using A*
Algorithm.
2. AO* (AND-OR graphs):
 When a problem is divided into sub problems, where each sub problem can be
solved separately and a combination of these will be a solution, AND-OR graphs or
AND-OR trees are used for representing the solution
 AO* Graph is a search graph used in Artificial Intelligence to solve problems that
can be divided into subproblems. It is mainly used in heuristic search techniques.
AND nodes – all child nodes must be solved.
OR nodes – only one child node needs to be solved.
 An AO* graph is a directed graph that represents problem decomposition where
some problems are solved by choosing one alternative (OR) and some by
completing all subproblems together (AND).

Chapter 4: Data Warehousing

1. INTRODUCTION TO DATA WAREHOUSE: -
 Data warehouse is a concept which supports decision support systems where a large
amount of data is merged together.
 A Data Warehouse (DW) is a repository which is at the top of multiple databases. It
can be defined as a process for collecting and managing data from different sources to
provide meaningful business insights.
 A data warehouse is a subject-oriented, integrated, time-variant, and non-volatile
(once the data is moved to a warehouse, it doesn't change) collection of data in
support of management’s decision-making process.

Advantage’s:

1. Better Decision Making: Helps managers analyze data and make informed
business decisions.
2. Data Integration: Combines data from multiple sources into one system.
3. Data Consistency & Quality: Cleaned and standardized data improves accuracy.
4. Supports Data Mining: Useful for advanced analysis techniques.

Disadvantage’s:

1. High Cost: Expensive to build, maintain, and manage.

2. Time-Consuming Setup: Takes a long time to design and implement.
3. Data Latency: Data may not always be real-time (updated periodically).
4. Maintenance Effort: Needs continuous updates and monitoring.

2. STRUCTURE OF DATA WAREHOUSE: -

1. Star Schema: - Star Schema is the most common schema in data warehouses. This is
widely used to design the data warehouse. The basic architecture of star schema includes
one fact table and many dimension tables. It is called a star schema because the diagram
looks like a star, with points radiating from a centre. Star schema contains one fact table
associated with many dimension tables. The centre of the star contains fact table and the
points of the star are the dimension tables.

2. Snowflake Schema: - Snowflake Schema is a modification of star schema where some

dimensional hierarchy is normalized into a set of smaller dimension tables, forming a
shape similar to snowflake. Snowflake schema is an arrangement of tables in a
multidimensional database system.
 3. ARCHITECTURE OF DATA WAREHOUSE: -
Data warehouse architecture defines the arrangement of data in different databases. As the
data must be organized and cleansed to be valuable, a modern data warehouse structure
identifies the most effective technique of extracting information from raw data.

Basic Components of Data Warehousing:

a. Data Migration: Data migration is a one-time process of transferring internal data

from one storage system to another; it may include preparing, extracting, and, if
necessary, transforming the data.
b. The Warehouse Database: The central component of a data warehousing architecture
is a database that stocks all enterprise data and makes it manageable for reporting.
c. Access Tools: These tools fall into four main categories: query and reporting tools,
application development tools, online analytical processing tools, and data mining
tools.

Properties for Data Warehouse System Architecture:

a. Separation: Analytical and transactional processing should be kept separately as much as

possible.
b. Scalability: Hardware and software architectures should be easy to upgrade. Because the data
volume and the number of users’ requirements progressively increase.
c. Security: Monitoring accesses is necessary because of the strategic data stored in data
warehouses.
d. Extensibility: Without redesigning the whole system, the architecture should be able to host
new applications and technologies.
e. Ability to administer: Data warehouse management must be easy.

ETL Tools:

 The data is extracted from the operational system. This extracted data must be reformatted,
integrated, cleaned, and summarized before loading into a warehouse.
 The data which is not required gets removed in the conversion process. This process is done
with the various tools available for transformation, cleaning, summarization and loading.
These tools are called as ETL tools (Extract, Transform and Load).
 The main responsibilities of ETL tools include:

- To remove the identity specifications of the data according to the rules.

- To remove unwanted data from the operational database before loading it to the data
warehouse
- To find and replace common names and definitions as the data is arriving from multiple
destinations.
- To summarize the data.
- To recover the data with default values if there are missing values.
- To remove the redundancy in data.

Types of Data warehouse Architectures:

1. Single-tier Architecture: - The purpose of a single-tier is to minimize the amount of data
stored. This goal is to remove data redundancy. This is a basic architecture of data warehouse
which is not frequently used today.
2. Two-tier Architecture: - Two-tier architecture separates physically available sources and
data warehouse. This architecture is not expandable. This is also not supporting a large
number of end-users. It has connectivity problems because of network limitations.
3. Three-tier Architecture: -
i. Bottom Tier: Bottom tier consists of an actual data warehouse server. It includes
summary data and metadata repository. Summary data is replication of detailed
information which stores a summary of the data present in the data warehouse.
Metadata is data about data. It is the card index describing how information is
structured within the data warehouse.
ii. Middle Tier: The middle tier is an OLAP server. It is implemented using a
Relational OLAP model or Multidimensional OLAP. Stores processed data in a
central Data Warehouse.
iii. Top Tier: Top tier consists of front-end tools that are used for querying the database
which is present in the data warehouse to get the information for analysis.

4. MULTIDIMENSIONAL DATA MODEL: -

 A multidimensional model views data in the form of a data-cube. Mostly, data
warehousing supports two or three-dimensional cubes.
 A data cube enables data to be modelled and viewed in multiple dimensions. It is
defined by dimensions and facts. Dimensions are organizations about which an entity
needs to hold information.
 Data warehouses and Online Analytical Processing (OLAP) tools are based on a
multidimensional data model. OLAP in data warehousing enables users to view data
from different angles and dimensions.

5. OLAP VS. OLTP: -

  Operational Database Management Systems also called as OLTP (Online
Transactions Processing Databases), are used to manage dynamic data in real-time
whereas Data Warehousing/Online Analytical Processing (OLAP) are used for large-
scale processing and analysis of data.

6. OLAP OPERATIONS: -
 OLAP operations are done on multidimensional data. This multidimensional data is
organized in various dimensions. Every dimension includes multiple levels of
abstraction. So, there are various OLAP operations to demonstrate these views

OLAP operations:

1. Roll-up Operation: o Roll-up operation performs aggregation on a data cube by either

climbing up a hierarchy for a dimension or by reducing the dimensions. When roll-up is
performed, some dimensions are reduced from the data.

2. Drill-down Operation: The drill-down operation (also called roll-down) is the reverse
operation of rollup.
 It is performed by either of the following ways:
- By stepping down a concept hierarchy for a dimension.
- By introducing a new dimension.

3. Slice and Dice Operation: The slice operation performs a selection on one dimension of the
given cube, resulting in a sub-cube. It reduces the dimensionality of the cubes.

4. Pivot (Rotate) Operation: o The pivot operation is also known as rotation. It rotates the data
axis in view in order to provide an alternative presentation of data. Consider the following
diagram that shows the pivot operation.

7. TYPES OF OLAP SERVERS: -

1. Relational OLAP (ROLAP):
 ROLAP stands for Relational Online Analytical Processing. ROLAP servers act as
mediators between backend server and user front end tools. The relational databases
handle warehouse data and OLAP acts as middleware to provide missing data.
 The main advantage of using ROLAP is that it can handle large amount of
information as RDBMS comes with lots of functionalities and ROLAP works on top
of that.
2. Multidimensional OLAP (MOLAP):
 MOLAP system is based on multidimensional data model and operations. Data is
stored in multidimensional arrays and positional techniques are used to access them.
Here, data is summarized and stored in a multidimensional cube. Relational model is
not used to store the data.
 MOLAP is extensively used in applications where iterative and comprehensive time-
series analyses of trends are done.

3. Hybrid OLAP (HOLAP):

 HOLAP stands for Hybrid OLAP, an application using both relational and
multidimensional techniques.
 HOLAP includes the best features of MOLAP and ROLAP into a single architecture.
HOLAP offers greater scalability than ROLAP and faster computation than MOLAP.
 HOLAP stores a large amount of data and it provides fast access at all levels of
aggregation.
 HOLAP balances the disk space requirement, as it stores the aggregate information
on the OLAP server and the detail record remains in the relational database. So, no
duplicate copy of the detail record is maintained.
CHAPTER 5: Data Mining

1. INTRODUCTION TO DATA MINING: -

 Mining refers to the extraction of valuable things. Data mining, in turn, refers to the study
of collecting, cleaning, processing and analyzing the data and to retrieve meaningful
information from huge data.
 Data Mining is analysis of data and use of software techniques and statistical methods to
find patterns in data.
 Data Mining deals with discovery of hidden knowledge, unexpected patterns and new
rules from large data sets.
  It is also known as the process of extracting hidden information from a large data set
mining knowledge from data.

2. DATA MINING TASK: -

A. Predictive Data Mining: -
Predictive data mining tasks include the prediction based on the available data set in
hand. These tasks give the model based on data and predict the future trends related to
that data or unknown values that may be of interest for the future. Predictive data mining
includes Classification, Regression, Prediction and Time Series Analysis.

1. Classification:
 In data mining, classification can be defined as arrangement of data items or
making groups of data items based on the data points or observed values. The
output of classification is a method that will decide the class of an object
based on its attributes.
2. Regression:
 Regression can be defined as a data mining technique that is generally used
for the purpose of predicting a range of continuous values (which can also be
called "numeric values") in a specific data set.
 It is used to map data items to a real valued variable. Regression is very
frequently used in business and market analysis. The main application
involves financial prediction or forecasting, Environmental modeling and
analyzing trends and patterns.
i. In Linear regression, the relationship between two variables is established
using a linear equation to observe the data. The output is a straight line which
has only line dependent variables.
ii. In Multiple regression, the relationship between two or more variables is
established to predict the output and a single continuous dependent variable.
3. Prediction:
 Prediction discovers the relationship between dependent variables and
relationship between independent variables.
 The prediction is based on the data in hand and predictions or future trends of
a phenomenon can be predicted using some predictive algorithms.
 Predication is the technique of identifying the unavailable numerical data for
a new process. Prediction applications include flooding, speech recognition,
machine learning, and pattern recognition.

4. Time Series Analysis:

 Time series analysis is the process of recording the data point at specific time
intervals. This data is then used to predict the future values based on the data
points recorded.
 Time series analysis can produce very important information for a business if
used efficiently.

B. Descriptive Data Mining: -

Descriptive data mining tasks include the analysis of available data patterns or models to
find out new interesting and significant information based on available data set.
 Descriptive data mining includes Clustering, Summarization, Association Rules and
Sequence Discovery.

1. Clustering:
 Clustering / Cluster Analysis is the method where the data points are grouped
together according to their characteristics. The data points in the same group
(called a cluster) are more similar to each other than to those in other groups
(clusters).
 Clustering can be used to find out the specific class of customer in the
market.
2. Summarization:
 Summarization is the process of finding the list of the generated data. The
process of Summarization divides the data into subsets with descriptions.
Summarization is also called Characterization or Generalization.
 It extracts or derives representative information about the database. This may
be accomplished by actually retrieving portions of the data.
3. Association Rules:
 Association rules find out the correlation among the data. Association rules
find out a specific type of association between the data items.
 These associations are used to identify the frequency occurrence in the
pattern and accordingly the strategies for business are changed or modified.
4. Sequence Discovery:
 Sequence discovery, or Sequential Pattern Mining, is a data mining technique
that discovers statistically relevant patterns in sequential data. This mining
program evaluates certain criteria, such as occurrence frequency, duration, or
values in a set of sequences to find interesting hidden patterns.

3. KDD (Knowledge Discovery in Databases): -

 The KDD process (Knowledge Discovery in Database process) is the process

of digging or finding the truth laid in databases which is not known yet and
the things which are previously not discovered.
 The stuff that has not been detected yet can be found with the KDD process.
This extraction of unknown stuff through the KDD process is useful in
automating summarization, pattern recognition and finding out the truth from
facts and figures.

The KDD process is divided into following steps:

1. Selection:
 The data which is to be mined may not be necessarily from a single source. The data
may have many heterogeneous origins. This data needs to be obtained from various
data sources and files. The data selection is based on your mining goal. Data relevant
to the mining task is selected from various sources.

2. Pre-processing:
 Pre-processing involves cleaning of the data and integration of the data. The data
selected for mining purposes may have some incorrect, irrelevant values which lead
to unwanted results. Some values may be missing or erroneous.
  when data is collected from heterogeneous sources, it may involve varying data types
and metrics. So, this data needs to be cleaned and integrated for noise elimination and
inconsistency.
3. Transformation:
 Data transformation is the process of converting the data into the format which is
suitable for processing. Here, data is created in the form which is required by the data
mining process.
4. Data Mining:
 The Data Mining process leads towards using methods, techniques to extract the
pattern present in the data. The process involves transformation of relevant data
records into patterns using classification.
 This step involves application of various data mining algorithms to the transformed
data. This process generates the desired results for which the whole KDD process is
undertaken.
5. Visualization/Interpretation:
 This is the last step in the KDD process. In this step, the data is presented to the user
in the form of reports, tables or graphs. The presentation of the data to the users
directly affects the usefulness of the results.

4. DATA PRE-PROCESSING: -

Pre-processing Process: The process of cleaning the data and making it useful for the
process of mining is called Data Preprocessing. Pre-processing can be performed manually or
automatically. the steps involved in Data Pre-processing are Data Cleaning, Transformation,
and Data Reduction.
1. Data Pre-processing:
The first step in data pre-processing is data cleaning. It is also known as scrubbing. Data
cleaning includes handling missing data and noisy data.
 (a) Missing data: Missing data is the case wherein some of the attributes or attribute
data is missing or the data is not normalized. This situation can be handled by either
ignoring the values or filling the missing value.
(b) Noisy data: This is data with error or data which has no meaning at all. This type of
data can either lead to invalid results or can create the problem to the process of
mining itself. The problem of noisy data can be solved with binning methods,
regression and clustering.
2. Data Integration and Transformation:
 Data integration is the process of combining data from disparate sources into a
meaningful and valuable data set for the purpose of analysis.
 a logical data source is prepared. This is done by collecting and integrating data from
multiple sources like databases, legacy systems, flat files, data cubes etc.
 Data used for data mining is the data which comes from various heterogeneous
platforms. This unstructured and structured data needs to be combined for smooth
processing of data mining. This homogeneous data is then analyzed to find out the
patterns.
 The various data transformation methods include:
(a) Smoothing: This is the process of removing the unnecessary data and cleaning
the data so as to improve the functionality of the data.
(b) Aggregation: This is the process of collecting the data from heterogeneous
platforms and converting it to a uniform format. This improves the quality of the
data.
(c) Discretization: Large data sets are complex to handle. Discretization is the
process of breaking up the data in small intervals. These chunks are continuous
chunks and these are supported by all the existing frameworks.
(d) Attribute construction: To improve the efficiency in the mining process, some
new attributes are generated from existing data sets.
(e) Generalization: This is the process of converting low level attributes to high
level attributes using hierarchy.
(f) Normalization: In the process of Normalization, attributes are scaled within a
specified range.

3. Data Reduction:
 Data reduction is a process that reduced the volume of original data and
represents it in a much smaller volume.
 The various data reduction methods include:
(a) Attribute Selection: When data is collected from various sources, it may
contain duplicate attributes. Some of the attributes are irrelevant. The
Attribute Selection method is used to remove such redundant and
unnecessary attributes from the data set. This process results in an improved
data set.
(b) Data Cube Aggregation: In this reduction method, aggregation property is
applied on selected data sets so as to get the data in a much simpler format.
 (c) Numerosity Reduction: In this reduction method, actual data is substituted
with a mathematical model of the data.
(d) Dimensionality Reduction: In this reduction method, duplicate attributes are
removed to reduce the data size.

5. ACCURACY MEASURES: -

1. Accuracy Measures:

 The accuracy of a classifier is given as the percentage of total correct predictions divided by
the total number of instances.
 The information system consists of a number of different documents. And the various
operations are done on these documents to retrieve useful information.
 The information is retrieved using queries. The similarity between the query and the retrieved
document is calculated. This similarity measure is a set membership function describing the
likelihood of the document that the retrieved document is relevant to user’s query.

[Link] and Recall:

 The effectiveness of the system in processing a query is measured by precision and recall.
 Precision and Recall are calculated by,
Precision = [Relevant and Retrieved] / [Retrieved]
Recall = [Relevant and Retrieved] / [Relevant]

3.F-Measure:

 F-measure or F-score is a measure of accuracy of a model on a data set. A measure that

combines precision and recall.
 The F-score is used for evaluating information retrieval systems such as search engines, an
also in natural language processing.

F = 2 – (Precision - Recall / Precision + Recall)

[Link] Matrix:

 A Confusion Matrix describes the accuracy of the solution to a classification problem. A

confusion matrix is a table that is often used to describe the performance of a classification
model.
 The matrix consists of predictions result in a summarized form, which has a total number of
correct predictions and incorrect predictions.
[Link] Validation:

 Cross Validation is a standard tool used to develop and fine-tune data mining models. In
Cross Validation, we train our model using the subset of the data set and then evaluate using
the complementary subset of the data set.
 The steps involved in Cross Validation are as follows:
o Reserve some portion of sample data set.
o Using the rest data set, train the model.
o Test the model using the reserve portion of the data set.

[Link]:

 The bootstrap method involves iteratively re-sampling a data set with replacement. The
bootstrap method samples the given training tuples uniformly with replacement.
 The major application of bootstrapping includes repeated sampling methods to build a more
confident measurement.

6. DATA MINING TECHNIQUES: -

1. Statistical techniques:
 Statistical techniques are at the core of most analytics involved in the data mining
process. The different analytics models are based on statistical concepts, which
output numerical values that are applicable to specific business objectives.
2. Classification:
 This technique is used to obtain important and actual information about data and
metadata.
 It is considered to be a complex data method among other data mining techniques.
Information is classified into different classes.
3. Clustering:
 In this technique, the pieces of information are grouped according to their
similarities. This technique helps to recognize the differences and similarities
between the data.

4. Regression:
 This data mining tool is designed to identify and analyze the interactions between
different variables. It's used for identification of the probability of a particular
variable from other variables' existence. This method is also known as predictive
power.
 Regression analysis is also used to predict the future value of a specific entity (the
given feature could be either linear or nonlinear). Ultimately the goal of
regression is to show the links between two pieces of information in one set.
 5. Association:
 This mining data technique is used to find an association between two or more
events or properties. It drills down to an underlying model in the database
systems.
6. Outer detection (Outlier analysis):
 This type of data mining technique relates to the observation of data items in
the data set, which do not match an expected pattern or expected behaviour.
This technique may be used in various domains like intrusion, detection, fraud
detection, etc. It is also known as Outlier Analysis or Outlier Mining.
7. Prediction:
 Prediction is considered to be an essential data mining technique. It used a
combination of other data mining techniques such as clustering, classification,
etc. To predict a future event, it analyzes instances or past events in the right
sequence.
8. Sequential patterns:
 This technique of data mining helps to discover or recognize similar patterns
in transaction data over some time.
9. Decision trees:
 This technique of data mining helps to discover or recognize similar patterns
in transaction data over some time. Decision trees are a specific type of
predictive model that lets organizations effectively mine data.
 This type of data mining tool is used quite often as it's the simplest for
understanding. At the root of such decision trees, there is a simple question
with many possible answers. Based on the responses, we can get the final
answer to the central question.

7. FREQUENT ITEMSETS AND ASSOCIATION RULE MINING: -

1. Frequent Itemset:
 The common method to find out the association rule in the mining task is to break
the problem into two parts:
o Find the large itemset.
o Generate rules from the frequent itemset.

ARGen algorithm:

1. R = φ; 4. if support(l) >= α then

2. for each l ∈ L do 5. support(x)

3. for each x ⊂ 1 such that x ≠ φ do 6. R=R U { x ==> (1 - x)};

2. Association Rule Mining:

 Use of data mining is to extract useful patterns from data irrespective of type of
data. Finding patterns is nothing but finding relationships among data. These
relationships help users to create groups of data items.
 The kind of purchasing items in group denotes some kind of relationships among
them. This relationship in data mining is called as association rule.
 These association rules are often used in marketing, advertising, inventory
applications and in retail business.
  The database should be in the form of tuples where association rule are to be
found. Each tuple is the list of items purchased at one time.
3. Apriori Algorithm:
This is one of the well-known best algorithms for generating association rules.
It is powerful algorithm for mining frequent itemsets for Boolean association rules.
Algorithm Apriori_generate(Lk):
1. for each itemset l1 in Lk
2. for each itemset l2 in Lk
3. If k − 1 elements in l1 and l2 are equal
// If l1[1] = l2[1] and l1[2] = l2[2] and … l1[k − 1] = l2[k − 1] and
//l1[k] < l2[k]
4. C = l1 × l2
5. add C to Ck + 1
6. for each k subset s of c
7. if s does not belong to Lk then
8. delete c
9. break
The Apriori Algorithm:
Ck: Candidate itemset of size k
Lk: frequent itemset of size k
L1 = {frequent items};
1. for(k = 1; Lk ! = Ø; k++) do
2. begin
3. Ck+! = Apriori_generate(Lk)
// candidates generated from Lk;
4. for each transaction t in database do
5. increment the count of all candidates in Ck+!
6. that are contained in t
7. Lk + 1 = candidates in Ck+1 with min_support

9. return ∪k Lk;
8. end

Advantages of Apriori Algorithm:

 This algorithm uses breadth-first search.

 Easy to implement.
 Uses large itemset (Apriori) property

4. FP-tree Algorithm:
 FP-tree (Frequent Pattern tree) is an algorithm for mining frequent itemsets from
a database by using association rules. It's an alternative to the apriori algorithm. A
frequent pattern is generated without the need for candidate generation.
 FP growth algorithm represents the database in the form of a tree called a
frequent pattern tree or FP tree.
  Frequent Pattern Tree is a tree-like structure that is made with the initial itemset
of the database. The purpose of the FP tree is to mine the most frequent pattern.
Each node of the FP tree represents an item of the itemset.

8. GRAPH MINING: FREQUENT SUB-GRAPH MINING: -

Graph Mining:
 Graph Mining is the set of tools and techniques used to:
(a) analyze the properties of real-world graphs.
(b) predict how the structure and properties of a given graph might affect some
application.
(c) develop models that can generate realistic graphs that match the patterns found
in real-world graphs of interest.

Frequent Sub-graph Mining:

 It is the process of finding graph structures that occur in a significant number of times
among a set of graphs.

9. SOFTWARE FOR DATA MINING: -

 A Data mining tool is a software application that is used to discover patterns and
trends from large sets of data and transform those data into more refined information.
It helps you to identify unsuspected relationships amongst the data for business
growth.
 Data Mining is the set of techniques that utilize specific algorithms, statistical
analysis, artificial intelligence, and database systems to analyze data from different
dimensions and perspectives.
 Data Mining tools have the objective of discovering patterns/trends/groupings among
large sets of data and transforming data into more refined information.
 It is a framework, such as RStudio or Tableau that allows you to perform different
types of data mining analysis.

1. R
 R is an open-source programming tool developed by Bell Laboratories. R is
a programming language and an environment for statistical computing and
graphics.
 R is popular for data mining as it is used to run a variety of statistical
analysis, such as time-series analysis, clustering, and linear and nonlinear
modelling.
 2. Weka
 Weka is a collection of machine learning algorithms for data mining tasks. It
is open-source software that provides tools for data pre-processing,
implementation of several Machine Learning algorithms. The algorithms can
either be applied directly to a data set or called from your own Java code.
 Weka contains tools for data pre-processing, classification, regression,
clustering, association rules, and visualization. It is also well-suited for
developing new machine learning schemes.
 Weka is comprehensive software that lets you pre-process the big data, apply
different machine learning algorithms on big data and compare various
outputs. This software makes it easy to work with big data and train a
machine using machine learning algorithms.
3. Sample Applications of Data Mining
 Data mining is used by many organizations to improve the customer base. They focus
on customer behavioral patterns, market analysis, profit areas and product improvement.
The essential areas where data mining is used are as follows:
(a) Education
(b) Health and Medicine
(c) Market Analysis
(d) Fraud Detection

Chapter 6: SPARK

1. INTRODUCTION TO SPARK: -
 Apache Spark is an open-source, distributed processing system used for big data
workloads. It is cluster computing designed for fast computation.
 It utilizes in-memory caching, and optimized query execution for fast analytic queries
against data of any size.
 Spark is a general engine for large-data processing. It is based on Hadoop MapReduce
and it extends the MapReduce model to efficiently use it for more types of
computations, which includes interactive queries and stream processing.
Features of Apache Spark:
1. Speed: The main feature of Spark is its in-memory cluster computing that increases the
processing speed of an application.
2. Multiple language support: Spark supports multiple languages. It provides various APIs
written in Java, Scala, Python and R.
3. Multiple platform support: Spark will run on multiple platforms while not moving the
processing speed.
4. Advanced Analytics: Spark not only supports 'Map' and 'reduce'. It also supports SQL
queries, Streaming data, Machine Learning (ML), and Graph algorithms.

2. APACHE SPARK ARCHITECTURE: -

3. COMPONENTS OF SPARK: -

1. Apache Spark:
 Spark Core is the underlying general execution engine for spark platform. All the
other functionality is built upon.
 It provides in-Memory computing and referencing datasets in external storage
systems.
2. Spark SQL:
 Spark SQL is a component above Spark Core. It contains a new data abstraction
called SchemaRDD.
 SchemaRDD provides support for structured and semi-structured data. It supports
many sources of data including Hive tablets, Parquet, JSON.
3. Spark Streaming:
 Spark Streaming leverages Spark Core's fast scheduling capability to perform
streaming analytics. It ingests data in mini-batches and performs RDD (Resilient
Distributed Datasets) transformations on those mini-batches of data.
 Spark uses Micro-batching for real-time streaming. Micro-batching is a technique
that permits a method or a task to treat a stream as a sequence of little batches of
information.

4. MLlib (Machine Learning Library):

 MLlib is a distributed machine learning framework above Spark because of the
distributed memory-based Spark architecture.
  According to benchmarks, it is done by the MLlib developers against the
Alternating Least Squares (ALS) implementations.
5. GraphX:
 GraphX is a distributed graph-processing framework on top of Spark.
 It provides an API for expressing graph computation that can model the user
defined graphs by using the Pregel abstraction API.
 It also provides an optimized runtime for this abstraction.

4. SPARK RDDs: -
Resilient Distributed Datasets (RDD):
 RDD is a fundamental data structure of Apache Spark. It is an immutable collection
of objects which computes on the different node of the cluster.
 Decomposing the name RDD:
o Resilient: fault-tolerant with the help of RDD lineage graph and so able to
recompute missing or damaged partitions due to node failures.
o Distributed: Since Data resides on multiple nodes.
o Dataset: It represents records of the data you work with. The user can load
the data set externally which can be either JSON file, CSV file, text file or
database via JDBC with no specific data structure.

 Formally, an RDD is a read-only, partitioned collection of records. RDD is a fault

tolerant collection of elements that can be operated in parallel.

Features of RDD:

1. Immutability
2. Fault-tolerant
3. Partitioning
4. Persistence
5. Lazy Evaluation
6. In-memory Computation

Methods to create RDDs:

1. Parallelizing collection (Parallelized): Parallelizing an existing collection in your driver

program.
2. Referencing External Dataset: Referencing a dataset in an external storage system, such as
a shared file system, HDFS, HBase, or any data source offering a Hadoop Input format.

5. SPARK SQL AND DATA FRAMES:

1. Spark SQL: -
 Spark SQL is a Spark module for structured data processing. Internally, Spark
SQL uses this extra information to perform extra optimizations.
 There are several ways to interact with Spark SQL including SQL and the Dataset
API. Spark SQL runs on top of the Spark Core. It allows developers to import
relational data from Hive tables and Parquet files.
2. Data Frames: -
 A Data Frame is a distributed collection of data, which is organized into named
columns. Data frames can be compared to relational databases.
 A Data Frame can be constructed from an array of different sources such as Hive
tables, Structured Data files, external databases, or existing RDDs.