By

Mr. Himansu Sekhar Rout

Assistant Professor-SOIS

Presidency University

Bangalore
 Module-4 [Introduction to Data Mining]

Introduction to Data mining, KDD versus data mining, data mining techniques(pre-
processing), tools and applications. Mining complex data objects, Spatial databases,
Multimedia databases, Time series and Sequence data; mining Text Databases and mining
Word Wide Web. Applications of data warehousing across different industries- Retail
industry, Manufacturing and distribution, Bank, insurance company, Government agencies
etc.
Assignment: Data Mining Techniques.

Introduction to Data mining:

 The process of extracting information to identify patterns, trends, and useful data that
would allow the business to take the data-driven decision from huge sets of data is
called Data Mining.
 Data mining is the act of automatically searching for large stores of information to
find trends and patterns that go beyond simple analysis procedures. Data mining
utilizes complex mathematical algorithms for data segments and evaluates the
probability of future events. Data Mining is also called Knowledge Discovery of
Data (KDD).
 Applications of data mining include customer profiling and segmentation, market
basket analysis, anomaly detection, and predictive modelling. Data mining tools and
technologies are widely used in various industries, including finance, healthcare,
retail, and telecommunications.
 In general terms, “Mining” is the process of extraction of some valuable material
from the earth e.g. coal mining, diamond mining, etc. In the context of computer
science, “Data Mining” can be referred to as knowledge mining from data,
knowledge extraction, data/pattern analysis, data archaeology, and data
dredging.

Main Purpose of Data Mining:

Types of Data Mining:

Data mining can be performed on the following types of data:

Relational Database:

A relational database is a collection of multiple data sets formally organized by tables, records,
and columns from which data can be accessed in various ways without having to recognize the
database tables. Tables convey and share information, which facilitates data searchability,
reporting, and organization.

Data warehouses:

A Data Warehouse is the technology that collects the data from various sources within the
organization to provide meaningful business insights. The huge amount of data comes from
multiple places such as Marketing and Finance. The extracted data is utilized for analytical
purposes and helps in decision- making for a business organization. The data warehouse is
designed for the analysis of data rather than transaction processing.

Data Repositories:

The Data Repository generally refers to a destination for data storage. However, many IT
professionals utilize the term more clearly to refer to a specific kind of setup within an IT
structure. For example, a group of databases, where an organization has kept various kinds of
information.

Object-Relational Database:

A combination of an object-oriented database model and relational database model is called an

object-relational model. It supports Classes, Objects, Inheritance, etc.

One of the primary objectives of the Object-relational data model is to close the gap between
the Relational database and the object-oriented model practices frequently utilized in many
programming languages, for example, C++, Java, C#, and so on.

Transactional Database:

A transactional database refers to a database management system (DBMS) that has the potential
to undo a database transaction if it is not performed appropriately. Even though this was a
unique capability a very long while back, today, most of the relational database systems support
transactional database activities.

Data Mining Applications:

Data Mining is primarily used by organizations with intense consumer demands- Retail,
Communication, Financial, marketing company, determine price, consumer preferences,
product positioning, and impact on sales, customer satisfaction, and corporate profits. Data
mining enables a retailer to use point-of-sale records of customer purchases to develop products
and promotions that help the organization to attract the customer.

Advantages of Data Mining

o The Data Mining technique enables organizations to obtain knowledge-based data.

o Data mining enables organizations to make lucrative modifications in operation and
production.
o Compared with other statistical data applications, data mining is a cost-efficient.
o Data Mining helps the decision-making process of an organization.
o It Facilitates the automated discovery of hidden patterns as well as the prediction of
trends and behaviours.
o It can be induced in the new system as well as the existing platforms.
o It is a quick process that makes it easy for new users to analyse enormous amounts of
data in a short time.

Disadvantages of Data Mining

o There is a probability that the organizations may sell useful data of customers to other
organizations for money. As per the report, American Express has sold credit card
purchases of their customers to other organizations.
o Many data mining analytics software is difficult to operate and needs advance training
to work on.
o Different data mining instruments operate in distinct ways due to the different
algorithms used in their design. Therefore, the selection of the right data mining tools
is a very challenging task.
 o The data mining techniques are not precise, so that it may lead to severe consequences
in certain conditions.

Challenges of Implementation in Data mining:

The process of data mining becomes effective when the challenges or problems
are correctly recognized and adequately resolved.

KDD versus data mining:

 KDD (Knowledge Discovery in Databases) is a field of computer science, which

includes the tools and theories to help humans in extracting useful and previously
unknown information (i.e., knowledge) from large collections of digitized data. KDD
consists of several steps, and Data Mining is one of them. Data Mining is the application
of a specific algorithm to extract patterns from data.
 KDD is a computer science field specializing in extracting previously unknown and
interesting information from raw data. KDD is the whole process of trying to make
sense of data by developing appropriate methods or techniques. This process deals with
low-level mapping data into other forms that are more compact, abstract, and useful.
This is achieved by creating short reports, modelling the process of generating data, and
developing predictive models that can predict future cases.

KDD Process Steps

Knowledge discovery in the database process includes the following steps, such as:
1. Goal identification: Develop and understand the application domain and the relevant
prior knowledge and identify the KDD process's goal from the customer perspective.
2. Creating a target data set: Selecting the data set or focusing on a set of variables or
data samples on which the discovery was made.
3. Data cleaning and pre-processing: Basic operations include removing noise if
appropriate, collecting the necessary information to model or account for noise,
deciding on strategies for handling missing data fields, and accounting for time
sequence information and known changes.
4. Data reduction and projection: Finding useful features to represent the data
depending on the purpose of the task. The effective number of variables under
consideration may be reduced through dimensionality reduction methods or conversion,
or invariant representations for the data can be found.
5. Matching process objectives: KDD with step 1 a method of mining particular. For
example, summarization, classification, regression, clustering, and others.
6. Modelling and exploratory analysis and hypothesis selection: Choosing the
algorithms or data mining and selecting the method or methods to search for data
patterns. This process includes deciding which model and parameters may be
appropriate (e.g., definite data models are different models on the real vector) and the
matching of data mining methods, particularly with the general approach of the KDD
process (for example, the end-user might be more interested in understanding the model
in its predictive capabilities).
7. Data Mining: The search for patterns of interest in a particular representational form
or a set of these representations, including classification rules or trees, regression, and
clustering. The user can significantly aid the data mining method to carry out the
preceding steps properly.
8. Presentation and evaluation: Interpreting mined patterns, possibly returning to some
of the steps between steps 1 and 7 for additional iterations. This step may also involve
the visualization of the extracted patterns and models or visualization of the data given
the models drawn.
 9. Taking action on the discovered knowledge: Using the knowledge directly,
incorporating the knowledge in another system for further action, or simply
documenting and reporting to stakeholders. This process also includes checking and
resolving potential conflicts with previously believed knowledge (or extracted).

Data Mining:

Data mining, also known as Knowledge Discovery in Databases, refers to the nontrivial
extraction of implicit, previously unknown, and potentially useful information from data stored
in databases.

Data Mining is only a step within the overall KDD process. There are two major Data Mining
goals defined by the application's goal: verification of discovery. Verification verifies the user's
hypothesis about data, while discovery automatically finds interesting patterns.

There are four major data mining tasks: clustering, classification, regression, and association
(summarization). Clustering is identifying similar groups from unstructured data.
Classification is learning rules that can be applied to new data.

Why do we need Data Mining?

The volume of information is increasing every day that we can handle from business
transactions, scientific data, sensor data, Pictures, videos, etc. So, we need a system that will
be capable of extracting the essence of information available and that can automatically
generate reports, views, or summaries of data for better decision-making.

KDD Vs Data Mining

 KDD is the overall process of extracting knowledge from data, while Data Mining is a
step inside the KDD process, which deals with identifying patterns in data.
 And Data Mining is only the application of a specific algorithm based on the overall
goal of the KDD process.

Data mining techniques:

Classification:
This technique is used to obtain important and relevant information about data and metadata. This
data mining technique helps to classify data in different classes.

Data mining techniques can be classified by different criteria, as follows:

i. Classification of Data mining frameworks as per the type of data sources mined:
This classification is as per the type of data handled. For example, multimedia, spatial data, text
data, time-series data, World Wide Web, and so on..
ii. Classification of data mining frameworks as per the database involved:
This classification based on the data model involved. For example. Object-oriented database,
transactional database, relational database, and so on..
iii. Classification of data mining frameworks as per the kind of knowledge discovered:
This classification depends on the types of knowledge discovered or data mining functionalities.
For example, discrimination, classification, clustering, characterization, etc. some frameworks tend
to be extensive frameworks offering a few data mining functionalities together..
iv. Classification of data mining frameworks according to data mining techniques used:
This classification is as per the data analysis approach utilized, such as neural networks, machine
learning, genetic algorithms, visualization, statistics, data warehouse-oriented or database-oriented,
etc.
The classification can also take into account, the level of user interaction involved in the data mining
procedure, such as query-driven systems, autonomous systems, or interactive exploratory systems.

2. Clustering:
Clustering is a division of information into groups of connected objects. Describing the data by a
few clusters mainly loses certain confine details, but accomplishes improvement. It models data by
its clusters. Data modeling puts clustering from a historical point of view rooted in statistics,
mathematics, and numerical analysis. From a machine learning point of view, clusters relate to
hidden patterns, the search for clusters is unsupervised learning, and the subsequent framework
represents a data concept. From a practical point of view, clustering plays an extraordinary job in
data mining applications. For example, scientific data exploration, text mining, information
retrieval, spatial database applications, CRM, Web analysis, computational biology, medical
diagnostics, and much more.

In other words, we can say that Clustering analysis is a data mining technique to identify similar
data. This technique helps to recognize the differences and similarities between the data. Clustering
is very similar to the classification, but it involves grouping chunks of data together based on their
similarities.

3. Regression:
Regression analysis is the data mining process is used to identify and analyze the relationship
between variables because of the presence of the other factor. It is used to define the probability of
the specific variable. Regression, primarily a form of planning and modeling. For example, we
might use it to project certain costs, depending on other factors such as availability, consumer
demand, and competition. Primarily it gives the exact relationship between two or more variables
in the given data set.

4. Association Rules:
This data mining technique helps to discover a link between two or more items. It finds a hidden
pattern in the data set.

Association rules are if-then statements that support to show the probability of interactions between
data items within large data sets in different types of databases. Association rule mining has several
applications and is commonly used to help sales correlations in data or medical data sets.

Advertisement

The way the algorithm works is that you have various data, For example, a list of grocery items
that you have been buying for the last six months. It calculates a percentage of items being
purchased together.

These are three major measurements technique:

o Lift:
This measurement technique measures the accuracy of the confidence over how often
item B is purchased.
(Confidence) / (item B)/ (Entire dataset)
o Support:
This measurement technique measures how often multiple items are purchased and
compared it to the overall dataset.
(Item A + Item B) / (Entire dataset)
o Confidence:
This measurement technique measures how often item B is purchased when item A is
purchased as well.
(Item A + Item B)/ (Item A)

5. Outer detection:
This type of data mining technique relates to the observation of data items in the data set, which do
not match an expected pattern or expected behavior. This technique may be used in various domains
like intrusion, detection, fraud detection, etc. It is also known as Outlier Analysis or Outilier
mining. The outlier is a data point that diverges too much from the rest of the dataset. The majority
of the real-world datasets have an outlier. Outlier detection plays a significant role in the data
mining field. Outlier detection is valuable in numerous fields like network interruption
identification, credit or debit card fraud detection, detecting outlying in wireless sensor network
data, etc.

6. Sequential Patterns:
The sequential pattern is a data mining technique specialized for evaluating sequential data to
discover sequential patterns. It comprises of finding interesting sub-sequences in a set of sequences,
where the stake of a sequence can be measured in terms of different criteria like length, occurrence
frequency, etc.

In other words, this technique of data mining helps to discover or recognize similar patterns in
transaction data over some time.

7. Prediction:
Prediction used a combination of other data mining techniques such as trends, clustering,
classification, etc. It analyses past events or instances in the right sequence to predict a future event.

Data Mining Implementation Process:

Data mining is described as a process of finding hidden precious data by evaluating the huge
quantity of information stored in data warehouses, using multiple data mining techniques such
as Artificial Intelligence (AI), Machine learning and statistics.
Steps of Data Pre-processing
Data pre-processing is an important step in the data mining process that involves cleaning
and transforming raw data to make it suitable for analysis. Some common steps in data pre-
processing include:
1. Data Cleaning: This involves identifying and correcting errors or
inconsistencies in the data, such as missing values, outliers, and duplicates.
Various techniques can be used for data cleaning, such as imputation, removal,
and transformation.
2. Data Integration: This involves combining data from multiple sources to create
a unified dataset. Data integration can be challenging as it requires handling data
with different formats, structures, and semantics. Techniques such as record
linkage and data fusion can be used for data integration.
3. Data Transformation: This involves converting the data into a suitable format
for analysis. Common techniques used in data transformation include
normalization, standardization, and discretization. Normalization is used to scale
the data to a common range, while standardization is used to transform the data
to have zero mean and unit variance. Discretization is used to convert
continuous data into discrete categories.
4. Data Reduction: This involves reducing the size of the dataset while preserving
the important information. Data reduction can be achieved through techniques
such as feature selection and feature extraction. Feature selection involves
selecting a subset of relevant features from the dataset, while feature extraction
involves transforming the data into a lower-dimensional space while preserving
the important information.
5. Data Discretization: This involves dividing continuous data into discrete
categories or intervals. Discretization is often used in data mining and machine
learning algorithms that require categorical data. Discretization can be achieved
through techniques such as equal width binning, equal frequency binning, and
clustering.
6. Data Normalization: This involves scaling the data to a common range, such as
between 0 and 1 or -1 and 1. Normalization is often used to handle data with
different units and scales. Common normalization techniques include min-max
normalization, z-score normalization, and decimal scaling.
Data pre-processing plays a crucial role in ensuring the quality of data and the accuracy of
the analysis results. The specific steps involved in data pre-processing may vary depending
on the nature of the data and the analysis goals.
By performing these steps, the data mining process becomes more efficient and the results
become more accurate.
Pre-processing in Data Mining
Data pre-processing is a data mining technique which is used to transform the raw data in a
useful and efficient format.
 Steps
Involved in Data Pre-processing
1. Data Cleaning: The data can have many irrelevant and missing parts. To handle this
part, data cleaning is done. It involves handling of missing data, noisy data etc.
 Missing Data: This situation arises when some data is missing in the data. It can
be handled in various ways.
Some of them are:
o Ignore the tuples: This approach is suitable only when the
dataset we have is quite large and multiple values are missing
within a tuple.
o Fill the Missing values: There are various ways to do this
task. You can choose to fill the missing values manually, by
attribute mean or the most probable value.

 Noisy Data: Noisy data is a meaningless data that can’t be interpreted by

machines. It can be generated due to faulty data collection, data entry errors etc.
It can be handled in following ways:
o Binning Method: This method works on sorted data in order
to smooth it. The whole data is divided into segments of equal
size and then various methods are performed to complete the
task. Each segmented is handled separately. One can replace
all data in a segment by its mean or boundary values can be
used to complete the task.
o Regression: Here data can be made smooth by fitting it to a
regression function. The regression used may be linear
(having one independent variable) or multiple (having
multiple independent variables).
 o Clustering: This approach groups the similar data in a
cluster. The outliers may be undetected or it will fall outside
the clusters.
2. Data Transformation: This step is taken in order to transform the data in appropriate
forms suitable for mining process. This involves following ways:
 Normalization: It is done in order to scale the data values in a specified range (-
1.0 to 1.0 or 0.0 to 1.0)
 Attribute Selection: In this strategy, new attributes are constructed from the
given set of attributes to help the mining process.
 Discretization: This is done to replace the raw values of numeric attribute by
interval levels or conceptual levels.
 Concept Hierarchy Generation: Here attributes are converted from lower level
to higher level in hierarchy. For Example-The attribute “city” can be converted
to “country”.
3. Data Reduction: Data reduction is a crucial step in the data mining process that involves
reducing the size of the dataset while preserving the important information. This is done to
improve the efficiency of data analysis and to avoid overfitting of the model. Some
common steps involved in data reduction are:
 Feature Selection: This involves selecting a subset of relevant features from the
dataset. Feature selection is often performed to remove irrelevant or redundant
features from the dataset. It can be done using various techniques such as
correlation analysis, mutual information, and principal component analysis
(PCA).
 Feature Extraction: This involves transforming the data into a lower-
dimensional space while preserving the important information. Feature
extraction is often used when the original features are high-dimensional and
complex. It can be done using techniques such as PCA, linear discriminant
analysis (LDA), and non-negative matrix factorization (NMF).
 Sampling: This involves selecting a subset of data points from the dataset.
Sampling is often used to reduce the size of the dataset while preserving the
important information. It can be done using techniques such as random
sampling, stratified sampling, and systematic sampling.
 Clustering: This involves grouping similar data points together into clusters.
Clustering is often used to reduce the size of the dataset by replacing similar
data points with a representative centroid. It can be done using techniques such
as k-means, hierarchical clustering, and density-based clustering.
 Compression: This involves compressing the dataset while preserving the
important information. Compression is often used to reduce the size of the
dataset for storage and transmission purposes. It can be done using techniques
such as wavelet compression, JPEG compression, and gif compression.

List Of Most Popular Data Mining Tools and Applications Here we go! Here we have
compared the list of free and commercial data modelling tools.

1) Xplenty: Xplenty provides a platform that has functionalities to integrate,

process, and prepare data for analytics. Businesses will be able to make most of
the opportunities offered by big data with the help of Xplenty and that too
without investing in related personnel, hardware, and software. It is a complete
toolkit for building data pipelines. You will be able to implement complex data
preparation functions through rich expression language. It has an intuitive
interface to implement ETL, ELT, or a replication solution. You will be able to
orchestrate and schedule pipelines through a workflow engine.

Xplenty is the data integration platform for all. It offers the no-code and low-
code options.

 An API component will provide advanced customization and flexibility.

 It has functionalities to transfer and transform data between databases and data
warehouses.

 It provides support through email, chat, phone, and an online meeting.

Availability: Licensed tools.

#2) Rapid Miner

Availability: Open source

Rapid Miner is one of the best predictive analysis system developed by the
company with the same name as the Rapid Miner. It is written in JAVA
programming language. It provides an integrated environment for deep
learning, text mining, machine learning & predictive analysis.

The tool can be used for over a vast range of applications including for business
applications, commercial applications, training, education, research, application
development, machine learning.

Rapid Miner offers the server as both on premise & in public/private cloud
infrastructures. It has a client/server model as its base. Rapid Miner comes with
template-based frameworks that enable speedy delivery with reduced number
of errors (which are quite commonly expected in manual code writing process).

Rapid Miner constitutes of three modules, namely

1. Rapid Miner Studio: This module is for workflow design, prototyping,

validation etc.

2. Rapid Miner Server: To operate predictive data models created in studio

3. Rapid Miner Radoop: Executes processes directly in the Hadoop cluster to

simplify predictive analysis.

3) Orange

Availability: Open source

 Orange is a perfect software suite for machine learning & data mining. It best
aids the data visualization and is a component based software. It has been
written in Python computing language. As it is a component-based software, the
components of orange are called ‘widgets'. These widgets range from data
visualization & pre-processing to an evaluation of algorithms and predictive
modeling.

Widgets offer major functionalities like

 Showing data table and allowing to select features

 Reading the data

 Training predictors and to compare learning algorithms

 Visualizing data elements etc. Additionally, Orange brings a more interactive

and fun vibe to the dull analytic tools.

It is quite interesting to operate. Data coming to Orange gets quickly formatted

to the desired pattern and it can be easily moved where needed by simply
moving/flipping the widgets. Users are quite fascinated by Orange. Orange
allows users to make smarter decisions in short time by quickly comparing.

4)SSDT (SQL Server Data Tools)

Availability: Licensed

SSDT is a universal, declarative model that expands all the phases of database
development in the Visual Studio IDE.

BIDS was the former environment developed by Microsoft to do data analysis and
provide business intelligence solutions. Developers use SSDT transact- a design
capability of SQL, to build, maintain, debug and refactor databases.

A user can work directly with a database or can work directly with a connected
database, thus, providing on or off-premise facility. Users can use visual studio tools
for development of databases like IntelliSense, code navigation tools, and
programming support via C#, visual basic etc. SSDT provides Table Designer to
create new tables as well as edit tables in direct databases as well as connected
databases.

Deriving its base from BIDS, which was not compatible with Visual Studio2010,
the SSDT BI came into existence and it replaced BIDS.

5) SAS Data Mining

Availability: Proprietary License

Statistical Analysis System (SAS) is a product of SAS Institute developed for

analytics & data management. SAS can mine data, alter it, manage data from
 different sources and perform statistical analysis. It provides a graphical UI for non-
technical users. SAS data miner enables users to analyze big data and derives
accurate insight to make timely decisions. SAS has a distributed memory processing
architecture which is highly scalable. It is well suited for data mining, text mining
& optimization.

6) DataMelt

Availability: Open source

DataMelt, also known as DMelt is a computation and visualization environment

that provides an interactive framework to do data analysis and visualization. It is
designed mainly for engineers, scientists & students.

DMelt is written in JAVA and it is a multi-platform utility. It can run on any

operating system which is compatible with JVM (Java Virtual Machine).

It contains Scientific & mathematical libraries. Scientific libraries: To draw 2D/3D

plots. Mathematical libraries:

To generate random numbers, curve fitting, algorithms etc.

DataMelt can be used for analysis of large data volumes, data mining, and stat
analysis.

It is widely used in the analysis of financial markets, natural sciences &

engineering.

Mining Complex Data Objects: Generalization of Structured Data:

 Set-valued attribute
 Generalization of each value in the set into its corresponding higher-level
concepts
 Derivation of the general behavior of the set, such as the number of
elements in the set, the types or value ranges in the set, or the weighted
average for numerical data
 E.g., hobby = {tennis, hockey, chess, violin, nintendo_games} generalizes to
{sports, music, video_games}
 List-valued or a sequence-valued attribute
 Same as set-valued attributes except that the order of the elements in the
sequence should be observed in the generalization

 Spatial data:
 Generalize detailed geographic points into clustered regions, such as
business, residential, industrial, or agricultural areas, according to land
usage
 Require the merge of a set of geographic areas by spatial operations
 Image data:
 Extracted by aggregation and/or approximation
  Size, color, shape, texture, orientation, and relative positions and
structures of the contained objects or regions in the image
 Music data:
 Summarize its melody: based on the approximate patterns that
repeatedly occur in the segment
 Summarized its style: based on its tone, tempo, or the major musical
instruments played

Generalizing Object Data:

 Object identifier: generalize to the lowest level of class in the class/subclass
hierarchies
 Class composition hierarchies
 generalize nested structured data
 generalize only objects closely related in semantics to the current one
 Construction and mining of object cubes
 Extend the attribute-oriented induction method
 Apply a sequence of class-based generalization operators on
different attributes
 Continue until getting a small number of generalized objects that
can be summarized as a concise in high-level terms
 For efficient implementation
 Examine each attribute, generalize it to simple-valued data
 Construct a multidimensional data cube (object cube)
 Problem: it is not always desirable to generalize a set of values to
single-valued data

Spatial Data Warehousing:

 Spatial data warehouse: Integrated, subject-oriented, time-variant, and
nonvolatile spatial data repository for data analysis and decision making
 Spatial data integration: a big issue
 Structure-specific formats (raster- vs. vector-based, OO vs. relational
models, different storage and indexing, etc.)
 Vendor-specific formats (ESRI, MapInfo, Integraph, etc.)
 Spatial data cube: multidimensional spatial database
 Both dimensions and measures may contain spatial components

Dimensions and Measures in Spatial Data Warehouse:

 Dimension modeling
 nonspatial
 e.g. temperature: 25-30 degrees generalizes to hot
 spatial-to-nonspatial
 e.g. region “B.C.” generalizes to description “western provinces”
 spatial-to-spatial
 e.g. region “Burnaby” generalizes to region “Lower Mainland”

 Measures
 numerical
 distributive (e.g. count, sum)
 algebraic (e.g. average)
 holistic (e.g. median, rank)
  spatial
 collection of spatial pointers (e.g. pointers to all regions with 25-
30 degrees in July)

Mining multimedia databases:

 Description-based retrieval systems

 Build indices and perform object retrieval based on image descriptions,
such as keywords, captions, size, and time of creation
 Labor-intensive if performed manually
 Results are typically of poor quality if automated
 Content-based retrieval systems
 Support retrieval based on the image content, such as color histogram,
texture, shape, objects, and wavelet transforms

Multidimensional Analysis of Multimedia Data:

 Multimedia data cube

 Design and construction similar to that of traditional data cubes from
relational data
 Contain additional dimensions and measures for multimedia
information, such as color, texture, and shape
 The database does not store images but their descriptors
 Feature descriptor: a set of vectors for each visual characteristic
 Color vector: contains the color histogram
 MFC (Most Frequent Color) vector: five color centroids
 MFO (Most Frequent Orientation) vector: five edge orientation
centroids
 Layout descriptor: contains a color layout vector and an edge layout
vector

Mining time-series and sequence data:

 Time-series database
 Consists of sequences of values or events changing with time
 Data is recorded at regular intervals
 Characteristic time-series components
 Trend, cycle, seasonal, irregular
 Applications
 Financial: stock price, inflation
 Biomedical: blood pressure
 Meteorological: precipitation
  A time series can be illustrated as a time-series graph which describes a point moving
with the passage of time
 Categories of Time-Series Movements
 Long-term or trend movements (trend curve)
 Cyclic movements or cycle variations, e.g., business cycles
 Seasonal movements or seasonal variations
 i.e, almost identical patterns that a time series appears to follow during
corresponding months of successive years.
 Irregular or random movements
Time Sequence:
 Time-sequence query language
 Should be able to specify sophisticated queries like
Find all of the sequences that are similar to some sequence in class A, but not similar
to any sequence in class B
 Should be able to support various kinds of queries: range queries, all-pair
queries, and nearest neighbor queries
 Shape definition language
 Allows users to define and query the overall shape of time sequences
 Uses human readable series of sequence transitions or macros
 Ignores the specific details
 E.g., the pattern up, Up, UP can be used to describe increasing degrees
of rising slopes
 Macros: spike, valley, etc.
 Mining text databases:
 Text databases (document databases)
 Large collections of documents from various sources: news articles, research
papers, books, digital libraries, e-mail messages, and Web pages, library
database, etc.
 Data stored is usually semi-structured
 Traditional information retrieval techniques become inadequate for the
increasingly vast amounts of text data
 Information retrieval
 A field developed in parallel with database systems
 Information is organized into (a large number of) documents
 Information retrieval problem: locating relevant documents based on user
input, such as keywords or example documents.
Types of Text Data Mining:
 Keyword-based association analysis
 Automatic document classification
 Similarity detection
 Cluster documents by a common author
 Cluster documents containing information from a common source
 Link analysis: unusual correlation between entities
 Sequence analysis: predicting a recurring event
 Anomaly detection: find information that violates usual patterns
 Hypertext analysis
 Patterns in anchors/links
 Anchor text correlations with linked objects

Mining the World-Wide Web:

 The WWW is huge, widely distributed, global information service centre for
  Information services: news, advertisements, consumer information, financial
management, education, government, e-commerce, etc.
 Hyper-link information
 Access and usage information
 WWW provides rich sources for data mining
 Challenges
 Too huge for effective data warehousing and data mining
 Too complex and heterogeneous: no standards and structure
 Growing and changing very rapidly
 Broad diversity of user communities

Internet growth

40000000
35000000
30000000
25000000
Hosts

20000000
15000000
10000000
5000000
0
Sep-69

Sep-72

Sep-75

Sep-78

Sep-81

Sep-84

Sep-87

Sep-90

Sep-93

Sep-96

Sep-99
 Only a small portion of the information on the Web is truly relevant or useful
 99% of the Web information is useless to 99% of Web users
 How can we find high-quality Web pages on a specified topic?

Mining the World-Wide Web:

Applications of data warehousing across different industries:
(Retail industry, Manufacturing and distribution, Bank, insurance company, Government
agencies etc)

I will Send another PDF for test cases.