18CA312 Data Mining and Applications

3-0-1

Dr. Vimina E R
Course Outline

• Credit Distribution: 3-0-1

• Mark Distribution: 70: 30
• Evaluation
• Periodical Examination
• Assignment (Theory+ Lab)
• Test (Theory + Lab)
• Text Book: Jiawei Han, Micheline Kamber and Jian Pei, “Data
mining concepts and Techniques”, Third Edition, Elsevier
Publisher, 2006.
Topics

• INTRODUCTION
• Knowing your Data
• PRE-PROCESSING
• ASSOCIATION RULE MINING
• CLASSIFICATION
• CLUSTERING
• DATA MINING TRENDS
INTRODUCTION
• Data Mining – why? and Evolution
• Definition
• Kinds of Data that Can Be Mined
• Kinds of patterns that Can Be Mined
• Technologies used
• Issues
Mining
What is mined?
Why Data Mining?
• We live in a world where vast amounts of data are collected daily.
• “We are living in the information age”
• Terabytes or petabytes of data
• telecommunication networks
• medical and health industry
• billions of Web searches
• Communities and social media
• We are drowning in data, but starving for knowledge
• Powerful and versatile tools are badly needed to automatically
uncover valuable information from the tremendous amounts of data
and to transform such data into organized knowledge. This necessity
has led to the birth of data mining.
• Knowledge Discovery in Database (KDD) – Analogous name to DATA
MINING
Data Mining - Evolution

• Earlier - file processing systems

• 1960s- database and information technology
• 1970s progressed from early hierarchical and network database systems to relational
database systems
• query languages, user interfaces, query optimization, and transaction management,
online transaction processing (OLTP)
• mid-1980s onward - advanced database systems, data warehousing, and data mining
• The progress of computer hardware technology in the past three decades led to large
supplies of powerful and affordable computers, data collection equipment, and storage
media
• a great boost to the database and information industry, and it enables a huge number
of databases and information repositories to be available for transaction management,
information retrieval, and data analysis
Data Warehouse

• This is a repository of multiple heterogeneous data sources organized under a

unified schema at a single site to facilitate management decision making.
• Includes data cleaning, data integration, and online analytical processing
(OLAP)analysis techniques with functionalities such as summarization,
consolidation, and aggregation.
• Huge volumes of data have been accumulated beyond databases and data
warehouses -
• additional data analysis tools - data mining tools that provide data
classification, clustering, outlier/anomaly detection, and the characterization
of changes in data over time
• The world is data rich but information poor.
• The fast-growing, tremendous amount of
data, collected and stored in large and
numerous data repositories, has far
exceeded our human ability for
comprehension without powerful tools.
• Data collected in large data repositories
become “data tombs”
• The widening gap between data and
information calls for the systematic
development of data mining tools that can
turn data tombs into “golden nuggets” of
knowledge.
What is data Mining?
• Data mining is the process of discovering interesting patterns and
knowledge from large amounts of data.
• The data sources can include databases, data warehouses, the
Web, other information repositories, or data that are
streamed into the system dynamically.
• Data mining aims to analyze large quantities of data in order to
discover valid, novel, potentially useful, and ultimately
understandable patterns in data.
• Valid: The patterns hold in general.
• Novel: We did not know the pattern beforehand.
• Useful: We can devise actions from the patterns.
• Understandable: We can interpret and comprehend the
patterns.
Knowledge
Discovery
Data
mining is
a
step in
KDD
Kinds of Data that Can Be Mined
• Database data
• Data warehouse data
• Transactional data
• data streams: ordered/sequence data, graph or networked data, spatial
data, text data, multimedia data, and the WWW
1. Database Data:
• Database management system (DBMS) - interrelated data, known as a
database, and a set of software programs to manage and access the
data.
• Eg: A relational database is a collection of tables, each of which is
assigned a unique name. Each table consists of a set of attributes
(columns or fields) and usually stores a large set of tuples.
• What can be mined from relational databases ? we can go further by
searching for trends or data patterns - For example, data mining systems
can analyze customer data to predict the credit risk of new customers based
on their income, age, and previous credit information.
• Data mining systems may also detect deviations—that is, items with sales
that are far from those expected in comparison with the previous year. Such
deviations can then be further investigated.
2. Data Warehouse Data:
• A data warehouse is a repository of
information collected from multiple sources,
stored under a unified schema, and usually
residing at a single site. Data warehouses are
constructed via a process of data cleaning,
data integration, data transformation, data
loading, and periodic data refreshing.

• For example, rather than storing the details

of each sales transaction, the data warehouse
may store a summary of the transactions per
item type for each store or, summarized to a
higher level, for each sales region.
• A data warehouse is usually
modeled by a multidimensional
data structure, called a data
cube, in which each dimension
corresponds to an attribute or a
set of attributes in the schema,
and each cell stores the value
of some aggregate measure
such as count.
3. Transactional Data:
• In general, each record in a transactional database
captures a transaction, such as a customer’s purchase, a
flight booking, or a user’s clicks on a web page. A
transaction typically includes a unique transaction identity
number (trans ID) and a list of the items making up the
transaction, such as the items purchased in the transaction.
• Fragment of a transactional database:

• “Which items sold well together?” This is a kind of market

basket data analysis
• Fortunately, data mining on transactional data can do so by
mining frequent itemsets buying together
4. Other Kinds of Data:
• Time-related or sequence data (e.g., historical records, stock exchange data, and time-series
and biological sequence data) – Mine Stock exchange data to uncover trends that could help
you plan investment strategies
• Data streams (e.g., video surveillance and sensor data, which are continuously transmitted) -
mine computer network data streams to detect intrusions based on the anomaly of message
flows
• Spatial data (e.g., maps), engineering design data (e.g., the design of buildings, system
components, or integrated circuits) - With spatial data, we may look for patterns that describe
changes in metropolitan.
• Hypertext and multimedia data (including text, image, video, and audio data) - mining text
data, such as literature on data mining from the past ten years, we can identify the evolution
of hot topics in the field.
• Graph and networked data (e.g., social and information networks) - mining user comments
on products (which are often submitted as short text messages), we can assess customer
sentiments and understand how well a product is embraced by a market.
• and the Web (a huge, widely distributed information repository made available by the
Internet) - Web mining can help us learn about the distribution of information on the WWW
An Interesting Incident
• In August of 1854 Soho, a suburb of London, was hit
hard by a terrible outbreak of cholera.
• Dr. John Snows lived near Soho, and tried to find the
reason for the outbreak .
• He collected and mapped data on the locations (street addresses)
where cholera deaths occurred. His process was laborious and slow,
but ultimately informative. His painstaking and detailed analysis led to
the identification of the epidemic’s source—a contaminated public
water source.
John Snow
British physician
Kinds of Patterns
That Can Be Mined
1. Data characterization and discrimination
2. frequent patterns, associations, and
correlations
3. classification and regression
4. clustering analysis
5. outlier analysis
1. Data characterization and discrimination
• Data characterization is a summarization of the general characteristics or
features of a target class of data. Data discrimination, by comparison of the
target class with one or a set of comparative classes
• Data characterization: Eg: Summarize the characteristics of customers who
spend more than $5000 a year at AllElectronics. :- Result is customers who are 40
or 50 age, employed or having excellent credit ratings.
• The output of data characterization can be presented in various forms. Examples
include pie charts, bar charts, curves, multidimensional data cubes, and
multidimensional tables, including crosstabs.
• Data discrimination – Eg: A customer relationship manager at AllElectronics
may want to compare two groups of customers—those who shop for computer
products regularly and those who rarely shop for such products.
2. Frequent patterns, associations, and correlations

• Frequent patterns, : including frequent itemsets, frequent subsequences (also known as sequential
patterns), and frequent substructures.
• A frequent itemset—for example, milk and bread.
• A frequently occurring subsequence - customers, tend to purchase first a laptop, followed by a digital
camera, and then a memory card, is a (frequent) sequential pattern
• A substructure can refer to different structural forms (e.g., graphs, trees, or lattices)
• Association analysis – Sigle-dimensional
association rule

• A confidence, or certainty, of 50% means that if a customer buys a computer, there is a

50% chance that she will buy software.
• 1% support means that 1% of all the transactions under analysis show that computer and
software are purchased together.
multi-dimensio
association rul
3. classification and regression -Predictive
• Classification is the process of finding a model (or function) that describes and
distinguishes data classes or concepts.

• The model are derived based on the analysis of a set of training data.
• The derived model may be represented in various forms, such as classification rules
(i.e., IF-THEN rules), decision trees, mathematical formulae, or neural networks

IF-Then rules

Decision-tree Neural network

• The term prediction refers to both numeric prediction and class label
prediction
• classification predicts - categorical (discrete, unordered) labels,
• Regression models predicts - continuous-valued functions.
• Regression is used to predict missing or unavailable numerical data
values rather than (discrete) class labels.
• Eg: Suppose as a sales manager want to classify a large set of items in
the store, based on three kinds of responses to a sales campaign: good
response, mild response and no response.
• Instead of predicting categorical response labels for each store item,
you would like to predict the amount of revenue that each item will
generate based on the previous sales data - This is an example of
regression analysis.
[Link] Analysis
• Unlike classification and regression, which analyze class-labeled (training) data
sets, clustering analyzes data objects without consulting class labels.
• Eg: Cluster analysis can be performed on AllElectronics customer data to
identify homogeneous subpopulations of customers –
• A 2-D plot of customer data with respect to
customer locations in a city:
5. Outlier Analysis

• A data set may contain objects that do not comply with the general
behavior or model of the data. These data objects are outliers.
• Many data mining methods discard outliers as noise or exceptions.
However, in some applications (e.g., fraud detection) the rare events
can be more interesting than the more regularly occurring ones. The
analysis of outlier data is referred to as outlier analysis or anomaly
mining.
• Note :
• support, representing the percentage of transactions from a
transaction database that the given rule satisfies.
• This is taken to be the probability P(X U Y), where X U Y indicates
that a transaction contains both X and Y, that is, the union of
itemset X and Y.
• confidence, which assesses the degree of certainty of the detected
association.
• This is taken to be the conditional probability P(Y|X), that is, the

• Eg : computer → antivirus_software [support =

probability that a transaction containing X also contains Y.

2%,confidence = 60%].
Technologies Used
1. Statistics
• Data mining has an inherent connection with statistics.
• A statistical model is a set of mathematical functions that describe the behavior of the objects in a target
class in terms of random variables and their associated probability distributions.
• For example, in data mining tasks like data characterization and classification, statistical models of target
classes can be built.
• For example, we can use statistics to model noise and missing data values. Then, when mining patterns in a
large data set, the data mining process can use the model to help identify and handle noisy or missing
values in the data.
2. Machine Learning
• Machine learning investigates how computers can learn (or improve their performance) based on data.
• computer programs automatically learn to recognize complex patterns and make intelligent decisions based
on data.
• For example, a typical machine learning problem is to program a computer so that it can automatically
recognize handwritten postal codes on mail after learning from a set of examples.
• Eg Techniques : Supervised learning, Unsupervised learning, Semi-supervised learning, Active
learning
3. Database Systems and Data Warehouses
• Database systems research focuses on the creation, maintenance, and
use of databases for organizations and end-users.
• data models, query languages, query processing and optimization methods,
data storage, and indexing and accessing methods.
• A data warehouse integrates data originating from multiple sources and
various timeframes
4. Information Retrieval
• Information retrieval (IR) is the science of searching for
documents or information in documents.
• Documents can be text or multimedia, and may reside on the Web.
• The differences between traditional information retrieval and
database systems are twofold:
• Information retrieval assumes that (1) the data under search are
unstructured; and (2) the queries are formed mainly by keywords.
• The typical approaches in information retrieval adopt
probabilistic models - language model, topic model
 Applications

• Business intelligence
• search engines
• Business Intelligence:
• BI:- perform effective market analysis, compare customer feedback on
similar products, discover the strengths and weaknesses of their
competitors, retain highly valuable customers, and make smart
business decisions.
• data mining is the core of business intelligence:
• Classification and prediction techniques are the core of
predictive analytics, for which there are many applications in
analyzing markets, supplies, and sales.
• Moreover, clustering plays a central role in customer
relationship management, which groups customers based on
their similarities.
• Using characterization mining techniques, we can better
understand features of each customer group and develop
customized customer reward programs
• Web Search Engines
• A Web search engine is a specialized computer server that
searches for information on the Web
• Web search engines are essentially very large data mining
applications.
• Various data mining techniques are used in all aspects of search
engines, ranging from
• crawling , Indexing and searching (e.g., deciding how pages
should be ranked, which advertisements should be added, and
how the search results can be personalized or made “context
aware”).
 Major Issues in Data Mining
• Assignment
• mining methodology
• user interaction
• efficiency and scalability
• diversity of data types
• and data mining and society
• Outline the major research challenges of data mining in one
specific application domain, such as stream/sensor data
analysis, spatiotemporal data analysis, or bioinformatics