Course Introduction
❑Data Science and Applications
❑ This course covers data analysis, data mining, and machine learning.
While data analysis focuses on business logic and workflow, data mining
targets the discovery of hidden patterns and prediction. Centered on
techniques like classification and clustering, this course explains how to
extract knowledge from massive data, a process known as Knowledge
Discovery in Databases (KDD).
1
1
You need to learn
❑ Grasp the fundamental concepts and the complete workflow of data
science.
❑ Proficiently use tools like Python for data analysis and processing.
❑ Demonstrate the practical ability to solve real-world data problems, for
instance, by showcasing a complete data analysis project.
2
Course Introduction
❑ Lecturers
❑ Qian Weizhong, associate professor
❑ Zhong Ting, professor
❑ TA
❑ Nisa
❑ Contact
❑ Wechat group
3
3
Course Composition
❑ Part 1: introduction, getting to know your data
❑ Part 2: data processing
❑ Part 3: clustering
❑ Part 4: Regression Analysis
❑ Part 5: classifying
❑ Part 6: Frontiers in Data Science
4
4
Course Introduction
❑Evaluation and Grading
❑ Your course grade will be based on assignments, classroom interaction,
attendance, a project report and a final-term exam.
❑ The weights are given to these components as following:
❑ Usual performance(50%):
❑ Assignments, classroom interaction, attendance
❑ Project Report & Presentation
❑ Final-term exam (50%)
5
5
Reference
❑ Text Book
❑ Data Mining, Jiawei Han, Micheline Kamber
and Jian Pei, Mechanical industry press(2012)
❑ Reference Book
❑Tamhane, Ajit C., and Dorothy D. Dunlop.
Statistics and Data Analysis: From Elementary to
Intermediate. Prentice Hall, 1999.
❑ Couresa
1）Machine Learning（Andrew Ng）
2）Data Mining（Stanford）
3）Statistical Thinking and Data Analysis
（MIT）
6
6
What’s big data?
7
7
Characteristics of Big Data
8
Data Science Process
9
9
Chapter 1. Introduction
❑ 1 What Is Data Mining?
❑ 2 Data Mining: An Essential Step in Knowledge Discovery
❑ 3 Diversity of Data Types for Data Mining
❑ 4 Mining Various Kinds of Knowledge
❑ 5 Data Mining: Confluence of Multiple Disciplines
❑ 6 Data Mining and Applications
❑ 7 Data Mining and Society
❑ Summary
10
1 What Is Data Mining?
❑ We live in a world where vast amounts of data are generated constantly and rapidly
❑ Data mining is the process of discovering interesting patterns, models and other
kinds of knowledge in large data sets
❑ “Data mining”: a misnomer? It should be “knowledge mining from data”
❑ Other terms: Knowledge mining from data, KDD (Knowledge Discovery from Data),
pattern discovery, knowledge extraction, data analytics, information harvesting
❑ Data mining is a young, dynamic, and promising field
❑ Example: Data mining turns a large collection of data into knowledge
❑ Google’s Flu Trends found a close relationship between the number of people
who search for flu-related info. and the number of people who have flu symptoms
❑ It can estimate flu activity up to two weeks faster than traditional systems
11
2 Data Mining: An Essential Step in Knowledge Discovery
❑ Knowledge Discovery Process
❑ Data preparation
❑ Data cleaning
❑ Data integration
❑ Data transformation
❑ Data selection
❑ Data mining
❑ Pattern/model evaluation
❑ Knowledge presentation
12
3 Diversity of Data Types for Data Mining (I)
❑ Structured vs. unstructured data
❑Structured: uniform, record- or table-like structures, defined by data dictionaries, with a
fixed set of attributes, each with a fixed set of value ranges and semantic meaning
❑ Ex. Data stored in relational databases, data cubes, data matrices, and many data
warehouses
❑ Semi-structured: allow a data object to contain a set value, a small set of heterogeneous
typed values, or nested structures, or to allow the structure of objects or sub-objects to
be defined flexibly and dynamically
❑ Data having certain structures with clearly defined semantic meaning, such as
transactional data set, sequence data set (e.g., time-series data, gene or protein data, or
Weblog data)
❑ Graph or network data: A more sophisticated type of semi-structured data set
❑ Unstructured data: text data and multimedia (e.g., audio, image, video) data
❑ The real-world data can often be a mixture of structured, semi-structured and unstructured
13
data
Structured data example
❑ Typical Example: A record in a relational database (like MySQL, Oracle).
Characteristic: Strict rows and columns, like a filled-out Excel spreadsheet.
❑ Data Example: A book record in a library system
book_id (Integer) title (String) author (String) price (Float) publish_date (Date)
1001 Introduction to AI Jiawei Han 89.00 2022-01-15
❑ Analysis:
❑ Every field has a strict data type (e.g., price must be a number).
❑ If you try to enter "This book is expensive" in the "price" column, the
database will throw an error.
❑ It's ideal for fast computation (e.g., calculating the average price of all
books).
14
Semi-structured data example
❑ Typical Example: JSON, XML, HTML.
Characteristic: Uses tags to separate data, has a flexible structure, and can
be nested, like a package with many labels.
❑ Data Example: A book's details on an e-commerce website (JSON format)
❑ Analysis:
❑ Flexibility: You can see
that author contains another nested
structure, and tags is a list.
❑ Not Mandatory: If the next book doesn't
have a reviews field, the data is still valid.
This flexibility is a key feature.
15
Unstructured data example
❑ Typical Example: Text documents, emails, images, videos.
Characteristic: No predefined data model or format. It's raw information.
❑ Data Example: A user's book review (Plain Text)
❑ Analysis:
❑ No Format: This is just a block of text. A computer can't immediately tell you who the author is or
what the rating is without complex processing.
❑ Rich in Information: It contains opinions, feelings, and specific details, but this information is "locked"
inside the text.
❑ Requires Special Tools: To analyze this, you'd need technologies like Natural Language Processing
(NLP) to extract meaning, such as identifying the book title or the user's sentiment.
16
Diversity of Data Types for Data Mining (II)
❑ Data associated with different applications
Different applications: different data sets and require different data analysis methods
❑
❑ Sequence data: Biological sequences vs. shopping transaction sequences
❑ Time-series: ordered set of numerical values with equal time interval
❑ Spatial, temporal and spatiotemporal data
❑ Graph and network data: Social networks, computer communication networks, biological
networks, and information networks may carry rather different semantics
❑ Stored vs. streaming data
❑ Stored data: Finite, stored in various kinds of large data repositories
❑ Streaming data (e.g., video or remote sensing): Dynamic, constantly coming, infinite, real-
time response―posing challenges on effective data mining
17
Different mining methods for mining
❑ Pattern mining
❑ On the same data set, finding different kinds of patterns: require different mining methods
❑ Scenario: The Same Dataset – "Supermarket Transaction Records"
❑ Imagine you have a dataset containing 1 million customer receipts. Each
record lists all the items a customer purchased in a single transaction (e.g.,
milk, bread, beer, diapers).
18
Different mining methods for mining(2)
❑ Goal 1: Finding "Association Rules"
❑ Business Question: If a customer buys item A, do they usually buy item B
as well? (The classic "Beer and Diapers" case)
❑ Required Pattern: Frequent Itemsets.
❑ Required Mining Method: Association Rule Mining.
❑ Result Example: {Diapers} -> {Beer} (If diapers are bought, there is an 80%
probability that beer is also bought).
19
Different mining methods for mining(3)
❑ Goal 2: Finding "Customer Segmentation"
❑ Business Question: Into what typical groups can my customers be divided?
(e.g., "Family-oriented“, "Young Professionals")
❑ Required Pattern: Clusters.
❑ Required Mining Method: Cluster Analysis.
❑ Result Example: Customers are divided into "Cluster 1 (mainly buys baby
products and fresh produce)" and "Cluster 2 (mainly buys fast food and
alcohol)".
20
Different mining methods for mining(4)
❑ Goal 3: Finding "Fraud Detection"
❑ Business Question: Are there any transactions that look highly unusual,
possibly indicating credit card fraud or data entry errors?
❑ Required Pattern: Outliers.
❑ Required Mining Method: Anomaly Detection.
❑ Result Example: Discovering a transaction that purchased "100 cases of
red wine + 500 lighters," which statistically deviates from normal patterns
and is flagged as an anomaly.
21
4 Mining Various Kinds of Knowledge
❑ Multidimensional Data Summarization
❑ Mining Frequent Patterns, Associations, and Correlations
❑ Classification and Regression for Predictive Analysis
❑ Cluster Analysis
❑ Deep Learning
❑ Outlier Analysis
❑ Are All Mining Results Interesting?
22
Multidimensional Data Summarization
❑ Information integration and data warehouse
construction
❑ Data cleaning, transformation, integration, and
multidimensional data model
❑ Data cube technology
❑ Scalable methods for computing (i.e.,
materializing) multidimensional aggregates
❑ OLAP (online analytical processing)
❑ Multidimensional concept description:
Characterization and discrimination
❑ Generalize, summarize, and contrast data
characteristics, e.g., dry vs. wet region
23
Database and data warehouse
❑ Database
❑ Stores current business data
❑ Data warehouses
❑ are constructed through data cleaning, data transformation, data
integration, data loading, and periodic data refreshing.
❑ Typical Scenarios
❑ Database: Real-time business systems such as bank transfers, e-
commerce orders, and user registration.
❑ Data Warehouse: Decision support scenarios such as sales trend analysis,
user profiling, and annual business reports.
24
Pattern Discovery: Mining Frequent Patterns,
Associations, and Correlations
❑ Frequent patterns (or frequent itemsets)
❑ What items are frequently purchased together in your Walmart?
❑ Association and Correlation Analysis
❑ A typical association rule
❑ Diaper → Beer [0.5%, 75%] (support, confidence)
25
Pattern Discovery: Mining Frequent Patterns,
Associations, and Correlations
❑ Are strongly associated items also strongly correlated?
❑ Association (Broad): Defined by the existence of any
relationship between two variables. This encompasses linear, non-
linear, or co-occurrence patterns.
❑ Correlation (Narrow): Typically refers specifically to a linear
relationship, where a change in one variable corresponds to a
proportional change in the other.
26
Pattern Discovery: Mining Frequent Patterns,
Associations, and Correlations
❑ How to mine such patterns and rules efficiently in large datasets?
❑ When the dataset is massive, exhaustively checking all
combinations is incredibly slow.
❑ need smart algorithms like Apriori or FP-Growth.
27
How to use such patterns
❑ How to use such patterns for classification, clustering, and other
applications?
❑ For Classification:
❑ Example: Discovering that "invoice" and "agency" frequently appear together in spam emails.
❑ Application: Any new email containing both words is automatically classified as spam.
❑ For Clustering:
❑ Example: Discovering that "deep learning" and "Python" often co-occur.
❑ Application: Automatically grouping articles containing these terms into one cluster (e.g., Computer
Science) without manual reading.
❑ For Recommendation:
❑ Example: Discovering that users who watch "cat" videos often watch "cooking" videos too.
❑ Application: When you watch a cat video, the system automatically recommends cooking videos to
you.
28
Classification and Regression for Predictive Analysis
❑ Classification and label prediction
❑ Construct models (functions) based on some training examples
❑ Describe and distinguish classes or concepts for future prediction
❑ Ex. 1. Classify countries based on (climate)
❑ Ex. 2. Classify cars based on (gas mileage)
❑ Predict some unknown class labels
❑ Typical methods
❑ Decision trees, naïve Bayesian classification, support vector machines, neural
networks, rule-based classification, pattern-based classification, logistic
regression, …
❑ Typical applications:
❑ Credit card fraud detection, direct marketing, classifying stars, diseases, web-
pages, …
29
Cluster Analysis
❑ Unsupervised learning (i.e., Class label is
unknown)
❑ Group data to form new categories (i.e.,
clusters), e.g., cluster houses to find
distribution patterns
❑ Principle: Maximizing intra-class similarity
& minimizing interclass similarity
❑ Many methods and applications
30
Deep Learning
❑ Deep learning: A fast expanding dynamic frontier in machine learning
❑ Deep learning has developed various neural network architectures
❑ Feed-forward neural networks
❑ Convolutional neural networks
❑ Recurrent neural networks
❑ Graph neural networks
❑ Transformer
❑ Deep learning has broad applications in computer vision, natural language
processing, machine translation, social network analysis, and so on
❑ Deep learning has been reshaping a variety of data mining tasks
❑ Ex. classification, clustering, outlier detection, and reinforcement learning
31
Outlier Analysis
❑ Outlier analysis
❑ Outlier: A data object that does not comply with the
general behavior of the data
❑ Noise or exception?―One person’s garbage could be
another person’s treasure
❑ Methods: by product of clustering or regression analysis, …
❑ Useful in fraud detection, rare events analysis
32
Other Data Mining Functions: Time and Ordering:
Sequential Pattern, Trend and Evolution Analysis
❑Sequence, trend and evolution analysis
❑ Trend, time-series, and deviation analysis
❑ Analyzing the long-term direction (trend) or short-term
fluctuations (time-series) of data over time.
❑ e.g., regression and value prediction
❑ Sequential pattern mining
❑ Finding event patterns that occur in a specific order within the
data.
❑ e.g., buy digital camera, then buy large memory cards
❑ Periodicity analysis
❑ Finding recurring cyclic patterns in data.
❑ e.g., sales increase every "Christmas" or "weekend"
33
Other Data Mining Functions: Time and Ordering:
Sequential Pattern, Trend and Evolution Analysis
❑ Motifs and biological sequence analysis
❑ Approximate and consecutive motifs
❑ Similarity-based analysis
❑ Finding similar patterns in data.
❑ e.g., comparing whether the historical trends of two
stocks are similar.
❑ Mining data streams
❑ Ordered, time-varying, potentially infinite, data streams
34
Other Data Mining Functions: Time and Ordering:
Sequential Pattern, Trend and Evolution Analysis
❑ Mining data streams
❑ Ordered, time-varying, potentially infinite, data streams
❑ Analysis: Unlike traditional static databases (like a saved Excel sheet),
data streams are dynamic, real-time, and theoretically endless.
❑ Examples:
❑ Real-time trading data from the stock market.
❑ Real-time monitoring data from network sensors.
❑ Real-time clickstreams on websites.
❑ Challenge: Since the data is "infinite" and "fast," you cannot store all of
it for later analysis. You must process and mine it in real-time the
moment it flows by.
35
Other Data Mining Functions: Structure and
Network Analysis
❑ Graph mining
❑ Finding frequent subgraphs (e.g., chemical compounds), trees (XML),
substructures (web fragments)
❑ Information network analysis
❑ Social networks: actors (objects, nodes) and relationships (edges)
❑ e.g., author networks in CS, terrorist networks
❑ Multiple heterogeneous networks
❑ A person could be multiple information networks: friends, family, classmates, …
❑ Links carry a lot of semantic information: Link mining
❑ e.g., in a social network, whether you are "colleagues" or "relatives" with
someone—the nature of this connection contains rich information.
36
Other Data Mining Functions: Structure and
Network Analysis
❑ Web mining
❑ Web is a big information network: from PageRank to
Google
❑ Analysis of Web information networks
❑ Web community discovery, opinion mining, usage
mining, …
❑ Web mining uses data mining techniques to discover
patterns from web content, structure, and usage logs.
Google's PageRank is a classic example that utilizes the
link structure between web pages to evaluate their
importance.
37
Evaluation of Knowledge
❑ Are all mined knowledge interesting?
❑ One can mine tremendous amount of “patterns”
❑ e.g., "People who buy toothpaste usually buy toothbrushes as well.
❑ Some may fit only certain dimension space (time, location, …)
❑ e.g., the pattern of "selling ice cream in summer" is only valid in a specific
season (time dimension).
❑ Some may not be representative, may be transient, …
❑ Some patterns could simply be random noise or the result of chance due to
having too few samples.
❑ a sudden spike in product sales caused by a hot internet trend
38
Evaluation of Knowledge
❑ Evaluation of mined knowledge → directly mining only interesting knowledge?
❑ Descriptive vs. predictive
❑ Summarize what happened in the past (descriptive), or to predict what will happen (predictive)?
❑ Coverage&Accuracy
❑ How much data can this pattern cover
❑ e.g., does this rule apply to 80% of the customers?)
❑ Typicality vs. novelty
❑ Is this pattern common knowledge (typical), or does it discover a surprising new rule (novel)?
❑ Timeliness
❑ Is this knowledge still useful now
❑ e.g., "Nokia phones are the best-selling" is outdated today.
39 ❑ …
5 Data Mining: Confluence of Multiple
Disciplines
40
Why Confluence of Multiple Disciplines?
❑ Tremendous amount of data
❑ Algorithms must be scalable to handle big data
❑ High-dimensionality of data
❑ Micro-array may have tens of thousands of dimensions
❑ High complexity of data
❑ Data streams and sensor data
❑ Time-series data, temporal data, sequence data
❑ Structure data, graphs, social and information networks
❑ Spatial, multimedia, text and Web data
❑ Software programs, scientific simulations
❑ New and sophisticated applications
41
6 Data Mining and Applications
❑ Web page analysis: classification, clustering, ranking
❑ Collaborative analysis & recommender systems
❑ Basket data analysis to targeted marketing
❑ Biological and medical data analysis
❑ Data mining and software engineering
❑ Data mining and text analysis
❑ Data mining and social and information network analysis
❑ Built-in (invisible data mining) functions in Google, Microsoft, LinkedIn, Meta, …
❑ Major dedicated data mining systems/tools
❑ SAS, MS SQL-Server Analysis Manager, Oracle Data Mining Tools)
42
Typical algorithms for applications
• #1: C4.5 (61 votes)
• #2: K-Means (60 votes)
• #3: SVM (58 votes)
• #4: Apriori (52 votes)
• #5: EM (48 votes)
• #6: PageRank (46 votes)
• #7: AdaBoost (45 votes)
• #7: kNN (45 votes)
• #7: Naive Bayes (45 votes)
• #10: CART (34 votes)
43
43
Requirement for different applications
❑ Data requirements:
❑ The data necessary as inputs to the analysis are specified based upon the
requirements of those directing application.
❑ The general type of entity upon which the data will be collected is referred to as an
experimental unit.
❑ Specific variables regarding of different system (e.g., age and income) may be
specified and obtained.
❑ Data may be numerical or categorical (i.e., a text label for numbers).
44
44
7 Data Mining and Society
❑ Data mining technology may benefit society
❑ Ex.: Help scientific discovery, business management, economy recovery, and
security protection (e.g., the real-time discovery of intruders and cyberattacks)
❑ Need to guard against the misuse of data mining
❑ Data mining also poses the risk of unintentionally disclosing some confidential
business or government information and disclosing an individual’s personal
information
❑ Studies on data security in data mining and privacy-preserving data publishing and
data mining are important, ongoing research theme
❑ The philosophy is to observe data sensitivity and preserve data security and
people’s privacy while performing successful data mining
45
Summary
❑ Data mining: Discovering interesting patterns and knowledge from massive
amounts of data
❑ A KDD process includes data cleaning, data integration, data selection,
transformation, data mining, pattern evaluation, and knowledge presentation
❑ Different data mining method on a wide variety of data
❑ Data mining functionalities: summarization, pattern discovery, classification,
clustering, deep learning, outlier analysis, trend and outlier analysis, …
❑ Data mining is a confluence of multiple disciplines
❑ Data mining has broad applications
❑ Promote secure data mining to benefit society
46