Data Mining
Section 0
Vu Ngoc Tu
Requirements
Quiz (at least 1 quiz): 20%
Final exam: 50%
Assignment: 30%
Prerequisites
● Basic Databases.
● Algorithms: Dynamic programming, basic data structures.
● Basic Math: Statistics, Linear algebra, Calculus,...
● Programming: Python/ R.
Data engineer, Data scientist, Data Analyst
Data engineer, Data scientist, Data Analyst
Data engineer, Data scientist, Data Analyst
Data engineer, Data scientist, Data Analyst
Course objectives
This course aims to introduce the knowledge discovery concepts, process,
technologies, and applications of data mining.
It is also to discuss data preprocessing issues, data mining tasks, algorithms and
tools that can be used to support data analysts and data mining application
developers.
What this course is about
This course about
- analyzed the data
- some recommendation algorithms
Data Mining ≈ Predictive Analytics ≈ Data Science ≈ Machine Learning ≈
Data-Centric AI
What this course is NOT about
This course will NOT FOCUS on:
- Tool, frameworks using
- Application building.
- How data is stores? - Systems.
- How data is manages? - Databases.
We let you learn a about data engineering and databases through your project, but
this will not our goals.
Learning outcome
● Understand the steps in the overall knowledge discovery process
● Describe basic concepts, technologies, and applications of data mining
● Explain popular data mining tasks including regression, classification,
clustering, and association rules mining
● Identify data related issues in the data preprocessing phase for data mining
tasks
● Understand how to use data mining to make better business decisions
● Use data mining algorithms and tools for data mining application development
● Have sufficient knowledge to do research on the data mining area
What is KDD?
Knowledge Discovery in Databases is the nontrivial process of identifying valid,
novel, potentially useful, and ultimately understandable patterns in data.
Here data is a set of facts (e.g., cases in a database) and pattern is an expression
in some language describing a subset of the data or a model applicable to that subset.
KDD Process is the process of using the database along with any required selection,
preprocessing, subsampling, and transformations of it; to apply data mining methods
(algorithms) to enumerate patterns from it; and to evaluate the products of data mnining
to identify the subset of the enumerated patterns deemed "knowledge".
What is data mining?
Data mining is the process of automatically discovering useful information in large
data repositories. [1]
Data Mining is a step in the KDD process consisting of applying data analysis and
discovery algorithms that, under acceptable computational efficiency limitations,
produce a particular enumeration of patterns over the data. [2]
Data mining is the analysis of (often large) observational data sets to find
unsuspected relationships and to summarize the data in novel ways that are both
understandable and useful to the data owner. [3]
[1] Introduction to Data Mining Pang Ning Tan
[2] Fayyad, U. M., Piatetsky-Shapiro, G., & Smyth, P. (1996, August). Knowledge Discovery and Data Mining: Towards a Unifying Framework. In KDD (Vol.
96, pp. 82-88).
[3]
Why Data Mining need so much stuffs?
● Tremendous amount of data
● Algorithms must be highly scalable to handle such as tera-bytes of data
● High-dimensionality of data
● Micro-array may have tens of thousands of dimensions
● High complexity of data
● Data streams and sensor data
● Time-series data, temporal data, sequence data
● Structure data, graphs, social networks and multi-linked data
● Heterogeneous databases and legacy databases
● Spatial, spatiotemporal, multimedia, text and Web data
● Software programs, scientific simulations
● New and sophisticated applications
KDD process
KDD process
How to build a data mining framework?
● Developing an understanding of the application domain and the relevant prior
knowledge, and identifying the goal of the KDD process from the customer's
viewpoint.
● Create the target dataset
● Data cleaning, preprocessing
● Data reduction and projection.
● Matching tile goals of the KDD process (step 1) to particular data mining method.
● Choosing the data mining algorithm(s).
● Data mining
● Presentation of the mining results
● Consolidating discovered knowledge.
KDD process
Origin of data mining.
● Overlaps with machine learning, statistics, artificial intelligence, databases,
visualization but more stress on
● scalability of number of features and instances
● algorithms and algorithms and architectures whereas foundations of methods and
formulations provided by statistics and machine learning
● automation for handling large automation for handling large, heterogeneous data
Kind of data
● Database-oriented data sets and applications
● Relational database, data warehouse, transaction database
● Advanced data sets and advanced applications
● Data streams and sensor data
● Time-series data, temporal data, sequence data (incl. bio-sequences)
● Structure data, graphs, social networks and multi-linked data
● Object-relational databases
● Heterogeneous databases and legacy databases
● Spatial data and spatiotemporal data
● Multimedia database
● Text databases
● The World-Wide Web
Major Issues in Data Mining
● Mining Methodology
● Mining various and new kinds of knowledge
● Mining knowledge in multi-dimensional space
● Data mining: An interdisciplinary effort
● Boosting the power of discovery in a networked environment
● Handling noise, uncertainty, and incompleteness of data
● Pattern evaluation and pattern- or constraint-guided mining
● User Interaction
● Interactive mining
● Incorporation of background knowledge
● Presentation and visualization of data mining results
Major Issues in Data Mining
● Efficiency and Scalability
● Efficiency and scalability of data mining algorithms
● Parallel, distributed, stream, and incremental mining methods
● Diversity of data types
● Handling complex types of data
● Mining dynamic, networked, and global data repositories
● Data mining and society
● Social impacts of data mining
● Privacy-preserving data mining
● Invisible data mining
Conferences and Journals on Data Mining
● KDD Conferences
● ACM SIGKDD Int. Conf. on Knowledge Discovery in Databases and Data Mining (KDD)
● SIAM Data Mining Conf. (SDM)
● (IEEE) Int. Conf. on Data Mining (ICDM)
● European Conf. on Machine Learning and Principles and practices of Knowledge Discovery and Data Mining
(ECML-PKDD)
● Pacific-Asia Conf. on Knowledge Discovery and Data Mining (PAKDD)
● Int. Conf. on Web Search and Data Mining (WSDM)
● WWW (The Web Conference)
● Other related conferences
● DB conferences: ACM SIGMOD, VLDB, ICDE, EDBT, ICDT, …
● Web and IR conferences: WWW, SIGIR, WSDM
● ML conferences: ICML, NIPS
● CV conferences: CVPR,
● Journals
● Data Mining and Knowledge Discovery (DAMI or DMKD)
● IEEE Trans. On Knowledge and Data Eng. (TKDE)
● KDD Explorations
● ACM Trans. on KDD
Applications
List some applications of data minings.
Applications
Applications
Applications
Applications