DATA MINING NOTES (MEDIUM FORMAT)
UNIT–1: BASIC STRUCTURE
1. Introduction to Data Mining
- Extracting useful patterns and insights from data.
- Applications: marketing, fraud detection, healthcare.
- CRISP–DM: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation,
Deployment.
- Data Mining Project Life Cycle: data collection → cleaning → modeling → deployment.
2. Working with Modeler
- Modeler uses nodes and streams.
- Streams: connected workflow of nodes.
- Nodes: source, processing, modeling, evaluation, output.
- Managing nodes, encapsulating nodes, generating nodes, running streams.
3. Collecting Initial Data
- Rectangular data structure.
- Unit analysis (customer/transaction).
- Field storage types: numeric, string, date.
- Measurement levels: nominal, ordinal, continuous.
- Import/export data using sources dialog.
4. Understanding Your Data
- Data audit: missing values, outliers, incorrect formats.
- Visualizing data: histogram, boxplot, scatterplot.
- Invalid values: blanks, nulls, inconsistent entries.
- Handling missing values: imputation, removal, replacement.
5. Data Reporting Blanks
- Blanks must be treated prior to modeling.
6. Setting Unit of Analysis
- Choosing correct granularity.
- Aggregating and filtering records.
7. Integrating Data
- Appending records, merging fields.
- Sampling data and caching intermediate results.
UNIT–2: BASIC COMPUTING
1. Deriving & Reclassifying Fields
- Using CLEM (Control Language for Expression Manipulation).
- Deriving new fields, converting blanks, grouping categories.
2. Looking for Relationships
- Cross-tabulation, correlation, matrix output.
- Histograms, plots, statistical outputs.
3. Using Functions
- Date/time functions, conversion functions, string functions.
- Statistical and missing value functions.
4. Data Transformations
- Filling missing values, binning, normalization, standardization.
5. Sequence Data
- Cross-record functions, counters, trends.
- Working with geospatial and time data.
6. Sampling Records
- Simple random sampling, complex sampling.
- Partitioning, balancing dataset.
7. Improving Efficiency
- SQL pushback and previewing SQL.
- Detecting atypical values.
- Handling missing values.
- Conditional execution and looping.
UNIT–3: ADVANCED THEORIES – AUTOMATED DATA MINING
1. Basics of Using a Modeler
- Adding nodes, creating streams, reading files.
2. Data Exploration
- Summary statistics, distributions, outliers.
3. Automated Data Preparation
- Automatic handling of missing values, scaling, encoding.
4. Data Partitioning
- Train-test split, validation sets.
5. Predictor Selection
- Automated feature selection using statistical and ML-based methods.
6. Automated Models
- Classification: logistic regression, trees, SVM, Naive Bayes.
- Regression: linear regression, tree regression, neural networks.
7. Model Evaluation
- Accuracy, precision, recall, F1, ROC, RMSE, MAE, R².
8. Deploying Models
- Exporting models, API deployment, batch scoring, real-time scoring.