Module 2

Descriptive Analytics I -Nature of Data, Big Data, and Statistical Modeling

Lecture Notes: 3.2 The Nature of Data in Analytics

1. Importance of Data

• Data = raw material for Business Intelligence (BI), Data Science, and
Analytics.

• Produces: Information → Insight → Knowledge.

• Past: relied on expert knowledge with little/no data.

• Present: data is essential; considered one of the most valuable assets of any
organization.

• Value: helps understand customers, competitors, and business processes.

2. Characteristics of Data

• Size: Small or very large (Big Data).

• Form: Structured (tables, databases) or Unstructured (text, images, videos).

• Flow: Continuous streams (real-time) or batch uploads.

• Collection Methods:

o Traditional: surveys, manual entry.

o Modern: sensors, RFID, Internet-based systems.

3. Analytics Continuum

• Data → Analytics → Actionable Information

• Data sources:

o Business processes (ERP, CRM, SCM).

o Internet/social media (Twitter, Facebook, YouTube, etc.).

o Machines/IoT.

• Requires: storage, protection, validation, and analytics platforms.

4. Making Data Analytics-Ready

• Data must comply with usability and quality metrics.

• Requirements:

o Relevant to the problem.

o Sufficient in quality and quantity.

o Structured properly (normalized values, key variables).

o Organization-wide agreed definitions (e.g., customer definition).

Example:

• Predictive analytics → requires flat file format with target variable.

• Neural networks → input variables must be numeric.

• Decision trees → can handle both numeric and categorical.

5. Risks of Poor Data Preparation

• “Wrong answer for the right problem.”

• Leads to inaccurate and untimely decisions.

6. Metrics of Data Readiness

1. Data Source Reliability

o Is the source original and trustworthy?

o Each transfer risks misrepresentation.

2. Data Content Accuracy

o Does the data match the intended reality?

o Example: Customer phone number in database = same as what customer

provided.

3. Data Accessibility

o Can data be accessed easily?

o Challenge: Multiple sources (databases, Hadoop, data lakes).

4. Data Security & Privacy

o Restricted access to authorized users only.

o Legal compliance (e.g., HIPAA in healthcare).

5. Data Richness (Comprehensiveness)

o Enough variables to portray subject matter fully.

o Example: Predictive model requires complete data for better results.

6. Data Consistency

o Properly merged and combined.

o Example: Avoid mixing demographic and medical test records of different

patients.

7. Data Currency / Timeliness

o Data must be up-to-date and recorded close to the event time.

o Prevents errors from delays.

 8. Data Granularity

o Level of detail must be appropriate.

o Example: Lab test results recorded with correct decimal precision.

9. Data Validity

o Data values must match defined rules/ranges.

o Example: Gender values = Male, Female, Other.

10. Data Relevancy

o Only include relevant variables.

o Avoid irrelevant data → can mislead algorithms.

7. Key Takeaways

• Data is the foundation of analytics.

• Data must be reliable, accurate, secure, comprehensive, timely, granular,

valid, and relevant.

• Different analytics projects require different types and structures of data.

• Good data preparation ensures correct, timely, and valuable insights.

Lecture Notes: 3.3 A Simple Taxonomy of Data

1. What is Data?

• Data (plural of datum): Collection of facts from experiments, observations,

transactions, or experiences.

• Can be: numbers, letters, words, images, audio, video, etc.

• Represents measurements of variables (characteristics of subjects/events).

• Hierarchy of abstraction:

o Data → Information → Knowledge (data is the lowest level).

2. High-Level Classification of Data

1. Structured Data

o Well-organized; suitable for computer processing.

o Used in data mining algorithms.

o Can be:

▪ Categorical → Nominal / Ordinal

▪ Numeric → Interval / Ratio

2. Unstructured or Semi-structured Data

o Examples: text, images, videos, audio, web logs, XML, JSON.

o Covered in text/web mining (later chapters).

3. Structured Data Types

A. Categorical (Discrete) Data

• Represents labels or groups (finite values, no continuum).

• Examples: race, sex, education level, age group.

 • Even if numeric codes are used, they are just symbols, not quantities.

i. Nominal Data

• Pure labels, no order implied.

• Examples:

o Marital status → single, married, divorced.

o Colors → brown, blue, green.

o Yes/No; True/False.

• Types:

o Binomial (2 values: Yes/No).

o Multinomial (3+ values: Single/Married/Divorced).

ii. Ordinal Data

• Labels with a rank order.

• Examples:

o Credit score → low, medium, high.

o Age group → child, young, middle-aged, senior.

o Education → high school, college, graduate.

• Some predictive models (e.g., ordinal logistic regression) use rank info.

B. Numeric (Continuous) Data

• Represents measurable quantities.

• Can be integer or real (fractions).

• Examples: age, number of children, income, distance, temperature.

• Infinite possible values between points (continuous scale).

i. Interval Data

• Numeric scale without absolute zero.

• Example: Temperature in Celsius or Fahrenheit.

• Zero point is arbitrary (not absence of quantity).

ii. Ratio Data

 • Numeric scale with a true zero.

• Examples: mass, length, time, energy, Kelvin temperature.

• Ratios are meaningful (e.g., 10kg is twice 5kg).

4. Other Data Types

• Textual, spatial, imagery, video, audio/voice.

• Must be converted into categorical/numeric for analytics.

• Example: Image → pixel values; Audio → frequency features.

5. Static vs. Dynamic Data

• Static data: Does not change (e.g., census data).

• Dynamic (temporal/time series): Changes over time (e.g., stock prices, sensor
readings).

6. Data Compatibility with Algorithms

• Different analytics methods require specific data formats:

1. Numeric-only methods (e.g., Neural Networks, Logistic Regression, SVM):

o All variables must be numeric.

o Categorical variables → converted into numeric (using 1-of-N dummy

variables).

o Caution: increases number of variables (curse of dimensionality).

2. Categorical-only methods (e.g., ID3 Decision Trees, Rough Sets):

o Require categorical inputs.

o Numeric variables must be discretized (binned into categories).

3. Modern tools

o Handle both numeric & categorical.

o Perform internal conversions automatically.

7. Key Takeaways

• Data can be structured or unstructured.

• Structured data → Categorical (Nominal, Ordinal) & Numeric (Interval, Ratio).

• Correct representation of data is crucial for analytics models.

• Many algorithms require transformations before processing.

• Business analytics tools are evolving to simplify data transformation &

representation.

Lecture Notes: The Art and Science of Data Preprocessing

1. Introduction to Data Preprocessing

• Definition: Data preprocessing is the process of converting raw, real-world data

into a clean, usable format for analytics.

• Why needed?

o Raw data is often dirty, incomplete, misaligned, complex, inconsistent.

o Preprocessing ensures data quality → improves the accuracy of analytics

models.

• Time factor:

o Data preprocessing is the most time-consuming phase (more time than

actual model building).

• Analogy: Just as chefs prepare raw ingredients before cooking, analysts must
prepare data before analysis.
2. Main Phases of Data Preprocessing (Figure 3.3)

(A) Data Consolidation

• Activities:

1. Collect relevant data from multiple sources (databases, OLTP, web, social
media, legacy systems, etc.)

2. Select useful records & attributes; remove irrelevant ones.

3. Integrate/merge multiple datasets (handle synonyms, homonyms).

• Modern term: Data Blending.

• Types of blending:

o Source blending (SQL, NoSQL, cloud, APIs)

 o Type blending (structured + unstructured)

o Tool blending (Python, R, KNIME, SAP, etc.)

• Example: Combining student data from admissions, financial aid, and academic
databases.

(B) Data Cleaning (a.k.a. Data Scrubbing)

• Why? Raw data is often dirty or inconsistent.

• Tasks:

o Handle missing values:

▪ Imputation (mean, median, mode, regression-based filling).

▪ Use a constant (e.g., "N/A") or drop the record.

▪ Example: Income field left blank by high earners.

o Handle noisy values (outliers):

▪ Detect with statistics (std dev, clustering).

▪ Treat by smoothing, binning, or removal.

o Handle inconsistencies/erroneous values:

▪ Example: Negative age, inconsistent gender labels ("M/F" vs.

"Male/Female").

▪ Use domain knowledge/expert validation.

(C) Data Transformation

• Purpose: Convert data into suitable forms for algorithms.

• Techniques:

o Normalization/Scaling: Rescale values into [0,1] or [−1,+1]. Prevents one

variable (e.g., income) from dominating others.

o Discretization: Convert continuous data into categories (e.g., Income →

Low, Medium, High).

o Aggregation/Concept hierarchies: Reduce many values into fewer

groups (e.g., U.S. states → regions).
 o Attribute construction: Derive new variables for better representation.

▪ Example: Instead of separate donor & recipient blood types, create

a single "blood-type match" attribute.

(D) Data Reduction

• Problem: Too much data may hinder analysis.

• Dimensions of reduction:

1. Reduce attributes (columns) → Dimensionality reduction.

▪ Methods: PCA, ICA, Chi-square, correlation analysis, decision tree

induction.

▪ Goal: Keep the most relevant features.

2. Reduce records (rows) → Sampling.

▪ Simple random sampling.

▪ Stratified sampling (ensure subgroup representation).

3. Balance skewed data → Oversample minority class or undersample

majority class.

▪ Example: Predicting rare disease patients, fraud detection, or

student dropout.

• Key insight: Balanced datasets → better prediction accuracy.

3. Summary Table of Preprocessing Tasks (Table 3.1)

Main Task Subtasks Popular Methods

Data
Collect, select, integrate SQL, APIs, domain expertise
consolidation

Handle missing, noise, Imputation, outlier detection, domain

Data cleaning
erroneous knowledge

Data Normalize, discretize, Scaling, binning, concept hierarchies,

transformation construct derived attributes
Main Task Subtasks Popular Methods

Reduce attributes, PCA, sampling, stratified sampling,

Data reduction
records, balance oversampling

4. Case Study: Improving Student Retention with Data Analytics (Analytics in Action
3.1)
Problem Context

• Issue: Student attrition (dropping out) is a serious problem.

o Only ~50% of students in U.S. higher education earn a degree.

o Consequences: financial losses, poor graduation rates, weaker

institutional reputation.

• Goal: Identify students at risk of dropping out → intervene early.

Proposed Solution

• Use historical student data (academic, financial, demographic).

• Apply predictive analytics → similar to churn analysis in marketing.

• Dataset: 16,000+ first-year students, 5 years, from a U.S. university.

Preprocessing Applied

• Data consolidation: Merge academic, financial aid, demographic databases.

• Data cleaning: Removed records with missing critical info (e.g., international
students without SAT).

• Data transformation:

o Aggregated "Major + Concentration" → simpler binary variables.

o Derived variables:

▪ Earned/Registered ratio = Earned Hours ÷ Registered Hours.

▪ YearsAfterHS = Enrollment Year − HS Graduation Year.

• Data balancing: Original dataset had 80% "Yes" (persisted) vs 20% "No"
(dropped out). Balanced using random sampling.

Modeling & Results

 • Models tested: Decision Trees, Neural Networks, SVM, Logistic Regression.

• Unbalanced data results:

o Overall accuracy ~87%, BUT poor performance for dropout prediction

(<50%).

• Balanced data results:

o SVM performed best (~81%).

o All models predicted dropouts much better.

• Ensemble models (bagging, boosting, fusion): Slightly improved results

(~82%).

• Sensitivity analysis: Ranked important predictors (e.g., GPA, student loans,

high school scores).

Key Findings

• Balanced datasets → more reliable dropout predictions.

• SVM had best accuracy, but Decision Trees preferred for interpretability.

• Data preprocessing (especially balancing & derived variables) was crucial for
success.

5. Key Takeaways

• Data preprocessing is essential for analytics success.

• Four major steps: Consolidation → Cleaning → Transformation → Reduction.

• Good preprocessing improves model performance more than simply choosing

complex algorithms.

• Case study demonstrates real-world impact: proper preprocessing helps

institutions predict and reduce student dropout rates.
Lecture Notes – Definition of Big Data (Section 3.5)

1. Introduction

• Businesses rely on data to understand customers and operations.

• Traditional tools fail to handle the scale, diversity, and speed of modern data.

• “Big Data” has become both a buzzword and a core concept in analytics.

2. What is Big Data?

• General View: Data sets that are too large and complex to be processed using
traditional methods.

• Relative Nature: "Big" depends on the size and capacity of each organization.

• Formal View: Data that exceeds the capability of current hardware/software to

capture, manage, and process within a tolerable time.

• Key Idea: Not only about volume but also about value extraction from varied
data sources.

3. Origins and Sources of Big Data

• Comes from everywhere:

o Web logs, RFID, GPS, IoT sensors, social media, emails, texts

o Medical records, genomics, astronomy, nuclear physics

o Scientific research, military surveillance, multimedia archives, e-

commerce

• Previously ignored sources are now seen as valuable gold mines due to new
technology.

4. Evolution of Big Data

• 1990s: Large Data Warehouses (terabytes considered “big”).

• Now: Petabytes → Exabytes → Zettabytes → Yottabytes → (future: Brontobytes,

Geobytes).

• Internet traffic & IoT accelerating growth.

• 2009: 0.8 ZB → 2011: 1.8 ZB → 2020 forecast: 44 ZB.

5. The “V”s of Big Data

1. Volume – Massive scale of data (transaction logs, social media, sensors, RFID,
GPS).

o Storage is less of a problem now → main issue is relevance and extracting

value.

2. Variety – Different formats: structured, semi-structured, unstructured (80–85%

of org. data).

o Examples: text, emails, XML, multimedia, logs.

3. Velocity – Speed of data generation and need for real-time/near real-time

processing.

o Example: stock trades, health monitoring, traffic data.

o Leads to streaming analytics (in-motion analytics).

4. Veracity – Accuracy, trustworthiness, and quality of data. (IBM emphasis).

5. Variability – Data flow inconsistency: seasonal peaks, trends, viral social media
events.

6. Value Proposition – True importance of Big Data lies in insight generation →

better decisions, competitive advantage.

6. Why Big Data Matters

• Enables organizations to:

o Detect patterns and anomalies.

o Gain business insights not possible with small data.

o Support innovation, growth, and differentiation.

• Serves as the foundation for Business Intelligence (BI), Predictive Analytics,

and Operational Intelligence.

7. Conceptual Architecture of Big Data Solutions (Fig. 3.7)

Flow:

• Sources → (ERP, SCM, CRM, IoT, Social Media, Logs, Multimedia)

 • Data Movement → Fast loading, filtering, processing

• Data Management → Data lakes, integrated data warehouses

• Analytics Layer → BI, predictive analytics, pattern detection, hypothesis testing

• Users → Business analysts, executives, frontline workers, partners, customers

8. Key Takeaways

• Big Data is not just about size but about multi-dimensional characteristics
(the Vs).

• It evolves continuously with technology and business needs.

• Value extraction is the ultimate goal.

• Critical for data-driven decision making in modern enterprises.

 Lecture Notes: Fundamentals of Big Data Analytics

1. Introduction to Big Data Analytics

• Big Data itself is not valuable until it is analyzed to deliver business value.

• Traditional reports/dashboards are limited → lack in-depth, on-demand analysis.

• Big Data Analytics provides new methods, technologies, and paradigms for
extracting actionable insights.

2. Why Big Data Analytics?

Organizations should consider Big Data Analytics when:

• Current platforms cannot handle required data volume.

• Need to include new data sources (e.g., social media, IoT, RFID, GPS, web logs,
text data).

• Require real-time or near real-time analysis.

• Schema-on-demand is needed → flexibility instead of rigid RDBMS structures.

• Data arrives at high velocity beyond traditional systems’ capability.

3. Critical Success Factors for Big Data Analytics

(Figure 3.8 framework)

1. Clear Business Need

o Align with organizational vision and strategy.

o Technology investment must serve business goals.

2. Strong, Committed Sponsorship

o Executive-level champions are essential.

o For enterprise-wide projects → sponsorship must be organization-wide.

3. Alignment Between Business & IT Strategy

o Analytics supports business strategy, not the other way around.

4. Fact-Based Decision-Making Culture

o Encourage data-driven decision making, not intuition alone.

o Senior management role:

▪ Be vocal supporters.

▪ Demand analytics-backed decisions.

▪ Link incentives to analytical use.

5. Strong Data Infrastructure

o Integrate traditional data warehouses with Big Data technologies.

o Ensure synergy between old and new platforms.

4. High-Performance Computing for Big Data

• Needed to manage computational complexity.

• Techniques:

o In-Memory Analytics → Real-time, distributed data processing.

o In-Database Analytics → Perform analytics directly within databases.

o Grid Computing → Use shared IT resources efficiently.

o Appliances → Integrated hardware + software solutions, scalable on

demand.
 5. Challenges in Big Data Analytics

• Data Volume → Capturing, storing, and processing huge datasets quickly.

• Data Integration → Combining heterogeneous data sources.

• Processing Capabilities → Need for real-time/stream analytics.

• Data Governance → Issues of security, privacy, ownership, and quality.

• Skills Availability → Shortage of skilled data scientists.

• Solution Cost → High experimentation cost; need cost-effective approaches.

6. Business Problems Addressed by Big Data Analytics

• Common applications:

o Process Efficiency & Cost Reduction (manufacturing, govt, energy,

utilities).

o Enhanced Customer Experience (retail, insurance).

o Risk Management (banking, education).

• Partial list of problems solved:

o Process efficiency & cost reduction.

o Brand management.

o Revenue maximization (cross-sell, up-sell).

o Churn detection & customer acquisition.

o Improved customer service.

o Identifying new products & markets.

o Regulatory compliance.

o Enhanced security capabilities.

7. Key Takeaway

• Big Data Analytics transforms raw data into business value.

• Success depends on business alignment, sponsorship, infrastructure, and

culture.
• Challenges exist (volume, velocity, integration, cost, skills), but benefits
outweigh risks.

• Ultimately, the goal is actionable insights that drive business performance.

Lecture Notes: Big Data Technologies

1. Common Characteristics of Big Data Technologies

• Use commodity hardware → enables scale-out and parallel processing.

• Employ nonrelational data storage → support for unstructured &

semistructured data.

• Apply advanced analytics & visualization tools → deliver insights to end users.

• Three key transformative technologies:

1. Hadoop

2. MapReduce

3. NoSQL

2. Hadoop

• Open-source framework for processing, storing, analyzing massive distributed

data.

• Created by Doug Cutting at Yahoo! (inspired by Google’s MapReduce).

• Handles petabytes to exabytes of data using parallel, distributed nodes.

• Runs on inexpensive commodity hardware → cost-effective scalability.

• Core idea: Breaks data into smaller parts → process simultaneously on multiple
nodes.

How Hadoop Works

1. Data sources: logs, social media feeds, internal stores.

2. Data broken into parts, stored in HDFS (Hadoop Distributed File System).

o Replicated across nodes for fault tolerance.

o Name Node → tracks data location & node status.

3. MapReduce used for processing:

o Client submits a Map job (Java query).

o Job Tracker coordinates tasks across nodes.

o Each node processes locally (parallel computing).

o Reduce job aggregates results into final output.

4. Processed data → used for analytics, transferred to RDBMS/DW if needed.

3. MapReduce

• Programming model, not a language.

• Developed by Google → distributes processing of large data across clusters.

• Programmer writes map & reduce functions, system handles parallelization,

fault tolerance.

Example (Colored Squares Counter)

• Map function: Groups data (e.g., squares by color).

• Reduce function: Aggregates counts (e.g., total squares of each color).

• Shuffle/Sort ensures results are merged correctly.

 Why Use MapReduce?

• Enables analysis of large, multi-structured datasets.

• Applications: indexing/search, graph analysis, text mining, ML, data

transformation.

• Programmers benefit:

o Don’t need expertise in distributed computing.

o Can use open-source/commercial libraries (e.g., Apache Mahout).

4. Hadoop Technical Components

• HDFS → Storage layer.

• Name Node → Metadata + coordination.

• Secondary Node → Backup for Name Node.

• Job Tracker → Assigns & manages jobs.

• Worker Nodes → Store/process data.

Key Subprojects in Hadoop Ecosystem

• Hive → SQL-like queries (HiveQL → MapReduce).

• Pig → High-level query language, supports long data pipelines.

• HBase → Nonrelational DB, low-latency lookups.

• Flume → Data ingestion tool.

• Oozie → Workflow/job scheduling.

• Ambari → Hadoop cluster management tool.

• Avro → Data serialization system.

• Mahout → Machine learning library (clustering, regression, classification).

• Sqoop → Move data between RDBMS/DW and Hadoop.

• HCatalog → Metadata management.

 5. Hadoop Pros & Cons

Pros

• Handles massive unstructured/semistructured data.

• Processes all data, not just samples.

• Cost-effective (commodity hardware, open source).

• Iterative, exploratory analytics possible.

Cons

• Immature & evolving technology.

• Requires specialized skills (Hadoop devs, data scientists).

• Complex to manage clusters.

• Batch-oriented → no native real-time processing (led to Spark development).

6. Spark vs. Hadoop

Hadoop

• Introduced 2004, batch processing model.

• Components: HDFS, YARN, MapReduce, Hadoop Common.

• Strengths: cost-effective, scalable, fault-tolerant.

• Limitation: slow, disk-based (no real-time).

Spark

• Introduced 2009 → in-memory processing.

• Components: Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX.

• Benefits:

o Much faster (RAM-based).

o Unified engine (SQL, streaming, ML, graph).

o Programmer-friendly APIs.

Comparison

• Performance: Spark faster (RAM vs disk).

• Cost: Hadoop cheaper (disk-based); Spark costlier (requires lots of RAM).

• Processing: Hadoop → batch jobs; Spark → real-time + streaming.

• Scalability: Hadoop scales quickly; Spark often runs on HDFS.

• Security: Hadoop stronger; Spark integrates with Hadoop for higher security.

• Analytics: Spark stronger (MLlib, advanced analytics).

When to Use

• Hadoop:

o Huge datasets > memory capacity.

o Batch jobs, archival analysis.

o Low-cost infrastructure.

• Spark:

o Real-time stream analytics.

o Iterative ML algorithms.

o Graph processing.

o Quick results needed.

7. NoSQL

• Not Only SQL → Designed for Big Data, unlike relational DBMS.

• Best for real-time, discrete data retrieval from large multistructured datasets.

• Often used with Hadoop (e.g., HBase on top of HDFS).

Examples

• HBase, Cassandra, MongoDB, Accumulo, Riak, CouchDB, DynamoDB.

Drawbacks

• Often trade ACID compliance for performance/scalability.

• Lack mature management & monitoring tools.

8. Data for Good Initiatives

• Use data science/analytics for social and environmental impact.

• Examples: poverty reduction, healthcare, climate change, humanitarian issues.

• Based on public/open data + analytics tools.

Notable Initiatives (Table 3.5)

• TIBCO4Good → Humanitarian efforts.

• SAS – Data for Good → Solve issues in poverty, health, rights, environment.

• IBM – Science for Social Good → AI + cloud for social impact.

• Teradata – Doing Good with Data → Helps nonprofits globally.

• DataKind → Harnesses data science for humanity.

• Data for Good (Canada) → Nonprofit chapters supporting NGOs.

• Facebook – Data for Good → Privacy-preserving community data.

• Data & Analytics for Good (Open Journal) → Research aligned with UN SDGs.

• Appsilon Data for Good → AI & visualization for global challenges.

• [Link] → Builds partnerships for data science for social impact.

Takeaway

• Good way for students & professionals to gain experience.

• Opportunity to apply analytics while contributing to society.

 Lecture Notes – Big Data and Stream Analytics

1. Introduction to Big Data Velocity

• Big Data is defined by Volume, Variety, and Velocity.

• Velocity → speed at which data is generated and streamed.

• Traditional analytics works on data at rest (stored data) → often too late or
contextually wrong.

• Organizations need real-time or near real-time analytics to make timely and

accurate decisions.

2. Need for Stream Analytics

• Businesses attempt to record every piece of data (might be useful now or later).

• Challenges:

o Rapid growth of data sources → storing everything is not feasible.

o Storage capacity < digital information generated.

o Real-time detection of changes and patterns is critical.

Solution:

• Stream Analytics (a.k.a. data-in-motion analytics, real-time analytics).

• Deals with unbounded data flows that cannot be stored permanently.

• Focuses on timely detection of patterns and anomalies.

3. Definition of Stream Analytics

• Stream = continuous sequence of data elements (Zikopoulos et al., 2013).

• Tuple = a data element (similar to a row in a relational DB, or a package of

attributes).

• Windows:

o Used when a single tuple is insufficient.

o A window of data = finite sequence of tuples.

o Continuously updated as new data arrives.

• Why popular now?

 o Faster time-to-action is required.

o Technological advances enable real-time processing.

4. Example Use Case: Smart Grids (Energy Industry)

• Smart grids integrate sensor data, smart meters, meteorological data.

• Goals:

o Predict electricity demand and production in real time.

o Detect anomalies (unexpected demand, renewable energy peaks).

o Optimize decisions (supply, production sources, pricing).

• Outcome → Better capacity planning, demand management, and pricing.

5. Applications of Stream Analytics

A. e-Commerce

• Every click, search, product view is captured and analyzed in real time.

• Enables:

o Personalized recommendations.

o Targeted pricing/offers.
 o Turning browsers into buyers.

B. Telecommunications

• Call Detail Records (CDR) generate massive streams.

• Applications:

o Churn analysis → identify influencers/leaders in networks.

o Social network analysis for marketing and retention.

o Combine with social media sentiment → measure campaign

effectiveness.

o Replicated for data networks using IP detail records.

C. Law Enforcement & Cybersecurity

• Streaming data enables:

o Real-time situational awareness.

o Video surveillance, face recognition.

o Cyberattack and intrusion detection.

o Network log monitoring.

D. Power Industry

• Smart meters now send readings every 15 minutes (or faster).

• Real-time analysis helps:

o Optimize supply and demand.

o Integrate weather/natural data for renewable generation.

o Better forecasting at different geographic levels.

E. Financial Services

• High-volume trading data analyzed with low latency.

• Benefits:

o Faster buy/sell decisions.

o Fraud detection.

o Regulatory oversight.

F. Health Sciences
• Medical devices (ECG, BP monitors, glucose monitors) generate fast diagnostic
data.

• Real-time analytics helps:

o Detect anomalies early (life-saving).

o Improve patient safety and treatment efficiency.

o Support doctors with faster decision-making.

G. Government

• E-Government + social media = large structured & unstructured streams.

• Applications:

o Natural disaster management (storm, fire, flood tracking).

o Environmental monitoring (air, water quality).

o Smart traffic management (traffic sensors, GPS, cameras).

6. Key Takeaways

• Stream Analytics = analyzing data as it flows (real-time).

• Critical because:

o Storage cannot keep up with data growth.

o Businesses need immediate insights.

• Applications span across industries: energy, commerce, telecom, security,

healthcare, finance, and government.

• Core Value: Turning fast data into fast actions.

 Lecture Notes – Statistical Modeling for Business Analytics

1. Introduction

• Business analytics has increased reliance on statistical methods.

• Once limited to statisticians → now widely used by business users & managers.

• Statistics provides tools for evidence-based decision-making.

• Two roles in analytics:

o Descriptive Analytics → summarizing & presenting data.

o Predictive Analytics → forecasting outcomes (e.g., regression,

clustering).

2. Statistics in Business Analytics

• Two branches of Descriptive Analytics:

1. OLAP (Online Analytical Processing)

▪ Multidimensional data analysis (data cubes).

▪ Used in business intelligence for summarizing structured data.

2. Statistics

▪ Describes data with descriptive & inferential methods.

3. Descriptive vs. Inferential Statistics

 • Descriptive Statistics: Summarize sample data (tables, charts, averages).

• Inferential Statistics: Make predictions or generalizations about population.

4. Descriptive Statistics for Business Analytics

• Converts raw data → meaningful insights.

• Identifies:

o Data concentration.

o Outliers (very large/small values).

o Distribution patterns.

• Two main categories:

1. Measures of Central Tendency.

2. Measures of Dispersion (Spread).

5. Measures of Central Tendency

a) Arithmetic Mean (Average)

• Most common measure of centrality.

• Balances deviations above & below mean.

• Unique for every dataset.

• Limitation: Sensitive to outliers and skewness.

b) Median

• Middle value in an ordered dataset.

• If odd → middle observation.

• If even → average of two middle observations.

• Resistant to outliers/skewness.

• Suitable for ordinal data.

 c) Mode

• Most frequent value.

• Useful for nominal data.

• Limitations: not informative if many unique values.

Best practice: Use mean, median, and mode together for a fuller picture.

6. Measures of Dispersion (Spread)

a) Range

• Simplest measure.

• Highly sensitive to outliers.

b) Variance

• Measures average squared deviation from mean.

• Larger variance → more spread out data.

• Units squared → less interpretable.

c) Standard Deviation

• Square root of variance.

• Widely used for business applications.

d) Mean Absolute Deviation (MAD)

• Simpler than variance.

• Avoids squared values.

e) Quartiles & Interquartile Range (IQR)

• Spread of middle 50% of data.

• Robust to outliers.

7. Visualization – Box and Whisker Plot

• Shows:

o Median & sometimes mean.

o Quartiles (Q1, Q3).

o Whiskers → data spread (±1.5×IQR).

o Outliers beyond whiskers.

• Helps detect:

o Symmetry vs. skewness.

o Range & variability.

 8. Shape of a Distribution

a) Normal Distribution

• Symmetric bell-shaped curve.

• Mean = Median = Mode.

b) Skewness

• Positive skew → long right tail, mean > median.

• Negative skew → long left tail, mean < median.

• Zero skew → symmetric.

 c) Kurtosis

• Positive kurtosis (>3) → tall, peaked distribution.

• Negative kurtosis (<3) → flat distribution.

• Normal distribution kurtosis = 3.

9. Technology Insight – Excel for Descriptive Statistics

• Analysis ToolPak provides ready-made statistical functions.

• Can calculate:

o Mean, Median, Mode.

o Variance, Std. Dev., Range.

o Quartiles, Skewness, Kurtosis.

• Also generates box plots easily.

• Limitation: ToolPak results are not dynamic (do not update automatically
with new data).

10. Analytics in Action – Town of Cary (North Carolina)

• Problem: Water leaks & inefficient usage.

• Solution: Installed wireless meters for 60,000 customers.

o Hourly readings → 8,760 data points/year per customer.

o Analyzed using SAS Analytics.

Benefits:

• Leak detection within days/hours.

• Customers set usage alerts.

• Online portal for tracking usage.

• Detected anomalies (e.g., faulty dishwasher running continuously).

• Supported future infrastructure planning (water plant expansion).

• Encouraged targeted conservation.

• Cost savings: > $10 million by removing manual readings + leak prevention.

• Drought readiness with real-time monitoring.

11. Key Takeaways

• Descriptive statistics → foundation of business analytics.

• Must consider both centrality & dispersion together.

• Box plots, skewness, kurtosis → add depth to data understanding.

• Software tools (Excel, SAS, R, Python) make implementation easier.

• Case study (Cary, NC) shows how analytics solves real-world municipal
problems.
 Lecture Notes: Regression Modeling for Inferential Statistics

1. Regression Modeling for Inferential Statistics

Introduction

• Regression → Statistical technique to model the dependence of a response

variable (Y) on one/more explanatory variables (X).

• Origin: Sir Francis Galton (sweet peas study) → Karl Pearson.

• Two main uses:

o Hypothesis testing → To explore/understand relationships.

o Prediction/forecasting → To estimate Y for given values of X.

Correlation vs Regression

• Correlation:

o Measures the strength/degree of association between two variables.

o No assumption of causality.

• Regression:

o Models dependency → assumes causal effect (X → Y).

o Explains/forecasts Y using one or more X variables.

Types of Regression

1. Simple Regression:

o One X → One Y.

o Example: Height (X) → Weight (Y).

2. Multiple Regression:

o Several X → One Y.

o Example: Height, Gender, BMI → Weight.

3. Linear vs Nonlinear:

o Linear regression assumes a straight-line relationship.

o Nonlinear regression is used if the relationship is not linear.

 Model Development Steps

1. Data Visualization → Scatter plot (X vs Y).

2. Model Estimation → Ordinary Least Squares (OLS) method:

o Minimizes sum of squared errors (residuals).

o Equation (Simple Linear Regression):

o Equation (Multiple Linear Regression):

o Where:

▪ β₀ → Intercept

▪ β₁ → Slope

Model Evaluation

• Goodness of Fit:

o R² (Coefficient of Determination) → Proportion of variance explained (0–

1 scale).

▪ Social Sciences: R² ≈ 0.3 acceptable.

▪ Engineering: R² ≥ 0.7 preferable.

o F-test → Significance of model.

o RMSE (Root Mean Square Error) → Prediction accuracy.

Key Assumptions of Linear Regression

1. Linearity → Relationship between X and Y is linear.

2. Independence of Errors → No autocorrelation in residuals.

3. Normality of Errors → Errors should follow normal distribution.

4. Constant Variance (Homoscedasticity) → Equal error variance across values of

X.
5. No Multicollinearity → Explanatory variables should not be highly correlated.

2. Logistic Regression

Introduction

• A classification algorithm (not regression in the traditional sense).

• Developed in 1940s.

• Uses logit transformation of probabilities.

• Used when response variable is categorical (binary or multinomial).

Characteristics

• Input: Explanatory variables (continuous or categorical).

• Output: Class label (e.g., Yes/No, Win/Loss).

• Equation (Logistic Function):

Probability values always between 0 and 1.

Estimation

• Uses Maximum Likelihood Estimation (MLE) (not OLS).

• Iterative process until coefficients converge.

 Applications

• Medicine (disease diagnosis).

• Marketing (customer churn prediction).

• Sports analytics (game outcomes).

Summary

• Regression → Explains and predicts relationships (continuous outcomes).

• Logistic Regression → Classifies categorical outcomes using probability

models.

• Time Series Forecasting → Predicts future values based on temporal patterns.

• Model quality depends on assumptions, fit statistics, and validation

techniques.