BIDA NOTES

➢ Data Warehousing
- A data warehouse (DW) is a centralized pool of data designed to support decision-
making processes.
- It serves as a repository of current and historical data, potentially valuable to
managers across the organization.
- Data in a DW are structured for analytical processing, including OLAP, data mining,
querying, reporting, and other decision-support applications.
- A DW is characterized as subject-oriented, integrated, time-variant, and non-
volatile, aiding management in decision-making.

➢ Characteristics of Data Warehousing

- Subject-Oriented: Data is organized by detailed subjects such as sales, products,
or customers, focusing on information relevant to decision support. This orientation
enables comprehensive business insights by contrasting operational databases
tuned for transactional updates.
- Integrated: Ensures data from different sources is in a consistent format,
addressing naming conflicts and discrepancies among units of measure for total
integration.
- Time Variant: Maintains historical data to detect trends, deviations, and long-term
relationships. This temporal aspect supports forecasting and decision-making using
multi-time-point views like daily or monthly data.
- Nonvolatile: Data entered into the warehouse cannot be modified. Obsolete data is
discarded, and updates are stored as new entries, optimizing data access.
- Web-Based: Designed for efficient computing environments supporting web-based
applications.
- Relational/Multidimensional: Utilizes either relational or multidimensional structures
for organizing data effectively.
- Client/Server Architecture: Provides easy access for end users through a client-
server setup.
- Real-Time Access: Modern warehouses enable real-time data access and analysis
capabilities.
- Metadata Inclusion: Stores metadata that defines how data is organized and
provides guidance for effective usage.
Data warehousing encompasses the entire process of creating systems for decision
support, facilitating business insights, and providing access to business information.
The three main types of data warehouses are:
- Data marts
- Operational Data Stores (ODS)
- Enterprise Data Warehouses (EDW)

➢ Data Marts
- Smaller and focused on a particular subject or department, serving as a subset of
a data warehouse.
- Two types:
• Dependent Data Mart:
- Created directly from a data warehouse, ensuring consistent data models and
quality.
- Supports an enterprise-wide data model but requires the data warehouse to be
built first.
- Provides a single version of data for all users.
• Independent Data Mart:
- Smaller, designed for a strategic business unit (SBU) or department.
- Does not rely on an enterprise data warehouse (EDW) as its source.
- Cost-effective alternative for firms with limited budgets.

➢ Enterprise Data Warehouses (EDW)

- Large-scale data warehouses used across enterprises for decision support.
- Integrates data from multiple sources into a standardized format for BI and
decision-making.
- Supports various Decision Support Systems (DSS), including:
- Customer Relationship Management (CRM).
- Supply Chain Management (SCM).
- Business Performance Management (BPM).
- Business Activity Monitoring (BAM).
 - Product Life-Cycle Management (PLM).
- Revenue Management.
- Knowledge Management Systems (KMS).

➢ DATA WAREHOUSING PROCESS OVERVIEW

- Organizations collect and store data at an accelerated rate, making its

management increasingly complex due to scalability issues and rising user access
demands via networks like the Internet.
- Managing multiple databases, integrated or not, is a challenging but highly
beneficial task, offering immense advantages over its costs.
- Data warehouses, large repositories of time-series data for decision support, are
crucial for many organizations.
- Data for warehouses is sourced from various internal and external systems,
cleansed, and organized to align with organizational needs.
- Post data warehouse population, data marts for specific areas or departments can
be loaded or created and integrated into an Enterprise Data Warehouse (EDW).
- In some cases, data is directly manipulated using Business Intelligence (BI) tools
without creating data marts.
- Major components of the data warehousing process include:
• Data Sources: Data originates from operational legacy systems, external
providers, OLTP/ERP systems, or web logs.
• Data Extraction and Transformation: Data is processed using ETL (Extraction,
Transformation, and Loading) tools.
• Data Loading: Data is staged, cleansed, and prepared for loading into the data
warehouse or data marts.
• Comprehensive Database: The EDW provides summarized and detailed
information for decision-making.
 • Metadata: Maintained to help users and IT personnel index, search, and
organize data efficiently.
• Middleware Tools: Facilitate data access through SQL queries, managed
query environments, or front-end applications like data mining, OLAP,
reporting, and visualization tools.

➢ DATA WAREHOUSING ARCHITECTURES

- Information system architectures for data warehousing can be categorized as

client/server or n-tier architectures, with one-tier, two-tier, and three-tier being the
most common. These architectures are well-suited for large-scale, performance-
intensive systems like data warehouses.
- Hoffer et al. (2007) divided the data warehouse architecture into three parts:
- The data warehouse containing data and associated software.
- Back-end (data acquisition) software for extracting, consolidating, summarizing,
and loading data.
- Front-end (client) software for user access and analysis.
• Three-Tier Architecture:
- The first tier includes operational systems and data acquisition software.
- The second tier houses the data warehouse.
- The third tier consists of DSS/BI/BA engines and clients.
- Offers advantages such as functional separation, resource efficiency, and the
ability to create data marts.
• Two-Tier Architecture:
- The DSS engine operates on the same hardware as the data warehouse.
- More economical but may face performance challenges with large, data-intensive
applications.
- Considerations for choosing an architecture:
 • Database Management System (DBMS): Relational DBMS (RDBMS) like
Oracle, SQL Server, and DB2 are commonly used. These systems support
both client/server and web-based architectures.
• Parallel Processing and Partitioning: Enhances query processing efficiency
and scalability. Designers must decide on partitioning criteria to handle large
data volumes.
• Data Migration Tools: Essential for transferring data into the warehouse. The
decision to use in-house, third-party, or integrated tools depends on the
complexity of the migration process.
• Data Retrieval and Analysis Tools: Necessary for data extraction,
transformation, and loading (ETL). Specialized ETL tools may be required for
complex or real-time migrations.

➢ DATA INTEGRATION AND THE EXTRACTION, TRANSFORMATION, AND

LOAD PROCESSES

- Before data warehouses, data marts, and BI software, accessing data sources was
a labor-intensive process, requiring database specialists even with modern Web-
based tools.
- As data warehouses grow, integrating data becomes increasingly complex.
- The ETL (Extraction, Transformation, and Load) process is central to data
warehousing and data-centric projects.
- ETL typically consumes 70% of the time in data-centric projects, posing challenges
for IT managers.
• ETL Process:
1. Extraction: Reads data from one or more databases.
2. Transformation: Converts data to the required format using rules, lookup
tables, or data combinations.
3. Load: Places transformed data into the data warehouse or target database.
- ETL tools integrate these functions, facilitating data movement between sources
and targets while managing metadata, scheduling, error handling, and audit logs.
- ETL is vital for data integration and ensures the warehouse is loaded with
integrated and cleansed data from sources like ERP applications, CRM tools, flat
files, and message queues.
- Data warehouses incorporate business rules for usage, summarization,
standardization, and calculations.
- Organizations face decisions between purchasing or building ETL tools due to
challenges such as high costs, steep learning curves, and difficulty assessing
performance early.
❖ Key criteria for selecting ETL tools:
- Compatibility with various data source architectures.
- Automatic metadata capturing and delivery.
- Adherence to open standards.
- User-friendly interface for both developers and functional users.

DATA WAREHOUSE DEVELOPMENT

• A data warehousing project is complex, involving multiple departments and
interfaces, and can be integral to a CRM business strategy.
• Direct benefits of a data warehouse:
o Enables extensive end-user analysis.
o Provides a consolidated view of corporate data (a single version of the
truth).
o Offers better, timely information processing, reducing operational system
load and improving request handling.
o Enhances system performance by shifting reporting tasks to decision
support systems (DSS).
o Simplifies data access for end users.
• Indirect benefits:
o Enhances business knowledge.
o Provides a competitive advantage.
o Improves customer service and satisfaction.
o Facilitates decision-making and business process reform.
 • The success of a data warehouse project depends on careful planning and
cost management.
• Costs to consider:
o Hardware, software, and network bandwidth.
o Internal development, support, and training.
o External consulting expenses.

Data Warehouse Development Approaches

 • Many organizations create data warehouses to support decision-making,
employing two main approaches: the Inmon model and the Kimball model.
• Inmon Model (EDW Approach): Known as the enterprise data warehouse
approach, Inmon's top-down methodology uses traditional database tools, like
entity-relationship diagrams (ERDs), and spiral development techniques. The
EDW provides a consistent, comprehensive enterprise-wide view, allowing for
data marts as needed.
• Kimball Model (Data Mart Approach): Kimball's bottom-up methodology
focuses on building data marts, which are subject-specific or department-
focused mini data warehouses. This approach uses dimensional data
modeling and adopts a "plan big, build small" philosophy.
• There is no universally superior model; the choice depends on an enterprise's
requirements, maturity in data management, and evolving user demands.
• Data marts are often a practical starting point, allowing organizations to gain
experience and demonstrate the business value of data warehousing before
scaling to a more complex data warehouse.

Representation of Data in Data Warehouse

• Data warehouse architecture can have various designs, but the foundation of
data representation relies on dimensional modeling, a system supporting high-
volume query access.
• Dimensional modeling enhances the processing of complex multidimensional
queries and is often implemented using star or snowflake schemas.
• Star Schema:
o Simplest and most commonly used style of dimensional modeling.
 o Contains a central fact table linked to several dimension tables via
foreign keys.
o The fact table includes descriptive attributes for decision analysis, such
as performance measures, metrics (e.g., sales volumes, profit margins),
and foreign keys to dimension tables.
o Dimension tables classify and aggregate information, facilitating analysis
and summarization of data in the fact table.
o Designed for fast query response, simplicity, and ease of maintenance
for read-only databases.
• Snowflake Schema:
o A more normalized version of the star schema, resembling a snowflake
in the entity-relationship diagram.
o Features centralized fact tables connected to multiple related dimension
tables.
o Normalized dimensions involve splitting attributes into multiple related
tables, unlike the denormalized single-table dimensions in the star
schema.

DATA WAREHOUSING IMPLEMENTATION ISSUES

• Implementing a data warehouse is a complex endeavor requiring thorough
planning and adherence to established methodologies.
• The success of Business Intelligence (BI) and data warehousing initiatives can
be benchmarked using maturity models like the one developed by The Data
Warehousing Institute (TDWI), which includes six stages: prenatal, infant,
child, teenager, adult, and sage.
• Key characteristics of the maturity model stages include scope, analytic
structure, executive perceptions, stewardship, and change management.
• Data warehouse projects are large-scale, resource-intensive, and risk-laden;
careful risk assessment at project inception is critical.
• Risks to avoid include:
o Starting with ineffective sponsorship chains or lacking a strong executive
sponsor and project driver.
o Setting unrealistic expectations during the selling phase of the project.
o Engaging in politically naive behavior, such as oversimplifying the
benefits of the warehouse.
 o Overloading the warehouse with unnecessary data, leading to
inefficiencies.
o Designing the data warehouse like a transactional database instead of
focusing on aggregates and multidimensional structures.
o Appointing a technology-oriented manager over a user-oriented one.
o Ignoring external data, unstructured data, or multimedia formats.
o Delivering inconsistent or conflicting data definitions without proper
cleansing.
o Underestimating capacity, scalability, and performance needs.
o Believing the process ends with deployment; data warehousing is
iterative and ongoing.
o Relying solely on periodic reporting instead of implementing alert
systems for real-time critical event monitoring.
• Senior management support and a high-level project champion are crucial for
securing the necessary resources and ensuring project success.
• A well-implemented data warehouse establishes an architectural framework
for decision-making and may integrate Supply Chain Management (SCM).
• Web-based data warehouses enhance accessibility but introduce challenges,
including security and cost considerations such as high-end processors and
storage.
• User participation in data and access modeling is critical to success.
o Data modeling determines needed data, business rules, and
aggregations.
o Access modeling defines data retrieval methods and indexes, and
identifies the need for dependent data marts.
• Teams must possess expertise in database technology, source systems, and
development tools to successfully implement and maintain a data warehouse.

REAL-TIME DATA WAREHOUSING

• Data warehousing and BI tools traditionally focus on assisting with strategic
and tactical decisions.
• The role of data warehouses is evolving due to increasing data volumes and
faster update speeds.
• Traditional data warehouses, often updated weekly, are insufficient for near-
real-time decision-making needs.
 • Real-time data warehousing (RDW), also known as active data warehousing
(ADW), enables immediate data availability for operational decision-making.
• RDW evolved from the enterprise data warehouse (EDW) concept, supporting
tactical decisions and empowering stakeholders with real-time information.
• Active data warehousing provides significant benefits for customer service,
supply chain management, logistics, and e-business applications.
• Companies like [Link] and Egg pie demonstrate successful adoption
of near-real-time customer data warehouses.
• The evolution of data warehousing includes reporting past events, analysis,
predictions, operationalization, and event-driven actions like marketing
campaigns.
• Business needs driving real-time data requirements include:
o The inability to wait for daily data updates for operational analysis.
o Capturing incremental real-time data instead of static snapshots.
o Reducing the complexity and cost of maintaining multiple systems
versus centralized data architecture.
o Addressing challenges with large nightly batch ETL loads.
• Challenges in RDW development include architectural complexities, data
modeling, database design, scalability, and ensuring consistent data accuracy.
• Terminologies like real-time data warehousing, near-real-time warehousing,
zero-latency warehousing, and active data warehousing are often used
interchangeably.

Definitions, Characteristics, and Benefits of Data mining

• Data mining is the process of discovering or "mining" knowledge from large
datasets, also referred to as knowledge extraction, pattern analysis, data
archaeology, information harvesting, pattern searching, and data dredging.
• It employs statistical, mathematical, and artificial intelligence techniques to
extract and identify useful information and patterns, such as business rules,
affinities, correlations, trends, or predictive models.
• Defined as a nontrivial process for identifying valid, novel, potentially useful,
and understandable patterns in structured databases, where:
o Process involves iterative steps.
o Nontrivial implies experimentation and inference beyond straightforward
computations.
 o Valid ensures discovered patterns hold true for new data with certainty.
o Novel indicates patterns previously unknown within the analyzed
system.
o Potentially useful ensures the patterns provide benefits or insights.
o Ultimately understandable means patterns must make business sense,
often leading to insights upon interpretation.
• Key characteristics and objectives of data mining include:
o Data is often deeply embedded within extensive databases, sometimes
spanning multiple years, and is cleansed and consolidated into data
warehouses.
o Typically performed in client/server or web-based information system
architectures.
o Advanced visualization tools assist in extracting information buried in
corporate archives or public records, sometimes incorporating
unstructured text (e.g., text files, intranets).
o End users, equipped with data drills and query tools, can quickly obtain
answers without extensive programming skills.
o The process may uncover unexpected results, requiring creative thinking
to interpret findings effectively.
o Data mining tools integrate with spreadsheets and other software,
enabling quick analysis and deployment of insights.
• Due to the large datasets and computational demands, parallel processing is
often necessary for effective data mining.

A Simple Taxonomy of Data

• Data is a collection of facts obtained through experiences, observations, or
experiments, consisting of numbers, letters, words, images, or voice
recordings. It is considered the lowest level of abstraction from which
information and knowledge are derived.
• At the highest level, data can be classified into structured and
unstructured/semistructured:
o Structured data is used by data mining algorithms and can be further
classified as categorical or numeric.
o Unstructured/semistructured data includes combinations of textual,
image, voice, and web content.
• Categorical Data:
 o Represents labels of multiple classes, dividing variables into specific
groups.
o Often referred to as discrete data, indicating a finite set of values
without a continuum.
o Subtypes:
▪ Nominal Data: Simple codes used as labels without
measurement.
▪ Example: Marital status (e.g., single, married, divorced).
▪ Ordinal Data: Codes that represent rank order among objects.
▪ Example: Credit score (e.g., low, medium, high).
• Numeric Data:
o Represents numeric values of variables, such as age, income, or
temperature.
o Can be integer (whole numbers) or real (including fractional values).
o Also called continuous data, indicating scalable measurements with
infinite fractional values.
o Subtypes:
▪ Interval Data: Measured on scales without an absolute zero.
▪ Example: Temperature on the Celsius scale.
▪ Ratio Data: Measurements involving absolute scales with
meaningful zero.
▪ Example: Mass, length, or energy.
• Other data types, including textual, spatial, imagery, and voice, must be
converted into categorical or numeric forms for processing by data mining
algorithms.
• Data can also be classified as static or dynamic (e.g., temporal or time-series
data).

How Data Mining Works

• Data mining identifies patterns in datasets using mathematical models to
uncover relationships among attributes. Patterns can be explanatory (showing
interrelationships) or predictive (forecasting future values).
• Data mining primarily focuses on four types of patterns:
 o Associations: Identifying commonly co-occurring groupings, such as
beer and diapers in market-basket analysis.
o Predictions: Forecasting future occurrences based on past data, like
predicting Super Bowl winners or daily temperatures.
o Clusters: Grouping items based on shared characteristics, such as
customer segmentation based on demographics and behavior.
o Sequential Relationships: Finding time-ordered events, such as
predicting the progression of a customer's banking habits.
• Tasks in data mining are categorized into prediction, association, and
clustering. Methods are classified as supervised or unsupervised learning
based on whether class attributes are included in the training data.
• Prediction: Involves forecasting outcomes using data and experiences.
Prediction methods include:
o Classification: Predicting class labels (e.g., "rainy" or "sunny") using
tools like decision trees, neural networks, and SVMs.
o Regression: Forecasting numerical values (e.g., temperature) through
linear models or machine learning techniques.
• Classification: Utilizes supervised learning to analyze historical data and
predict future behaviors. Tools include:
o Neural Networks: Effective for complex relationships but require
significant training and computational resources.
o Decision Trees: Efficient for categorical and interval data, using
hierarchical if-then statements.
o Rule Induction: Generates direct if-then rules from training data.
o Emerging methods like SVMs, rough sets, and genetic algorithms are
expanding classification capabilities.
• Clustering: Groups objects with shared characteristics without predefined
class labels. Popular techniques include:
o K-Means: Groups items based on similarity.
o Self-Organizing Maps: Uses machine learning for clustering.
o Applications include market segmentation and identifying natural
groupings in data for targeted marketing.
• Associations: Finds relationships among variables in large datasets, often
used in market-basket analysis. Derivatives include:
o Link Analysis: Identifies linkages between objects, like web pages or
academic authorship.
 o Sequence Mining: Explores time-ordered associations, uncovering
patterns over time.
o Algorithms like Apriori, FP-Growth, and Eclat facilitate association rule
discovery.
• Data mining enables firms to perform tasks like market segmentation, targeted
marketing, and customer behavior analysis using these methodologies
effectively.

VISUALIZATION AND TIME-SERIES FORECASTING

• Visualization and time-series forecasting are two commonly used techniques
in data mining.
• Visualization helps in understanding underlying relationships and can be
combined with other data mining techniques.
• The term "visual analytics" refers to the integration of analytics and
visualization into a single environment for quicker knowledge creation.
• Time-series forecasting involves analyzing data recorded at regular intervals
to develop models for predicting future values of the same variable.

Data Mining Versus Statistics

• Similarities: Data mining and statistics both focus on identifying relationships
within data. Statistics is often considered the foundation of data mining.
• Differences:
 o Statistics starts with a well-defined hypothesis, while data mining begins
with a loosely defined discovery statement.
o Statistics collects primary data (sample data) to test a hypothesis,
whereas data mining and analytics use existing secondary data to
discover patterns.
o Data mining typically works with large datasets, aiming for as much data
as possible, while statistics works with the "right size" of data, often
using samples if the dataset is too large for analysis.
o The definition of "large data" differs: statistics considers a few hundred to
a thousand data points large, while data mining considers millions to
billions of data points as large.

DATA MINING APPLICATIONS

• Customer relationship management (CRM): CRM aims to create one-on-
one relationships with customers by understanding their needs and wants.
Data mining helps in:
o Identifying likely responders/buyers (customer profiling).
o Analyzing customer attrition (churn analysis).
o Discovering associations between products and services to maximize
sales.
o Identifying the most profitable customers and their needs.
• Banking: Data mining can aid banks in:
o Automating loan application processes by predicting probable defaulters.
o Detecting fraudulent credit card and online transactions.
o Maximizing customer value by recommending products and services.
o Forecasting cash flow for banking entities (e.g., ATM machines).
• Retailing and logistics: Data mining in retail helps to:
o Predict sales volumes at specific locations to optimize inventory.
o Identify sales relationships between products for better store layouts.
o Forecast consumption based on seasonal and environmental factors.
o Discover patterns in product movement in the supply chain, particularly
for perishable goods.
• Manufacturing and production: Manufacturers can use data mining for:
o Predicting machinery failures to enable condition-based maintenance.
 o Identifying anomalies to optimize manufacturing capacity.
o Discovering patterns to improve product quality.
• Brokerage and securities trading: Data mining aids brokers and traders by:
o Predicting bond price changes.
o Forecasting stock fluctuations.
o Assessing the effects of events on market movements.
o Identifying fraudulent activities in securities trading.
• Insurance: Data mining in the insurance sector helps:
o Forecast claim amounts for better business planning.
o Determine optimal rate plans based on claims and customer data.
o Predict customers likely to buy new policies.
o Identify and prevent fraudulent claims.
• Computer hardware and software: Data mining is used to:
o Predict disk drive failures before they occur.
o Filter unwanted web content and emails.
o Detect and prevent network security breaches.
o Identify unsecure software products.
• Government and defense: In defense, data mining helps:
o Forecast the cost of moving military personnel and equipment.
o Predict an adversary's moves for better strategies.
o Predict resource consumption for improved planning and budgeting.
o Share knowledge from military operations to enhance learning.
• Travel industry (airlines, hotels, rental car companies): Data mining
assists the travel industry by:
o Predicting sales to optimize pricing and maximize revenue.
o Forecasting demand to allocate resources effectively.
o Identifying the most profitable customers and offering personalized
services.
o Retaining valuable employees by addressing the causes of attrition.
• Healthcare: In healthcare, data mining is used to:
o Identify people without health insurance and the underlying factors.
 o Identify cost-benefit relationships between treatments.
o Forecast demand for healthcare services to optimize resource allocation.
o Understand the reasons for customer and employee attrition.
• Medicine: Data mining in medicine helps:
o Improve cancer survival rates by identifying novel patterns.
o Predict success rates of organ transplants for better matching policies.
o Identify gene functions in human chromosomes (genomics).
o Discover the relationships between symptoms, illnesses, and
treatments.
• Entertainment industry: Data mining is used to:
o Analyze viewer data to optimize prime-time programming and
advertisement placement.
o Predict the financial success of movies before production.
o Forecast demand to optimize event scheduling and resource allocation.
o Develop optimal pricing policies to maximize revenue.
• Homeland security and law enforcement: Data mining supports security
and law enforcement by:
o Identifying terrorist behavior patterns.
o Discovering crime patterns to solve cases faster.
o Predicting and preventing biological and chemical attacks.
o Stopping malicious attacks on critical infrastructure.
• Sports: Data mining is used in sports to:
o Improve the performance of NBA teams and other professional sports
teams.
o Utilize predictive analytics in Major League Baseball for resource
optimization.
o Help professional teams use data crunchers to increase chances of
winning.

DATA MINING PROCESS

• CRISP-DM Process: The Cross-Industry Standard Process for Data Mining
(CRISP-DM) is a widely adopted methodology, proposed in the mid-1990s by
a European consortium, to standardize data mining practices. It consists of six
 sequential steps, starting with a clear understanding of the business problem
and ending with the deployment of a solution that addresses the business
need. Although the steps are sequential, data mining is an iterative process
that often requires backtracking based on experience and experimentation.
Special attention should be given to earlier steps to avoid errors that could
lead the entire project astray.

• Step 1: Business Understanding: The first step focuses on understanding

the business need. The goal is to identify the problem, define specific business
objectives, and develop a project plan, including the identification of
responsible people, data collection, and analysis, as well as the establishment
of a budget.
• Step 2: Data Understanding: This step involves identifying relevant data from
available sources, based on the defined business objectives. Understanding
the data sources and variables is crucial, as is exploring the data using
statistical and graphical techniques. Quantitative and qualitative data are
considered, and variables are carefully selected for analysis.
• Step 3: Data Preparation: This phase involves preparing data for analysis by
addressing issues such as missing values, noisy data, and inconsistencies.
Data is collected, cleaned, transformed, and reduced to ensure it is suitable for
mining. This phase consumes a significant amount of time in the process and
involves tasks such as imputation of missing values, smoothing outliers,
normalization, discretization, and dimensional reduction.
 • Step 4: Model Building: In this step, various modeling techniques are applied
to the prepared data to address the specific business problem. The process
involves testing and comparing different models, adjusting parameters, and
ensuring that the chosen method aligns with the business needs. Popular
methods include classification (e.g., decision trees), clustering (e.g., k-means),
and association rule mining (e.g., Apriori).
• Step 5: Testing and Evaluation: The models developed in the previous step
are assessed and evaluated to determine their effectiveness in meeting the
business objectives. This step ensures that the model performs well and
delivers the desired results, making any necessary adjustments before final
deployment.
• Step 6: Deployment in CRISP-DM involves more than just model
development and assessment. The results must be presented in a way that
end users can understand and apply. The deployment phase can range from
generating a simple report to implementing a repeatable data mining process
across the enterprise. While customers often carry out the deployment, data
analysts should ensure that customers are aware of the necessary steps to
utilize the models effectively. Additionally, maintenance of deployed models is
essential, as changes in the business environment can render models
outdated. A maintenance strategy should be in place to prevent the incorrect
use of results over time, and a monitoring plan is needed to track the
performance of complex models.

DATA MINING METHODS

• A variety of methods are available for performing data mining studies,
including classification, regression, clustering, and association. Most data
 mining software tools employ more than one technique (or algorithm) for each
of these methods.
• Classification is perhaps the most frequently used data mining method for
real-world problems. It learns patterns from past data to place new instances
into their respective groups or classes. Popular classification tasks include:
o Credit approval (e.g., good or bad credit risk)
o Store location (e.g., good, moderate, bad)
o Target marketing (e.g., likely customer, no hope)
o Fraud detection (e.g., yes, no)
o Telecommunication (e.g., likely to switch phone companies)
• In classification, the predicted output is a class label, whereas in regression, it
is a numeric value (e.g., temperature).
• Clustering, another data mining method, determines groups of objects without
supervised learning, while classification uses a supervised learning process to
learn the relationship between input and output variables.
• Classification Methodology involves two main phases:
o Model Development/Training: Using input data with class labels to train
the model.
o Model Testing/Deployment: Testing the model with a holdout sample
for accuracy and deploying it for real-world predictions.
• Factors considered in assessing classification models:
o Predictive accuracy: The model’s ability to correctly predict class labels
for unseen data.
o Speed: The computational cost of generating and using the model.
o Robustness: The model’s ability to handle noisy or missing data.
o Scalability: The ability to construct models efficiently with large
datasets.
o Interpretability: The model’s ability to provide insights into how
predictions are made.
• Classification Techniques include various algorithms, such as:
o Decision tree analysis: One of the most popular machine learning
techniques for classification.
o Statistical analysis: Includes techniques like logistic regression and
discriminant analysis, which assume linear relationships and
independent variables.
 o Neural networks: A popular machine learning approach for
classification problems.
o Case-based reasoning: Uses historical cases to categorize new data
based on similarities.
o Bayesian classifiers: Utilizes probability theory to place instances into
the most probable category.
o Genetic algorithms: Based on natural evolution, using directed-search
mechanisms for classification.
o Rough sets: Accounts for partial membership of class labels to
predefined categories when building classification models.

DECISION TREES
Before describing decision trees, it's important to understand some basic
terminology.
• Attributes: Decision trees involve multiple input variables that influence the
classification of patterns. For example, in a loan risk classification model,
income and credit rating would be attributes, and the output would be the class
label (e.g., low, medium, or high risk).
• Branches and Nodes: A tree consists of branches and nodes. A branch
represents the outcome of a test on an attribute, and a leaf node at the end
represents the final class choice. The path from the root node to the leaf node
can be represented as a chain of if-then statements.
• Recursive Division: A decision tree recursively divides a training set until
each division is primarily or entirely made up of examples from one class.
• Nonleaf Nodes and Splitpoints: Each nonleaf node contains a split point,
which is a test on one or more attributes that determines how the data is
divided.
• Tree Building and Pruning: In the growth phase, the tree is built by
recursively dividing the data until each division is pure or relatively small. Once
the tree is built, it is pruned to improve generalization and prediction accuracy.
• Split Criteria: The type of split used depends on the attribute. For continuous
attributes, splits are of the form value(A) < x. For categorical attributes, splits
are based on membership in a set (e.g., "Male versus Female").
• General Algorithm:
1. Create a root node and assign all training data to it.
2. Select the best splitting attribute.
 3. Add a branch to the root node for each split value and partition the data
into subsets.
4. Repeat steps 2 and 3 for each leaf node until the stopping criterion is
met (e.g., when a node is dominated by a single class label).

Cluster Analysis for Data Mining

• Cluster analysis is a key data mining method used to classify items, events, or
concepts into groups called clusters. It finds applications in various fields such
as biology, medicine, genetics, social network analysis, anthropology,
archaeology, astronomy, character recognition, and MIS development.
• In business, cluster analysis is widely used, particularly in marketing, fraud
detection (credit card and e-commerce fraud), and customer segmentation in
CRM systems. Its applications continue to expand as its strength is recognized
in business.
• Cluster analysis is an exploratory data analysis tool used for solving
classification problems. The goal is to sort cases (e.g., people, things, or
events) into clusters, ensuring strong associations within the same cluster and
weak associations between different clusters.
• A simple example of cluster analysis is assigning class grades in a college
course based on score ranges.

Association Rule Mining

• Association rule mining identifies interesting relationships between variables
(items) in large databases, commonly referred to as market-basket analysis
due to its use in retail business problems.
• The primary goal of market-basket analysis is to discover strong relationships
between products or services that are typically purchased together, either in
physical or virtual shopping environments.
o Example relationships:
▪ 65% of customers who buy automobile insurance also buy health
insurance.
▪ 80% of those who buy books online also buy music online.
▪ 60% of those who have high blood pressure and are overweight
also have high cholesterol.
▪ 70% of customers who buy a laptop and virus protection software
also purchase an extended service plan.
 • Input to market-basket analysis: Point-of-sale transaction data, where
products/services bought together are tabulated under a single transaction.
• Outcome of the analysis: Information that helps businesses understand
customer-purchase behavior and increase profits, including:
o Increased sales volume by placing related items together for easy
access.
o Promotion of products as packages, ensuring products on sale are
not sold separately.
o Product placement strategies, such as separating items to encourage
customers to explore more.
• Applications of market-basket analysis: Used in cross-marketing, cross-
selling, store design, catalog design, e-commerce site design, online
advertising optimization, product pricing, and sales/promotion configuration.
• Additional uses outside business: Association rules are used in various
fields like:
o Medical diagnostics, to identify relationships between symptoms,
illnesses, diagnoses, patient characteristics, and treatments (in medical
decision support systems).
o Genomics, to discover relationships between genes and their functions.
Common areas for association rule mining applications:
• Sales transactions: To improve product placement and promotional pricing.
• Credit card transactions: To predict products customers are likely to
purchase or detect fraudulent activity.
• Banking services: To identify additional services customers may need based
on their transaction history.
• Insurance service products: To propose additional products or detect
unusual insurance claims that may indicate fraud.
• Telecommunication services: To optimize product bundles and pricing based
on commonly purchased options.
• Medical records: To identify risks and correlations between conditions,
treatments, and outcomes.

DATA MINING SOFTWARE TOOLS

• Many software vendors provide powerful data mining tools, such as:
 o IBM (IBM SPSS Modeler, formerly SPSS PASW Modeler and
Clementine)
o SAS (Enterprise Miner)
o StatSoft (Statistica Data Miner)
o KXEN (Infinite Insight)
o Salford (CART, MARS, TreeNet, RandomForest)
o Angoss (KnowledgeSTUDIO, KnowledgeSeeker)
o Megaputer (PolyAnalyst)
• The most popular data mining tools are developed by well-established
statistical software companies (SPSS, SAS, and StatSoft) because statistics
form the foundation of data mining, and these companies have the resources
to develop them into full-scale data mining systems.
• In addition to commercial tools, several open-source and/or free data mining
software tools are available, including:
o Weka: Developed by the University of Waikato in New Zealand, it
includes many algorithms for various data mining tasks with an intuitive
user interface.
o RapidMiner: A free tool for noncommercial use, developed by Rapid-I,
featuring a graphical user interface, many algorithms, and data
visualization features.
o KNIME: Another open-source tool with a user-friendly interface,
available at [Link].
• The main difference between commercial tools (such as Enterprise Miner, IBM
SPSS Modeler, and Statistica) and free tools (such as Weka, RapidMiner, and
KNIME) is computational efficiency. Tasks involving large datasets may take
significantly longer to complete with free software, and some algorithms may
fail to run due to inefficient memory usage.

Data Mining and Privacy Issues

• Data collected, stored, and analyzed in data mining often includes personal
information such as identification data (name, address, Social Security
number, driver's license number, employee number, etc.), demographic data
(e.g., age, sex, ethnicity, marital status, number of children, etc.), financial data
(e.g., salary, family income, account balances, mortgage or loan specifics,
credit card details, etc.), purchase history (e.g., transactions from vendors or
credit card records), and other personal details (e.g., anniversaries,
pregnancy, illness, bankruptcy filings, etc.).
 • These data are often accessible through third-party data providers, raising
concerns about privacy. To protect individual privacy, data mining
professionals have ethical and legal obligations. One method of ensuring
privacy is through de-identification of customer records, making it impossible
to trace them back to specific individuals. Many publicly available datasets
(e.g., CDC, SEER, UNOS) are already de-identified, and users must often
consent not to attempt to identify the individuals behind the data before
accessing these sources.

Data Mining Myths and Blunders

Data mining is a powerful analytical tool that enables business executives to

advance from describing the nature of the past to predicting the future. It helps
marketers find patterns that unlock the mysteries of customer behavior. The results
of data mining can be used to increase revenue, reduce expenses, identify fraud,
and locate business opportunities, offering a whole new realm of competitive
advantage. As an evolving and maturing field, data mining is often associated with a
number of myths.
The following 10 data mining mistakes are often made in practice:
• Selecting the wrong problem for data mining.
• Ignoring what your sponsor thinks data mining is and what it really can and
cannot do.
• Leaving insufficient time for data preparation. It takes more effort than is
generally understood.
• Looking only at aggregated results and not at individual records. IBM's DB2
IMS can highlight individual records of interest.
• Being sloppy about keeping track of the data mining procedure and results.
 • Ignoring suspicious findings and quickly moving on.
• Running mining algorithms repeatedly and blindly. It is important to think hard
about the next stage of data analysis. Data mining is a very hands-on activity.
• Believing everything you are told about the data.
• Believing everything you are told about your own data mining analysis.
• Measuring your results differently from the way your sponsor measures them.

Key Points from the Content

Understanding Reports in Business Contexts

• Definition: A report is a communication tool designed to convey information in
an organized, presentable manner.
• Business Reports: These focus on business matters, drawing from internal
and external data sources, often using ETL (Extract, Transform, Load)
processes and reporting tools.

Purpose of Business Reports

1. Ensure proper functioning of departments.
2. Provide and analyze information.
3. Persuade action.
4. Create organizational memory (knowledge management).

Types of Business Reports

1. Informal: Routine, internal, up to 10 pages, uses personal tone.
2. Formal: Extensive, structured, research-based, for internal or external
audiences.
3. Short: Situational, periodic, investigative.

Business Performance Management (BPM)

• Synonyms: Corporate Performance Management (CPM), Enterprise
Performance Management (EPM), Strategic Enterprise Management
(SEM).
 • Purpose: Integrates methodologies, metrics, and applications to enhance
financial and operational performance.

Reporting Types in BPM

1. Metric Management Reports: Track KPIs and SLAs, linked to strategies like
Six Sigma or TQM.
2. Dashboard Reports: Visual summary of performance indicators, often color-
coded for attention.
3. Balanced Scorecard Reports: A comprehensive view of performance,
covering financial, customer, process, and growth perspectives.

Business Reporting Process Components

1. OLTP (Online Transaction Processing): Records real-world
transactions/events.
2. Data Supply: Transfers and processes recorded events.
3. ETL: Prepares data for reporting.
4. Data Storage: Organizes data in formats like RDBMS, data marts, or OLAP
cubes.
5. Business Logic: Converts data into actionable metrics and visuals.
6. Publication: Delivers reports in appropriate formats to users.
7. Assurance: Ensures data quality and timely delivery.

Dashboard vs Balanced Scorecard

Balanced Scorecard Versus Six Sigma

PERFORMANCE DASHBOARDS
Dashboards are integral to performance management and business intelligence (BI)
systems, offering visual displays of critical data on a single screen. Their purpose
is to allow quick digestion of information and facilitate deeper exploration where
needed.
❖ Three Layers of Information in Dashboards
1. Monitoring Layer
Provides real-time tracking of key performance indicators (KPIs) for an
immediate overview of performance.
o Key Features:
▪ Real-Time Data: Live metrics for activities like daily sales or stock
levels.
▪ High-Level Summaries: Simplified, snapshot views of essential
data.
▪ Alerts: Color-coded notifications or automated warnings for
threshold breaches.
Example: A customer service dashboard highlights in red if average call handling
time exceeds limits.
 2. Analysis Layer
Enables users to explore root causes of trends or anomalies in
performance.
o Key Features:
▪ Drill-Down Capabilities: From broad metrics to granular data
(e.g., sales by region or product category).
▪ Trend Analysis: Patterns over time, like monthly sales
fluctuations.
▪ Interactive Visualizations: Filters and charts for tailored insights.
▪ Comparative Insights: Benchmarks and historical comparisons.
Example: A marketing analyst examines campaign performance across channels
and timeframes to identify engagement drivers.

3. Management Layer
Transforms insights into actionable strategies and facilitates decision-
making.
o Key Features:
▪ Decision Support: Recommendations for operational
adjustments.
▪ Goal Setting: Tools to define and track progress toward
objectives.
▪ Scenario Planning: Exploration of potential outcomes for
informed decisions.
▪ Collaboration Tools: Shared insights and coordinated team
actions.
Example: A CEO evaluates revenue goals and reallocates budgets to higher-
performing marketing channels if needed.

❖ Dashboard Design Principles

• Clarity & Context: Present data without distractions, ensuring it is easily
interpretable.
• Comparative Analysis: Incorporate baseline, target, or trend comparisons to
contextualize metrics.
• Use of Widgets: Include specialized components to enhance visual
evaluation and focus on actionable data.
Dashboards should combine functionality with intuitive design, enabling businesses
to derive actionable insights and optimize performance effortlessly.

BUSINESS PERFORMANCE MANAGEMENT

• Definition and Terminology:

o Business Performance Management (BPM) is a term for processes,
methodologies, metrics, and technologies used to measure, monitor, and
manage business performance.
o Alternate names include Corporate Performance Management (CPM),
Enterprise Performance Management (EPM), and Strategic Enterprise
Management (SEM).
o CPM was coined by Gartner, while EPM is associated with Oracle.
• Key Components of BPM (Colbert, 2009):
o Integrated, closed-loop management and analytic processes that
address financial and operational activities.
o Tools for defining strategic goals and measuring performance against
them.
o Core processes such as financial/operational planning, consolidation,
reporting, modeling, analysis, and KPI monitoring linked to strategy.
• Closed-Loop BPM Cycle:
o A strategy-focused approach that links goal-setting, execution,
monitoring, and corrective actions in a continuous improvement cycle.
o Processes include:
 1. Strategize: Define mission, vision, and objectives to craft long-
term strategies.
2. Plan: Develop operational and financial plans to achieve strategic
goals.
3. Monitor/Analyze: Track performance using key performance
indicators (KPIs).
4. Act and Adjust: Take corrective actions to refine and improve
strategies.
• Detailed Processes in the BPM Cycle:
o Strategize:
1. Develop high-level strategies involving corporate executives and
board approval.
2. Strategy provides overall direction and aligns organizational
objectives.
o Plan:
1. Translate strategies into operational and financial plans.
2. Ensure integration between tactics, initiatives, and strategic goals.
3. Involve collaboration across departments and account for costs
and funding requirements.
o Monitor/Analyze:
1. Focus on key measures (KPIs) to track performance.
2. Establish strategies to monitor and respond to significant trends
effectively.
o Act and Adjust:
1. Identify changes to improve operations or achieve growth.
2. Address high failure rates in new projects with informed
adjustments.
• Statistics on Project Failures:
o Hollywood movies: ~60% failure rate.
o Mergers and acquisitions: ~60% failure rate.
o Large IT projects: ~70% failure rate.
o New food products: ~80% failure rate.
o New pharmaceutical products: ~90% failure rate.
PERFORMANCE MEASUREMENT
• Performance measurement systems help managers track strategy
implementation by comparing actual results with strategic goals and objectives
(Simons, 2002).
• Components:
o Goal Setting: Systematic methods to define business goals.
o Feedback Reports: Periodic updates to monitor progress against goals.
• Focus areas: Comparisons of strategies, goals, and objectives.
• Operate using Key Performance Indicators (KPIs): Metrics for performance
evaluation.

Key Performance Indicators (KPIs)

• Represent strategic objectives and evaluate performance against goals.
• Key Features:
1. Strategy: Embody strategic objectives.
2. Targets: Measure performance against specific goals.
3. Ranges: Define performance levels (e.g., above, on, or below target).
4. Encodings: Use visual indicators (e.g., green/yellow/red) to display
performance.
5. Time Frames: Assign deadlines with interim milestones.
6. Benchmarks: Compare performance to baselines or external standards.
• Types of KPIs:
1. Outcome KPIs (Lagging Indicators): Measure past activity outputs
(e.g., revenue).
2. Driver KPIs (Leading Indicators): Measure impactful activities on
outcomes (e.g., sales leads).
Examples of Operational Metrics
• Customer Performance: Satisfaction, issue resolution speed, retention.
• Service Performance: Call resolution rates, renewal rates, delivery
performance.
• Sales Operations: Pipeline accounts, meeting conversions, call closure
times.
• Sales Plan/Forecast: Accuracy ratios, fulfillment rates, closed contracts.
Examples of Industry-Specific KPIs
1. Retail Industry: Sales per square foot, customer retention rate, inventory
turnover.
2. Healthcare Industry: Patient satisfaction, average length of stay, bed
occupancy rate.
3. Manufacturing Industry: Overall equipment effectiveness, cycle time, defect
rate.
4. E-commerce Industry: Conversion rate, cart abandonment, customer lifetime
value.
5. Financial Services Industry: Net interest margin, loan default rate, return on
assets.
Performance Measurement vs. Performance Management
• Performance Measurement System: Tracks performance against goals.
• Performance Management System: Broader system incorporating
measurement to manage performance comprehensively.
• Popular framework: Balanced Scorecard (BSC) by Kaplan and Norton.

BALANCED SCORECARDS

Bulletin 1: Overview of the Balanced Scorecard (BSC)

• The Balanced Scorecard (BSC) is a widely recognized performance
management system developed by Kaplan and Norton in 1992.
• It integrates financial and nonfinancial measures to align an organization's
activities with its vision and strategy.
• BSC assesses organizational performance through four key perspectives:
 1. Customer – Focus on customer satisfaction as a leading indicator of
success.
2. Financial – Traditional financial data complemented by risk
assessments and cost-benefit analysis.
3. Internal Business Processes – Evaluates the efficiency and
effectiveness of internal operations.
4. Learning and Growth – Encourages continuous improvement,
employee development, and adaptability.

Bulletin 2: Key Perspectives of the Balanced Scorecard

1. The Customer Perspective
• Emphasizes customer satisfaction and the importance of meeting customer
needs.
• Poor customer satisfaction can predict future business decline despite strong
financial performance.
2. The Financial Perspective
• Stresses timely, accurate financial data while addressing the imbalance
caused by over-reliance on financial metrics.
• Recommends incorporating additional insights like risk assessments to
complement traditional measures.
3. The Learning and Growth Perspective
• Focuses on continuous improvement through employee training, knowledge
sharing, and fostering a supportive corporate culture.
• Encourages mentorship and streamlined communication to promote problem-
solving and innovation.
4. The Internal Business Process Perspective
• Assesses the quality of internal operations and ensures that products and
services meet or exceed customer expectations.

Bulletin 3: The Balanced Approach in BSC

• Balance in the BSC reflects a mix of:
o Financial and nonfinancial measures.
o Leading and lagging indicators.
o Internal and external factors.
 o Quantitative and qualitative metrics.
o Short-term and long-term objectives.
• The BSC bridges the gap between vision and actionable initiatives, making it a
holistic performance management tool.

SIX SIGMA AS A PERFORMANCE MEASUREMENT SYSTEM

Bulletin 1: Understanding Six Sigma
• Objective: Six Sigma focuses on measuring and monitoring key business
processes to enhance profitability and improve overall performance.
• Core Concept:
o It aims to reduce process variability, which is synonymous with defects in
quality terms.
o Defect rates in typical business processes range from 6,200 to 67,000
defects per million opportunities (DPMO), representing a 3- to 4-sigma
level of performance.
o Achieving Six Sigma equates to no more than 3.4 defects per million
opportunities, pushing processes closer to zero defects.
• Practical Example: An insurance company processing 1 million claims under
a 3- to 4-sigma level might see 6,200 to 67,000 mishandled claims. Achieving
Six Sigma would dramatically reduce this to only 3.4 defective claims.

Bulletin 2: DMAIC - The Performance Improvement Model

• Definition: DMAIC is a closed-loop, five-step process improvement
methodology within Six Sigma:
1. Define: Set goals and boundaries for improvement. Align these with
strategic objectives at higher levels or operational goals at
project/departmental levels.
2. Measure: Quantitatively evaluate the current system. Collect data to
monitor progress against defined goals.
3. Analyze: Identify performance gaps and root causes of inefficiencies in
processes.
4. Improve: Develop and implement actions to bridge performance gaps,
leveraging tools to enhance efficiency and cost-effectiveness.
5. Control: Ensure sustainability of improvements through updates to
policies, procedures, incentives, or systems.
 • Alternative for New Processes:
For entirely new systems, Six Sigma applies the DMADV model: Define,
Measure, Analyze, Design, and Verify.

Excel vs. Tableau

Parameter Excel Tableau

Organizes data in spreadsheets; Detailed analysis and creating

Primary Focus
allows analysis using formulas. interactive visualizations.

Business analysts and casual BI professionals and data

User Base
users. scientists.

Data Strong data processing Excels at creating interactive

Processing capabilities; limited visualization. dashboards for large datasets.

Basic to intermediate analytics

Analytics Advanced analytics with R
(e.g., pivot tables, VBA, Power
Capabilities and Python integration.
Pivot).

Stand-alone desktop application or

Available as desktop, server,
Deployment part of Microsoft Office 365 (online
and online versions.
version available).

Affordable as a stand-alone app or More expensive; subscription-

Cost Structure
part of Office 365. based pricing.

Ideal for quick, dynamic data

Best for data storage, organization,
Usage representation and
and statistical analysis.
storytelling.

Basic interface; experience Intuitive and user-friendly; no

User Interface
required for better usability. advanced coding needed.

Built-in features but less advanced Advanced and customizable

Security
than Tableau. security features.

Requires plug-ins to integrate with Seamlessly connects to real-

Data Sources
non-Excel sources. time data sources.

Descriptive tool focused on data Advanced visualization tool for

Key Highlights manipulation and validation merging and manipulating
through formulas. data.
Parameter Excel Tableau

Quick, visually appealing, and

Convenient for data transformation
Benefits interactive for analyzing big
and statistical analysis.
data.

Handles structured data well but Efficient and fast, even with
Performance
may lag with large datasets. large datasets.

Limited integrations (~60 Broad integration capabilities

Integration
applications). for enhanced multitasking.

Generates quick reports and Delivers analytics and

Business
dashboards for operational visualizations that optimize
Purpose
insights. business operations.

What is Data Visualization?

Data visualizations are graphic representations of data that help people understand
patterns and trends, identify relationships between variables, and spot outliers—
even in complex datasets. Visual elements like charts, graphs, and maps make
information accessible and easy to understand.
❖ Why use Data Visualization?
• Detect patterns
• Identify trends
• Reveal correlations
It can also prompt users to:
• Draw inferences
• Anticipate outcomes and trajectories
• Ask new questions about the data

Different Types of Charts and Graphs

Choosing the right visualization ensures your data story resonates effectively. Below
is a guide to 20 key visualization types, their uses, and best practices.
1. Line Charts
Connects data points with lines to showcase trends and patterns over time.
• Best for: Trends, progressions, project timelines.
 • Tips:
o Ensure logical data order.
o Add context via annotations and labels.
o Use transparency for large datasets.
2. Bar Charts
Represents data using rectangular bars, either horizontal or vertical.
• Best for: Comparing values across categories.
• Tips:
o Limit bars/categories to avoid clutter.
o Use colors to highlight key data.
3. Scatter Plots
Shows data points scattered across a graph to explore relationships.
• Best for: Correlations, trends, and clusters between two variables.
• Tips:
o Highlight outliers.
o Add trendlines to show relationships.
4. Pie Charts
Circular charts dividing data into slices to represent proportions.
• Best for: Datasets with fewer than 6 categories.
• Tips:
o Limit slices to maintain clarity.
o Use consistent colors.
5. Column Charts
Simple vertical bar charts for comparing data across categories.
• Best for: Rankings, trends, and order comparisons.
• Tips:
o Avoid 3D effects or excessive gridlines.
o Highlight specific data using contrasting colors.
6. Treemaps
Nested rectangles to represent hierarchical data and part-to-whole relationships.
 • Best for: Illustrating data composition.
• Tips:
o Ensure rectangle sizes are proportional.
o Label rectangles clearly.
7. Heatmaps
Uses color gradients to represent data intensity across a matrix.
• Best for: Visualizing correlations and patterns in grids.
• Tips:
o Choose intuitive color palettes.
o Highlight key patterns with visual cues.
8. Pareto Charts
Combines bar and line graphs to highlight the 80/20 principle (20% of causes create
80% of outcomes).
• Best for: Identifying the most significant factors.
• Tips:
o Order bars by impact or frequency.
o Use colors for clarity.
9. Geo Charts
Maps spatial data to illustrate regional trends or distributions.
• Best for: Geographic analysis and trends.
• Tips:
o Select a suitable map projection.
o Use color shading for emphasis.
10. Waterfall Charts
Depicts cumulative changes in values over time.
• Best for: Visualizing sequential changes and net results.
• Tips:
o Use color coding for positive/negative changes.
o Add labels for clarity.
11. Donut Charts
A variation of pie charts with a hole in the middle, suitable for multi-series data.
 • Best for: Showing contributions to a whole.
• Tips:
o Limit slices for clarity.
o Use labels for names and percentages.
12. Funnel Charts
Represents data reduction across stages, shaped like a funnel.
• Best for: Highlighting progressions or attrition.
• Tips:
o Use at least three stages.
o Highlight key reductions with annotations.
13. Bubble Charts
A scatter plot variation that uses bubble size to represent a third variable.
• Best for: Spotting patterns across three variables.
• Tips:
o Maintain consistent bubble sizes.
o Avoid excessive overlap by using spacing or transparency.
14. Histogram
Visualizes the distribution of a single continuous dataset in bins.
• Best for: Analyzing data spread and outliers.
• Tips:
o Use consistent bin sizes.
o Ensure enough data points for meaningful analysis.
15. Candlestick Charts
Used for financial data, showing price movements within a period.
• Best for: Analyzing stock or currency price trends.
• Tips:
o Use distinct colors for bullish/bearish patterns.
o Add annotations for key trends.
16. Area Charts
Fills the area beneath a line chart to emphasize cumulative values.
 • Best for: Comparing changes in values over time.
• Tips:
o Use a zero baseline for comparison.
o Avoid overcrowding with too many data series.
17. KPI Charts
Focuses on Key Performance Indicators to track progress.
• Best for: Monitoring critical business metrics.
• Tips:
o Focus on relevant datasets.
o Avoid overloading with too many KPIs.
18. Scatter Maps
Shows data points geographically, combining scatter plots and maps.
• Best for: Spotting spatial trends or outliers.
• Tips:
o Avoid overplotting densely populated areas.
o Use color coding for better interpretation.
19. Sankey Charts
Visualizes flows between categories with proportional arrows or bands.
• Best for: Depicting many-to-many relationships or processes.
• Tips:
o Use distinct colors for key flows.
o Proportion arrow width to flow quantity.
20. Radar Charts
Displays multi-dimensional data in a circular "web" format.
• Best for: Comparing multiple variables across categories.
• Tips:
o Avoid overlapping lines.
o Apply transparency to overlapping areas for clarity.
Visual analytics
• Visual analytics combines visualization with predictive analytics, contrasting
with information visualization, which focuses on presenting "what happened"
and "what is happening."
• Information visualization is closely linked to business intelligence, including
routine reports, scorecards, and dashboards.
• Visual analytics addresses questions like "why is it happening" and "what is
more likely to happen," aligning more with business analytics such as
forecasting, segmentation, and correlation analysis.
• Many information visualization vendors are expanding their capabilities to
provide visual analytics solutions.
• SAS Institute integrates advanced analytics into high-performance data
visualization, branding it as visual analytics.
• Business reporting, whether visual or not, automated or manual, resembles
storytelling in its essence.

Location-Based Analytics for Organizations.

• A consolidated view of organizational performance is often represented

through visualization tools that provide actionable insights, including current
and forecasted values of business factors and KPIs.
• Viewing KPIs as overall numbers via graphs and charts can be overwhelming,
leading to missed growth opportunities or unaddressed problem areas.
• Organizations utilize geographic visual maps based on traditional location
data, such as postal codes, to gain location-based insights. However, this
approach lacks granularity and may not identify specific regional growth
opportunities.
 • Static postal code-based geocoding offers an aggregated view of geographic
areas but fails to capture dynamic changes, such as shifting customer
locations or targeted promotional impacts.
• To overcome these limitations, organizations adopt location and spatial
analytics extensions, incorporating latitude and longitude data into traditional
business analyses to add a "where" dimension.
• Geographic Information Systems (GIS) integrate data from technologies like
GPS, RFID, and sensors to provide location intelligence (LI), offering critical
insights and enabling better decision-making.
• Location intelligence allows analysts to investigate new trends, pinpoint
patterns in sales and profitability, and correlate location-specific factors across
multiple KPIs.
• Retailers can assess sales variations by population levels, proximity to
competitors, and supply chain efficiency, while consumer product companies
identify customer-specific needs and trace complaints.
• USTRANSCOM uses location intelligence to monitor aircraft details, including
crew, equipment, and maintenance history, aiding in global operational
coordination.
• Geospatial data is now incorporated into enterprise data warehouses, allowing
organizations to perform complex in-database calculations for spatial analysis,
revealing hidden trends and opportunities.
• Advanced tools like Teradata's geospatial data features enable organizations
to represent geographic areas using geometric data types and geocoding
services like NAVTEQ and Tele Atlas for precise spatial mapping.
• Geospatial analytics is widely employed across industries, with applications
such as Sabre Airline Solutions' Traveler Security, which uses interactive maps
to monitor risks and assist travelers during disruptions.
• Location-based insights drive decisions such as site selection for business
expansion, optimizing processes, and uncovering new opportunities.

Recommendation Engines.
• Recommendation engines assist users in decision-making by aggregating
inputs, such as ratings, reviews, and behavioral data, to provide tailored
suggestions for products or services.
• Recommender systems operate as Web-based information filtering tools,
predicting user preferences based on gathered data.
 • Data collection for recommendation systems includes user ratings, item
rankings, textual feedback, time spent on items, and social networking
activities.
• Two main approaches to recommendation systems:
o Collaborative Filtering:
▪ Builds recommendations based on users' past behaviors, such as
purchase history and ratings.
▪ Employs user-item profiles grouped by shared preferences,
utilizing methods like matrix factorization to predict unranked item
ratings.
▪ Includes user-based filtering (grouping similar users) and item-
based filtering (grouping items with similar ratings).
▪ Challenges include the "cold start" problem and high
computational requirements.
o Content-Based Filtering:
▪ Focuses on item characteristics and creates profiles for users
based on their past preferences.
▪ Employs techniques like Bayesian classifiers, cluster analysis,
decision trees, and neural networks to recommend items with
similar traits.
• Collaborative filtering is prevalent in e-commerce platforms (e.g., Amazon,
Facebook, LinkedIn) for recommending books, movies, or other products
based on user behavior.
• Content-based filtering is commonly used for textual recommendations (e.g.,
news articles, music, movies), with applications like Pandora and Patients Like
Me employing this technique.
• The effectiveness of recommendation systems depends on their ability to
balance user behavior insights and item-specific attributes.

The Web 2.0 Revolution and Online Social Networking.

• Web 2.0 Overview:
Web 2.0 encompasses advanced web technologies like blogs, wikis, RSS,
mashups, user-generated content, and social networks. It focuses on
enhancing creativity, information sharing, and collaboration, distinguishing
itself from the traditional Web by fostering greater collaboration among users,
content providers, and enterprises.
• Key Features of Web 2.0:
 o Harnesses collective intelligence, increasing site value with user
contributions.
o Supports data remixing ("mashups") through web service interfaces.
o Emphasizes user-generated and user-controlled content.
o Enables easy web development with lightweight programming tools.
o Eliminates software-upgrade cycles, promoting perpetual beta states
and rapid prototyping.
o Allows access to applications through browsers.
o Encourages digital democracy with participatory architectures.
o Prioritizes social networks, information sharing, and continuous business
model innovation.
• Technology and Applications:
Web 2.0 relies on dynamic content, rich user interfaces, metadata, scalability,
open-source platforms, and net neutrality. Techniques like Ajax enable
responsive, interactive web applications by exchanging small data packets
without reloading entire pages. Successful Web 2.0 ideas quickly spread
globally, with sites like Twitter having numerous local adaptations.
• Social Networking:
Social networking facilitates sharing and connections, creating a more open
and connected world.
o Members use personal spaces to post blogs, photos, videos, and music
while tagging content with keywords for searchability.
o Mobile social networking extends connectivity to those without regular
computer access through smartphones and apps.
• Notable Platforms:
o Facebook: Launched in 2004, it grew rapidly due to the "network effect"
and offers features like photos, groups, events, and News Feed updates.
o Orkut: Developed by Google, popular in countries like Brazil but less so
in the United States.
o Google+: Leveraged Gmail integration but remained smaller compared
to Facebook.
• Business and Enterprise Applications:
o Social networks enable advertising, sales, and commercial activities,
particularly in business-oriented platforms like LinkedIn.
 o Twitter: Offers real-time market insights by analyzing "tweets."
Businesses use social media analytics tools to monitor and manage
brand reputations and customer feedback.

Impacts of Analytics in Organizations: An Overview.

• Cultural Transformation: Analytics is a crucial factor in the rapidly evolving
information and knowledge revolution, significantly impacting managerial,
economic, and social spheres.
• Integration Challenges: Separating analytics' impacts from other
computerized systems is challenging due to increasing integration with other
information systems, influencing individuals, departments, and organizations
on both micro and macro levels.
• New Organizational Units: Companies are creating analytics-focused
departments such as BI, decision support, and knowledge management units.
Examples include Target and Walmart leveraging data for marketing and
supply chain efficiencies.
• Industry Growth: The BI industry's expansion has led to the formation of
specialized business units within companies, collaborations, and acquisitions,
such as IBM acquiring Demandtec and SAS collaborating with Teradata.
• Restructuring Business Processes: Implementing analytics often requires
restructuring processes, with large-scale changes sometimes categorized as
business process reengineering (BPR).
• Impacts of ADS Systems:
o Reduction in middle management roles.
o Customer and business partner empowerment.
o Improved customer service and help desk productivity.
o Transformation of industries through profitability models and
optimization.
• Job Satisfaction: Analytics can enrich roles by automating routine tasks,
enabling managers to focus on strategic and creative endeavors, thereby
increasing job satisfaction.
• Job Stress and Anxiety: Increased workloads, connectivity demands, and
information overload can create stress, necessitating management
intervention through workload redistribution and training.
• Managerial Activities and Performance:
o Enhanced decision-making speed and quality due to data availability
and automation.
 o Managers gain time for planning and fieldwork, benefiting from alerts
and decision-support systems.
o Strategic management is improved with tools like data mining and
intelligent agents.
o Managers handle fewer tasks daily but achieve higher productivity.
o Leadership dynamics may shift with the rise of virtual communication
platforms.
• Potential Managerial Job Impacts:
o Reduced reliance on expertise for decisions due to automation and
intelligent systems.
o Faster and higher-quality decision-making processes.
o Redistribution of power based on access to analytical capabilities.
o Middle management roles are the most susceptible to automation.
o Lower-level managers' jobs are partially automatable, focusing more on
behavioral and supervisory tasks.
o Top management roles remain the least routine and least automatable.
Analytics fundamentally reshapes organizational structures, employee roles, and
decision-making processes, with transformative effects across industries and
societal dynamics.

Issues of Legality, Privacy, and Ethics.

Legal Issues
• Analytics introduces complex legal challenges, including liability concerns for
decisions based on intelligent machines.
• Specific questions include:
o Legal validity of expert opinions encoded in computers.
o Liability for incorrect advice from intelligent applications (e.g., medical
misdiagnoses leading to harm).
o Responsibility for errors or disasters caused by misjudged analytic
inputs.
o Ownership of knowledge in knowledge bases.
o The right to enforce expertise contributions.
Privacy
 • Privacy is a broad concept that balances individual rights against societal
needs.
• Key rules influencing privacy decisions:
o Privacy rights are not absolute and must be balanced with societal
interests.
o The public's right to know often supersedes individual privacy.
Challenges in Privacy
• Data Collection: Internet and databases simplify accessing private
information, raising risks for misuse.
• Mobile User Privacy: Devices and networks track users' movements and
activities, sparking concerns over data usage and ownership.
• Government Surveillance: Analytics tools for security, such as mining
personal data or monitoring phone calls, often compromise privacy.
• Corporate Practices: Organizations monetize user data to profile
preferences, raising privacy violation risks.
Recent Privacy Concerns
• Companies like Rapleaf and Bluecava use analytics to create detailed user
profiles, sometimes leading to unauthorized data collection.
• Technologies like employee behavior tracking via sensors raise ethical
questions about workplace surveillance.
Ethics in Decision Making and Support
• Ethical challenges in analytics include:
o Electronic surveillance.
o Invasion of privacy.
o Software piracy and intellectual property misuse.
o Data and information accuracy.
o Employee exposure to unsafe environments.
o Accessibility concerns for users with disabilities.
Ethical Frameworks
• Ethical reasoning involves evaluating:
o The agent and action taken.
o Consequences and fairness for stakeholders.
• Ethical judgments are guided by rules, principles, and underlying theories.
The Analytics Ecosystem

• Analytics Ecosystem Overview

o The analytics industry presents various opportunities across multiple
sectors, offering roles in descriptive, predictive, and prescriptive
analytics.
o The industry comprises nine key sectors, with technology providers
being the primary group. These organizations develop technologies and
solutions, enabling effective analytics for user organizations.
o Analytics professionals have the chance to move across sectors, as
many companies operate in multiple areas within the ecosystem.
• Analytics Industry Clusters
o Companies are grouped into clusters based on their roles in the
analytics space, with technology providers forming the core group. The
sector list provided is not exhaustive and aims to illustrate the diversity
of opportunities.
o Companies within each sector may operate in multiple roles, providing
opportunities for both horizontal and vertical career movement.
• Data Infrastructure Providers
o This sector includes major players in data hardware and software,
offering the foundational infrastructure for data management solutions.
o Key players include hardware providers like IBM, Dell, HP, and Oracle,
along with storage solution providers such as EMC and NetApp.
o Many companies offer both hardware and software platforms, while
others, like Microsoft’s SQL Server, provide software solutions that are
hardware-independent.
 o Emerging players in cloud computing, such as Amazon and Salesforce,
provide data storage and related services via the cloud.
o Big Data companies like Cloudera and Hortonworks focus on
infrastructure services and training to build Big Data platforms, such as
Hadoop clusters and NoSQL technologies.
• Data Warehouse Industry: Companies in this group focus on integrating data
from multiple sources to provide value from data assets. They often include
their own hardware for efficient data storage, retrieval, and processing. Recent
trends involve performing analytics directly in memory. Major players include
IBM, Oracle, and Teradata, all of which offer academic alliance programs to
provide students with access to data warehousing software.
• Middleware Industry: Initially focused on bringing together data from various
stores, this industry now provides easy-to-use tools for reporting and analytics.
Major players include MicroStrategy, Plum, and acquired companies like
Hyperion (by Oracle), SAP (acquiring Business Objects), and IBM (acquiring
Cognos). These companies focus on descriptive analytics and reporting as
core components of Business Intelligence (BI) and analytics.
• Data Aggregators/Distributors: Companies in this category specialize in
collecting, aggregating, and distributing data within specific industry sectors.
Notable examples include Nielsen (retail purchase behavior), Experian
(household data), Omniture (Web clicks), and Google Analytics (website data).
Many niche platforms exist to serve specific data collection needs across
various domains.
• Analytics-Focused Software Developers: These companies develop
general-purpose analytics software for data collected in data warehouses or
from Big Data platforms. Examples of industry players include software
providers for reporting, predictive analytics, and prescriptive analytics. Open
source platforms like R and RapidMiner have gained popularity in predictive
analytics, while companies like Alteryx provide process sharing and analytics
solutions.
• Reporting/Analytics: This category involves developing various reports,
queries, and visualizations for business analysis. Tools include general
visualizations, dashboards, and industry-specific platforms. Companies like
Microsoft (SQL Server BI), Tableau, and SAS provide reporting and
visualization tools, along with many open-source alternatives for data
visualization.
• Predictive Analytics: This rapidly growing field uses statistical methods and
data mining techniques for forecasting. Key software providers include SAS
(Enterprise Miner), IBM (SPSS Modeler), KXEN, and Statsoft. Open-source
platforms like R and RapidMiner are widely used in predictive analytics, and
companies like Alteryx focus on process-sharing capabilities to improve
predictive analytics workflows.
• Prescriptive Analytics: This segment offers modeling tools and algorithms for
operational optimization, typically within management science and operations
research. Providers include IBM (ILOG), SAS (SAS/OR), and FICO
(XPRESS), as well as companies like AMPL, Gurobi, and Frontline. These
 tools support decision-making in optimization, simulation, and multiobjective
analysis, with new companies combining Big Data and optimization
techniques, such as Teradata Aster.

• Application Developers or System Integrators: Industry-Specific or

General
o This group focuses on creating custom solutions using data
infrastructure, middleware, data aggregators, and analytics software for
specific industries or users.
o They bridge the gap between analytics technology and its practical
application, ensuring that solutions are useful in real-world scenarios.
o Companies like IBM, SAS, and Teradata often have significant
consulting practices, leveraging analytics tools to help clients.
o Many analytics technology providers develop industry-specific solutions,
such as Cerner for healthcare and Sabre Technologies for travel.
o Start-ups also contribute by creating domain-specific analytics solutions,
such as FICO for credit scoring and Sense Networks for location-based
analytics.
o The rise of analytics in web, social media, and location-based services
has given birth to many start-ups offering targeted promotional solutions,
like Shazam, Siri, and Simulmedia.
o This sector represents one of the biggest growth areas in analytics due
to technological advances and funding opportunities.
• Analytics User Organizations
o Analytics user organizations represent the backbone of the entire
analytics industry as they directly apply analytics in their operations,
across sectors like private businesses, government, education, and
military.
o These organizations recognize the importance of analytics in gaining
competitive advantages and improving operational efficiency.
o Leadership in these organizations, particularly in information technology
roles (like Chief Information Officer), plays a key role in implementing
analytics applications.
o Companies use analytics to reduce costs, optimize processes, and
develop strategies for fraud prevention and customer wellness, as seen
in healthcare organizations.
 o The demand for analytics professionals is growing, with roles ranging
from analysts to project managers, senior managers, and even
executives like Chief Executive Officer.
o User organizations are vital to the analytics ecosystem, providing
numerous career paths for professionals and creating an environment
where analytics can flourish.
Analytics Industry Analysts and Influencers
• Professional Organizations: These provide advice to analytics industry
providers and users. Their services include marketing analyses, coverage of
new developments, evaluation of technologies, and development of
training/white papers. Examples include:
o Gartner Group
o The Data Warehousing Institute
o General and technical publications/websites covering the analytics
industry
• Professional Societies/Organizations: These provide similar services but
are membership-based and organized. They promote analytics through
publications, conferences, and user groups. Examples include:
o INFORMS (focuses on promoting analytics)
o Special Interest Group on Decision Support Systems (SIGDSS) under
the Association for Information Systems
o Membership-based user groups from major vendors like Teradata and
SAS
• Analytics Ambassadors/Influencers/Evangelists: Individuals who promote
analytics through seminars, books, and other publications. Notable figures
include:
o Steve Baker
o Tom Davenport
o Charles Duhigg
o Wayne Eckerson
o Bill Franks
o Malcolm Gladwell
o Claudia Imhoff
o Bill Inmon
 o Others who have written books and/or given presentations to promote
analytics applications
o Authors of textbooks on business intelligence/analytics, including those
who aim to train the next generation of analytics professionals
Academic Providers and Certification Agencies
• Universities and Academic Programs: Universities play a crucial role in
preparing professionals for the analytics industry. This includes:
o Business schools (information systems, marketing, management
sciences)
o Other disciplines such as computer science, statistics, mathematics, and
industrial engineering
o Graphics developers designing new ways to visualize information
o Undergraduate and graduate programs in analytics (often labeled
differently across disciplines)
• Certificate Programs: These programs allow current professionals to retrain
and gain proficiency in specific software. Examples include:
o Graduate programs in analytics (often growing and evolving)
o Shorter certificate programs for practicing analysts
• Certification Programs: These provide certificates of expertise in specific
software, ensuring new hires possess essential tool skills. Examples include:
o Certification programs from major technology providers (IBM, Microsoft,
MicroStrategy, Oracle, SAS, Teradata)
o Certified Analytics Professional (CAP) program from INFORMS, aimed
at testing general analytics competency
• Growth of Analytics Programs: The number of academic programs in
analytics is rapidly growing. Although the demand appears to be outstripping
supply, it remains to be seen if the industry can absorb the increasing number
of qualified graduates.

What is Tableau and its Importance in Data Visualization?

• Data generated globally is often messy, complex, and unclear, making it
difficult to analyze and understand. To extract insights, this data needs to be
cleaned, organized, and visualized.
• Tableau is a powerful data visualization tool that helps in cleaning, shaping,
and visualizing data, enabling users to gain insights through interactive
visualizations.
• Widely used by data analysts, scientists, and statisticians, Tableau offers quick
and efficient data visualization, allowing companies to make informed
decisions based on data analysis.
• It also provides a secure platform for managing and analyzing data,
addressing any security issues promptly.
• Tableau supports various data types and formats, and can be used for
individual analysis or scaled to suit business teams and organizations. It is
used by prominent organizations like Amazon, Lenovo, and Walmart.
• Products Offered by Tableau:
• Tableau Desktop
• A desktop application for creating interactive data visualizations, charts, and
dashboards.
• Connects to data from various sources such as SQL databases,
spreadsheets, Google Analytics, and more.
• Available as part of the Tableau Creator package for 70 USD per month with a
14-day free trial.
• Tableau Public
• A free version used for creating public data visualizations and charts that can
be embedded in blogs or shared on social media.
• Data visualizations created are publicly accessible, making it ideal for
students, hobbyists, journalists, and bloggers.
• Tableau Online
• A cloud-hosted service for creating and sharing data visualizations,
storyboards, and charts.
• Offers ease of use without concerns over software upgrades or hardware
scaling.
• Available as part of the Tableau Creator package for 70 USD per month with a
14-day free trial.
• Tableau Server
• A server product installed on Windows or Linux servers, widely used in
industries for secure data visualization and exploration.
• Integrates with existing security protocols like Kerberos, Active Directory, and
OAuth.
• Available as part of the Tableau Creator package for 70 USD per month with a
14-day free trial.
• Advantages of Tableau:
• Create Great Visualizations
• Tableau enables users to clean, combine, and shape data, turning messy data
into clear and detailed visualizations such as charts and dashboards.
• Obtain Detailed Insights
• By exploring data from various angles and asking open-ended questions,
Tableau helps uncover patterns and insights, especially with real-time data.
• User-friendly Approach
• Designed for users with little technical expertise, Tableau’s drag-and-drop
interface allows easy creation of complex visualizations without coding.
• Support for Different Data Sources
• Tableau can connect to a wide range of data sources, including cloud storage,
spreadsheets, and big data, enabling users to blend and visualize disparate
data types seamlessly.
• Tableau data types:
o Text (String) Values:
Represents sequences of characters. These values can include letters,
numbers, and symbols, typically used to store words, sentences, or any
alphanumeric characters. Example: "John Doe", "Product Name".
o Date Values:
Represent specific points in time, usually formatted as day/month/year or
year/month/day, depending on the system. Used for tracking time-related data
such as birth dates, event dates, etc. Example: "2024-11-25".
o Date & Time Values:
Includes both date and time in a specific format to represent timestamps,
capturing not only the day but also the exact time. Example: "2024-11-25
14:30:00".
o Numerical Values:
Used for quantitative data and are represented as integers or floating-point
numbers. These are useful for performing mathematical operations and
analysis. Example: 100, 56.75.
o Boolean Values (Relational Only):
Represent logical binary values, typically used to indicate truth (True) or
falsehood (False). They are commonly used in relational databases for
conditional checks. Example: True, False.
o Geographic Values (Used with Maps):
Used to represent location-based data that can be plotted on maps, often
including latitude and longitude coordinates. Example: "Latitude: 40.7128,
Longitude: -74.0060".
 o Image Role (Used with Image Link URLs):
Refers to the purpose or context of an image in a dataset, typically
represented by a URL pointing to the image. This is useful for datasets
involving visual content or links to images.
o Cluster Group (Used with Find Clusters in Data):
A classification or grouping of similar data points based on certain attributes,
often used in machine learning or data mining for pattern recognition. This
type organizes data into clusters or categories to find relationships. Example:
Cluster 1, Cluster 2.

Row & Column oriented organization of data warehouse

Feature Row-oriented Organization Column-oriented Organization

Data is stored row by row, Data is stored column by

Data Storage meaning each row is stored column, meaning each column is
together. stored together.

Best suited for transactional Best suited for analytical queries

Query
queries where entire rows are that involve reading specific
Performance
needed. columns.

Optimized for Online

Optimized for Online Analytical
Use Case Transaction Processing
Processing (OLAP) systems.
(OLTP) systems.

Slower retrieval of large Faster retrieval of specific

amounts of data, especially columns in large datasets, as
Data Retrieval
when only a few columns are only relevant columns are
required. accessed.

Less efficient for data More efficient for data

Storage
compression, as rows may compression, as similar data
Efficiency
contain different types of data. types are stored together.

Update and Faster for frequent updates Slower for updates and inserts,
Insert and inserts, as rows can be as updating individual columns
Performance added or updated directly. can be more complex.

Faster for aggregation queries,

Slower for aggregation
Data as entire columns are stored
queries, as more data needs
Aggregation together and can be processed
to be scanned.
quickly.
Feature Row-oriented Organization Column-oriented Organization

Columnar databases like

Relational databases like
Example Apache Cassandra, Google
MySQL, PostgreSQL.
BigQuery, and Amazon Redshift.

Define BI
• Business Intelligence is a set of concepts and methodologies aimed at
improving decision-making in business through the use of facts and fact-based
systems.
• The goal of Business Intelligence is to enhance decision-making through
analysis and business ideas. It is a collection of concepts and methodologies,
combining analytics and gut feelings for decision-making.
• Business Intelligence involves mathematical models and analysis methods
that use data to generate valuable insights, aiding in making important
decisions.

What are the steps for successful BI implementation

• Real-time data tracking: In the volatile modern world, waiting for reports can
result in irrelevant information. BI implementation provides instant access to
critical data, allowing for proactive decision-making rather than reactive
responses.
• Data harmonization: Fragmented data across departments often prevents a
complete view. Business intelligence creates a single version of the truth,
essential for data-driven decisions.
 • Enhanced visibility: Rather than sifting through pages of statistics, BI gives
you a clear view of organizational workflows, helping to identify areas needing
attention.
• Efficiency of your organization: Identifying problem areas is just the first
step. BI helps optimize operations by improving employee performance,
eliminating redundancies, and assessing the impact of actions to fine-tune
strategies.
• Deeper customer insights: Understanding customer behavior and
purchasing patterns is key to improving products and customer service,
leading to better customer relationships.
• Consistent sales insights: With access to both real-time and historical sales
data, BI helps spot trends, predict sector success, and uncover ways to boost
revenue.
• A competitive edge: BI allows you to compare internal metrics with
competitors' performance, helping you identify gaps and areas for
improvement, enabling you to plan strategies to win customers from
competitors.

What are the benefits and challenges of BI implementation?

Benefits of BI Implementation:
• Real-time Data Tracking: BI tools allow businesses to track data in real-time,
providing immediate access to insights and trends. This helps organizations
react quickly to market changes or operational needs.
• Deeper Customer Insights: With BI tools, companies can analyze customer
behavior and preferences in detail, allowing for more tailored marketing
strategies and improved customer experiences.
• Consistent Sale Insights: BI systems enable continuous analysis of sales
performance, ensuring that decision-makers have consistent and accurate
sales insights. This helps with better forecasting and planning.
• Data Harmonization: BI systems integrate data from various sources and
formats, creating a unified dataset. This eliminates silos and ensures a
consistent, reliable data stream for decision-making.
• Enhanced Visibility: BI systems provide enhanced visibility into various
business processes. This transparency aids in identifying inefficiencies,
optimizing workflows, and improving overall business performance.
• A Competitive Edge: Implementing BI tools gives businesses a competitive
advantage by providing actionable insights, enabling better decision-making,
and fostering a data-driven culture.
 • Boosted Efficiency of Your Organization: BI tools automate data collection
and reporting, reducing manual work and errors, which leads to higher
operational efficiency and faster decision-making.
Challenges of BI implementation
• Data integration
It's crucial to avoid creating a disorganized data silo from various sources with
unstructured data in different formats. Poor data quality is another challenge.
To address this, use proper data integration middleware and BI tools. Adopting
data governance practices, like multi-level access, can help maintain data
quality.
• Resource allocation and costs
Initial BI implementation requires purchasing software, possibly hardware, and
hiring experts. Ongoing costs include maintenance, system improvements,
and training personnel.
To manage costs, implement the BI system in phases, evaluating the ROI at
each stage. Consider using cloud BI solutions to reduce infrastructure
expenses.
• User adoption
Not everyone is ready to use new advanced tools. BI systems require skills
like expert data input, processing, and documentation management.
To address this adoption gap, introduce new tools gradually, providing custom
guides, instructions, workshops, and live mentor support from staff.
• Complexity and scalability
BI systems are often technically complex and can become bottlenecks when
scaling if not properly planned.
Choose scalable BI solutions from the beginning, employ experienced tech
leads, or consult with senior professionals before scaling.
• Security and privacy
Handling sensitive corporate and user data requires strong protection against
unauthorized access, breaches, and regulatory violations.
Use multi-layered data access controls, data encryption, resilient cybersecurity
software, and regular security audits to safeguard valuable data.

List and describe the major components of BI./ Briefly describe the
architecture of BI with an appropriate diagram.
A Business Intelligence (BI) system is a framework for gathering, storing, analyzing,
and presenting business data to help organizations make informed decisions. It has
four major components:
1. Data Warehouse
The data warehouse is the centralized repository for storing large amounts of
structured data from various sources. It integrates data from operational systems,
external sources, and transactional databases. This component ensures data is
clean, consistent, and readily available for analysis. Key elements include:
• Source Data: The raw data collected from different operational systems (e.g.,
sales, finance, customer service), external data providers, and other sources.
• Data Integration: Data is consolidated, cleaned, transformed, and loaded into
the warehouse through processes like Extract, Transform, and Load (ETL).
• Data Storage: Data is organized in a way that allows for fast querying and
analysis. It might use techniques like data cubes, star schemas, or snowflake
schemas to optimize storage and retrieval.
• Historical Data: Data warehouses often store large volumes of historical data
for trend analysis, forecasting, and reporting.
2. Business Analytics
Business analytics involves tools and techniques used to analyze the data stored in
the data warehouse. This component includes:
• Data Manipulation: Tools for cleaning, structuring, and transforming raw data
into usable formats. Data might need to be aggregated, joined, or filtered
based on the specific needs of analysis.
• Data Mining: The use of statistical algorithms, machine learning models, and
pattern recognition to discover insights from the data. This can help in
identifying trends, customer behaviors, or anomalies that may not be
immediately obvious.
• Predictive Analytics: Using historical data to make predictions about future
trends. This involves techniques like regression analysis, time-series
forecasting, and other modeling techniques.
• Descriptive Analytics: Summarizing past data and events to understand
trends and patterns. This can include generating reports, charts, and
dashboards for easy interpretation.
• Diagnostic Analytics: Investigating past performance to understand the
reasons behind certain outcomes. It helps to identify what worked well or
poorly and why.
3. Business Performance Management (BPM)
BPM refers to monitoring, measuring, and managing business performance through
key metrics and goals. It ensures that the organization is meeting its objectives and
that strategies are on track. Key aspects of BPM include:
 • Key Performance Indicators (KPIs): Metrics used to evaluate the success of
an organization in achieving its business objectives. KPIs are typically linked
to business goals like revenue growth, customer satisfaction, and operational
efficiency.
• Performance Dashboards: Visual tools that provide real-time tracking of
business performance against KPIs. Dashboards are often customizable,
allowing users to focus on the most important metrics.
• Balanced Scorecard: A strategic planning and performance management
system used to align business activities to the vision and strategy of the
organization. It provides a comprehensive view of performance from multiple
perspectives, such as financial, customer, internal processes, and learning &
growth.
• Benchmarking: Comparing an organization's performance against industry
standards or best practices to identify areas for improvement.
4. User Interface (UI) - Dashboard
The user interface, often in the form of dashboards or reporting tools, provides a
means for business users to interact with the data and insights generated by the BI
system. Key components include:
• Dashboards: These are visual representations of data, designed to present
key insights and KPIs in an easily digestible format. Dashboards often include
charts, graphs, tables, and maps. They can be customized based on the
user's role or preferences.
• Reporting Tools: These tools allow users to generate detailed reports based
on predefined queries or templates. Reports can be static or dynamic, and
they may include detailed visualizations, trend analyses, and forecasts.
• Self-Service Analytics: Many modern BI systems allow business users to
explore data on their own without requiring technical expertise. This is
achieved through intuitive drag-and-drop interfaces, prebuilt visualizations,
and guided analysis features.
• Data Exploration Tools: These allow users to interact with data, drill down
into specifics, and generate ad-hoc reports. It helps non-technical users
explore data and uncover insights independently.
Interaction Among Components
• Data Warehouse and Business Analytics: The data warehouse provides the
foundational data that is analyzed using business analytics tools. Data
manipulation and mining techniques transform raw data into valuable insights.
 • Business Analytics and BPM: The insights generated through business
analytics are often used in BPM for tracking performance against KPIs,
monitoring trends, and making data-driven decisions.
• User Interface and BPM: Dashboards and reporting tools in the UI present
performance metrics, KPIs, and insights, helping users monitor business
health and make informed decisions in real time.
• User Interface and Business Analytics: The user interface allows users to
interact with data, create visualizations, and explore analytics results to draw
actionable insights.
Together, these four components form the backbone of a modern Business
Intelligence system, enabling organizations to make data-driven decisions that
optimize operations, improve performance, and drive strategic growth.

List the benefits of BI

1. Improved Decision Making: BI tools provide accurate and timely information,
enabling informed decision-making at all levels of an organization.
2. Increased Efficiency and Productivity: BI streamlines processes and
automates repetitive tasks, leading to time and cost savings while improving
productivity.
3. Better Customer Insights: BI helps organizations analyze customer data to
understand behavior patterns, preferences, and trends, allowing for
personalized services and improved customer experiences.
4. Competitive Advantage: By analyzing data, businesses can identify market
trends, competitors' strategies, and emerging opportunities, giving them an
edge over competitors.
 5. Cost Reduction: BI helps organizations identify inefficiencies, optimize
operations, and reduce costs by providing insights into areas that require
improvement.
6. Data-Driven Culture: BI fosters a data-driven decision-making culture,
reducing reliance on gut feeling or intuition by focusing on data and evidence.
7. Increased Revenue: Through accurate forecasting and trend analysis, BI
enables organizations to identify new revenue opportunities, optimize pricing
strategies, and increase sales.
8. Risk Management: BI tools can help identify potential risks and fraud by
analyzing patterns in business processes, enabling proactive risk
management strategies.
9. Better Collaboration and Communication: BI tools make data more
accessible and understandable, promoting collaboration across departments
and teams.

Identify some typical applications of BI

1. Financial Management: Analyzing financial data for budgeting, forecasting,
and financial reporting.
2. Customer Relationship Management (CRM): Using BI tools to track
customer behavior, segment customers, and predict future sales.
3. Supply Chain Management: Monitoring inventory, procurement, and logistics
to improve supply chain efficiency and reduce costs.
4. Human Resources (HR) Analytics: Analyzing employee performance,
turnover, and recruitment data to optimize workforce management.
5. Marketing and Sales Analytics: Analyzing sales data, customer
demographics, and marketing campaigns to increase sales and improve
customer targeting.
6. Healthcare Analytics: Tracking patient data, operational efficiency, and
treatment outcomes to improve healthcare delivery and reduce costs.
7. Retail Analytics: Analyzing customer buying patterns, inventory levels, and
market trends to optimize product placement and pricing strategies.
8. Operations Management: Monitoring production processes, resource
utilization, and operational performance to improve efficiency.
List the steps of intelligence creation and use
1. Data Collection: Gathering raw data from various sources, such as internal
systems, sensors, social media, and external databases.
2. Data Integration: Combining data from different sources and formats into a
single, cohesive structure for analysis.
3. Data Cleaning and Transformation: Ensuring data quality by cleaning errors,
removing duplicates, and transforming data into a consistent format.
4. Data Analysis: Using statistical methods, machine learning algorithms, and BI
tools to identify trends, patterns, and insights.
5. Decision Making: Utilizing the insights generated from data analysis to make
informed decisions and guide strategic actions.
6. Action and Monitoring: Implementing decisions and continuously monitoring
results to assess effectiveness and adjust strategies accordingly.

What is BI Governance?
BI Governance refers to the framework and policies that ensure the proper
management, quality, security, and ethical use of data and BI tools within an
organization. It involves establishing guidelines for data access, usage, quality
assurance, privacy, and compliance with regulations. Effective BI governance
ensures that data and insights are used responsibly, consistently, and securely
across the organization.
Key aspects of BI governance include:
1. Data Quality Management: Ensuring data accuracy, consistency, and
reliability.
2. Data Security: Protecting sensitive and confidential data from unauthorized
access.
3. Data Access Control: Defining roles and permissions for users accessing BI
tools and data.
4. Compliance: Ensuring that BI practices comply with legal and regulatory
standards.
5. Ethical Use of Data: Establishing policies to ensure that data is used ethically,
avoiding misuse or manipulation.
Difference Between Transaction Processing and Analytical Processing

Transaction Processing
Aspect Analytical Processing (OLAP)
(OLTP)

To handle day-to-day To analyze and report on historical

Purpose
operations and transactions. data for decision-making.

Deals with current, operational Deals with historical data, often

Data Type
data (real-time). aggregated.

Simple queries and fast

Complex queries involving large
Complexity updates to small amounts of
datasets.
data.

Optimized for read-heavy

High transaction speed, low
Performance operations and complex
latency, and high volume.
computations.

Relational databases
Data Multidimensional databases (cubes)
optimized for quick
Structure optimized for analytical queries.
transactions.

Operational staff and front-line Analysts, decision-makers, and

Users
employees. managers.

Order processing, inventory Sales forecasting, trend analysis,

Examples
management. market research.

Response Very fast, real-time Slower, optimized for deep analysis

Time processing. and reporting.

Insert, update, delete Select, aggregate, drill-down

Operations
operations. operations.

Define Data Analytics and discuss it various types?

Data Analytics is the process of examining raw data to uncover patterns, trends,
relationships, and useful information that can aid in decision-making. It involves
using statistical, computational, and logical techniques to extract meaningful insights
from data.
Types of Data Analytics:
1. Descriptive Analytics:
o Focus: What happened?
o Purpose: Summarizes past data to describe trends or patterns.
 o Techniques: Reports, dashboards, and data visualizations.
o Example: Monthly sales reports.
2. Diagnostic Analytics:
o Focus: Why did it happen?
o Purpose: Identifies the cause or reasons behind past events.
o Techniques: Drill-down analysis, correlation analysis.
o Example: Analyzing a drop in sales due to customer behavior or market
changes.
3. Predictive Analytics:
o Focus: What is likely to happen?
o Purpose: Uses historical data to predict future outcomes or trends.
o Techniques: Machine learning, regression analysis, time-series
forecasting.
o Example: Predicting future sales or customer churn rates.
4. Prescriptive Analytics:
o Focus: What should be done?
o Purpose: Suggests actions to optimize outcomes or improve
performance.
o Techniques: Optimization algorithms, simulations.
o Example: Recommending the best marketing strategy to increase sales.
5. Cognitive Analytics:
o Focus: What will be the impact?
o Purpose: Emulates human decision-making and reasoning to create
intelligent systems.
o Techniques: Artificial intelligence, deep learning, natural language
processing (NLP).
o Example: Chatbots that provide customer service based on previous
interactions.
Each type of data analytics serves a different purpose but works together to enable
businesses to gain insights, make decisions, and optimize operations.