AD8551 BUSINESS ANALYTICS LTPC 3003

OBJECTIVES:
1. To understand the Analytics Life Cycle.
2. To comprehend the process of acquiring Business Intelligence
3. To understand various types of analytics for Business Forecasting
4. To model the supply chain management for Analytics.
5. To apply analytics for different functions of a business

UNIT I INTRODUCTION TO BUSINESS ANALYTICS 9

Analytics and Data Science – Analytics Life Cycle – Types of Analytics – Business
Problem Definition – Data Collection – Data Preparation – Hypothesis Generation –
Modeling – Validation and Evaluation – Interpretation – Deployment and Iteration

UNIT II BUSINESS INTELLIGENCE 9

Data Warehouses and Data Mart - Knowledge Management – Types of Decisions -
Decision Making Process - Decision Support Systems – Business Intelligence –
OLAP –, Analytic functions

UNIT III BUSINESS FORECASTING 9

Introduction to Business Forecasting and Predictive analytics - Logic and Data
Driven Models – Data Mining and Predictive Analysis Modeling – Machine Learning
for Predictive analytics.

UNIT IV HR & SUPPLY CHAIN ANALYTICS 9

Human Resources – Planning and Recruitment – Training and Development -
Supply chain network - Planning Demand, Inventory and Supply – Logistics –
Analytics applications in HR & Supply Chain
UNIT V MARKETING & SALES ANALYTICS 9
Marketing Strategy, Marketing Mix, Customer Behavior – selling Process – Sales
Planning – Analytics applications in Marketing and Sales
TOTAL: 45 PERIODS
COURSE OUTCOMES:
 Explain the real world business problems and model with analytical solutions.
 Identify the business processes for extracting Business Intelligence
 Apply predictive analytics for business fore-casting
 Apply analytics for supply chain and logistics management
 Use analytics for marketing and sales.

REFERENCES:
1. R. Evans James, Business Analytics, 2017
2. R.N Prasad seema Acharya,fundamenals of business analytics,2016
3. Philip Kotler and Kevin Keller, Marketing Management, 15th edition, PHI, 2016
4. VSP RAO, Human Resource Management, 3rd Edition, Excel Books, 2010.
5. Mahadevan B, “Operations Management -Theory and Practice”,3rd Edition, Pearson Education, 2018.
AD8551 BUSINESS ANALYTICS OBJECTIVES:
1. To understand the Analytics Life Cycle.
2. To comprehend the process of acquiring Business Intelligence
3. To understand various types of analytics for Business Forecasting
4. To model the supply chain management for Analytics.
5. To apply analytics for different functions of a business

UNIT I INTRODUCTION TO BUSINESS ANALYTICS 9

Analytics and Data Science – Analytics Life Cycle – Types of Analytics – Business
Problem Definition – Data Collection – Data Preparation – Hypothesis Generation –
Modeling – Validation and Evaluation – Interpretation – Deployment and Iteration

Business Analytics:
Def:
•Business intelligence (BI) can be defined as a set of processes and technologies that
convert data into meaningful and useful information for business purposes.
•While some believe that BI is a broad subject that encompasses analytics, business
analytics, and information systems (Bartlett, 2013, p.4), others believe it is mainly focused
on collecting, storing, and exploring large database organizations for information
useful to decision-making and planning (Negash, 2004).

Business analytics, or simply analytics, is the use of data, information

technology, statistical analysis, quantitative methods, and mathematical or
computer-based models to help managers gain improved insight about their
business operations and make better, fact-based decisions. Business analytics
is “a process of transforming data into actions through analysis and insights in
the context of organizational decision making and problem solving.”
Business analytics is supported by various tools such as Microsoft Excel and
various Excel add-ins, commercial statistical software packages such as SAS or
Minitab, and more-complex business intelligence suites that integrate data
with analytical software.
 Tools and techniques of business analytics are used across many areas in a wide
variety
of organizations to improve the management of customer relationships, financial and
marketing activities, human capital, supply chains, and many other areas.
 Leading banks use analytics to predict and prevent credit fraud. Manufacturers use
analytics for production planning, purchasing, and inventory management.
 Retailers use analytics to recommend products to customers and optimize marketing
promotions. Pharmaceutical firms use it to get life-saving drugs to market more
quickly. The leisure and vacation industries use analytics to analyze historical sales
data, understand customer behavior, improve Web site design, and optimize schedules
and bookings. Airlines and hotels use analytics to dynamically set prices over time to
maximize revenue. Even sports teams are using business analytics to determine both
game strategy and optimal ticket prices.4 Among the many organizations that use
analytics to make strategic decisions and manage day-to-day operations are Harrah’s
Entertainment, the Oakland Athletics baseball and New England Patriots football
 teams, [Link], Procter & Gamble, United Parcel Service (UPS), and Capital
One bank. It was reported that nearly all firms with revenues of more than $100
million are using some form of business analytics.
Some common types of decisions that can be enhanced by using analytics include
• pricing (for example, setting prices for consumer and industrial goods, government
contracts, and maintenance contracts),
• customer segmentation (for example, identifying and targeting key customer
groups in retail, insurance, and credit card industries),
• merchandising (for example, determining brands to buy, quantities,
and
allocations),
• location (for example, finding the best location for bank branches and
ATMs, or where to service industrial equipment)

Analytics and Data Science

Comparing Business Analytics to Data Science

Business Analytics Data Science
Business Analytics is the statistical Data science is the study of data using
study of business data to gain insights. statistics, algorithms and technology.
Uses both structured and unstructured
Uses mostly structured data.
data.
Coding is widely used. This field is a
Does not involve much coding. It is combination of traditional analytics
more statistics oriented. practice with good computer science
knowledge.
The whole analysis is based on Statistics is used at the end of analysis
statistical concepts. following coding.
Studies trends and patterns specific to Studies almost every trend and pattern
business. related to technology.
Top industries where business
Top industries/applications where data
analytics is used: finance, healthcare,
science is used: e-commerce, finance,
marketing, retail, supply chain,
machine learning, manufacturing.
telecommunications.

What is concept business analytics?

Business analytics bridges the gap between information technology and business
by using analytics to provide data-driven recommendations. The business part
requires deep business understanding, while the analytics part requires an
understanding of data, statistics and computer science.
What does a business analyst do?

A business analyst acts as a communicator, facilitator and mediator, and

seeks the best ways to improve processes and increase effectiveness
through technology, strategy, analytic solutions, and more.
What are the skills of a business analyst?

 Interpretation: Businesses managers manage a vast amount of

data. As a business analyst, you should have the ability to clean
data and make it useful for interpretation.
 Data Visualization and storytelling: Data visualization is an
evolving discipline, and Tableau defines data visualization as a
graphical representation of data and information. A business
analyst uses such visual elements as charts, graphs and maps, and
provides an accessible way to see and understand trends, outliers
and patterns in data.
 Analytical reasoning ability: Consists of logical reasoning,
critical thinking, communication, research and data analysis. A
business analyst requires these to apply descriptive, predictive and
prescriptive analytics in business situations to solve business
problems.
 Mathematical and statistical skills: The ability to collect,
organize and interpret numerical data is used for modeling,
inference, estimation and forecasting in business analytics.
 Written and communication skills: If you have better
communication skills, it becomes easy to influence the
management team to recommend improvements and increase
business opportunities.
What is data science?
Data science is the study of data using statistics, algorithms and
technology. It is the process of using data to find solutions and predict
outcomes for a problem statement.

What does a data scientist do?

Data scientists apply machine-learning algorithms to numbers, text,
images, videos and audio, and draw various understanding from them.
According to Hugo Bowne-Anderson writing in the Harward Business
Review, “Data scientists lay a solid data foundation in order to perform
robust analytics. Then they use online experiments, among other
methods, to achieve sustainable growth.”
Finally, they build machine learning pipelines and personalized data
products to better understand their business and customers and to make
better decisions. In others, in tech, data science is about infrastructure,
testing, machine learning, decision-making, and data products.”
What are the skills of a data scientist?

The core skills required in data science are as follows:

 Statistical analysis: You should be familiar with statistical tests,
likelihood estimators for a keen sense of pattern and anomaly
detection.
 Computer science and programming: Data scientists encounter
massive datasets. To uncover answers to problems, you will have
to write computer programs and should be proficient in computer
programming languages such as Python, R and SQL.
 Machine learning: As a data scientist, you should be familiar with
algorithms and statistical models that automatically enable a
computer to learn from data.
 Multivariable calculus and linear algebra: This significant
mathematical knowledge is needed for building a machine
learning model.
 Data visualization and storytelling: After you have the data, you
have to present your findings. Data scientists use data
visualization tools to communicate and describe actionable
insights to technical and non-technical audiences.

So, what is better for me?

Business analysts take a hands-on approach to their work by having to
interact and manage the data while data scientists tend to focus more on
data’s development. As I see it, a business analyst can transition into a
data science role with more training hours and experience. I decided to
jump into business analytics because of this particular flexibility.
Business Intelligence vs. Analytics:

Business Analytics Life Cycle:

[Link] the Right Question

Asking questions helps you define what insight you’re trying to gain, problem you’re
trying to solve or larger business goal you’re trying to meet. Asking the right question(s)
helps you build the model to collect relevant data. So how do you determine the “right”
question to ask?

If, for example, sales of a particular product are flat and your goal is to increase sales by
10% to 15%, you’d want to be sure you’re processing the right sales data to discover the
source of your highest and lowest sales. Who within these markets is purchasing the product
and how? Have online sales ticked up? Are in-store or same-store purchases flat or in
decline? Then…

[Link] the Data to Answer the Question

The sources and streams of data flowing into your organization are abundant—
from email, mobile apps, social networks, e-commerce sites and web traffic—
providing information on sales figures, customer engagement, demographic
profiles and the overall health of your business. All that data feeds into or is
collected and fed into your central data center. Effective data management and
programming strategies can help you gather it, clean it and apply it to answer
your question.
 Gather: Extract data from your database through advanced queries using Structured
Query Language (SQL), which supports, extracts, transforms and loads data in
preparation for analytics model development.
 Clean: Data cleaning or cleansing identifies and removes errors, corruptions,
inconsistencies or outliers that can affect data accuracy and, ultimately, its
functionality.

3. Visualize and Analyze the Data

Data visualization—presented in statistical graphs, plots, charts or other visual

context—descriptively summarizes the information so you can apply the right
structured data model for analysis. Specialized analytics systems based on your
model can provide the analysis and statistics.

 Statistics and Algorithms: This can include correlation analysis,

hypothesis testing, and regression analysis to see whether simple
predictions can be made.
 Machine Learning: Decision trees, neural networks and logistics
regression are all machine learning-based predictive analytics techniques
that can turn data into proactive solutions.
As you go through data analysis, different comparisons can derive different insights (i.e.
applying differing machine learning approaches). With enhanced analysis capabilities, you
can realize new opportunities and develop innovative solutions to your business questions.

[Link] the Analysis

Ready to ask more questions? Examining “Is it accurate?”, “Is it appropriate?”
and running “what-if” scenarios will help you determine if your analysis is valid
or on the right track.

Statistical analysis, inference and predictive modeling are applied to define and
validate target parameters, leading to an optimal solution and model most
aligned to your business goal.

[Link] It to the Business

This is where all that data starts providing solutions to your business questions.
You’ve deepened your insight and understanding and now can start making
business recommendations and taking action based on your data analysis,
visualization and models.
As you begin to measure outcomes and derive new insights, new questions will
arise: “Was the action effective?” “Is a different decision or solution needed?”
“What was the return on investment?”
Evaluating appropriateness, managing the value creation of your analytics project, and being
able to identify and emphasize the success factors are all critical to leveraging your analytics
for a competitive advantage.
Types of Analytics:
Business analytics begins with the collection, organization, and manipulation of data and
is supported by three major components:20
1. Descriptive analytics. Most businesses start with descriptive analytics—the use of data to
understand past and current business performance and make informed decisions. Descriptive
analytics is the most commonly used and most well-understood type of analytics. These techniques
categorize, characterize, consolidate, and classify data to convert it into useful information for the
purposes of understanding and analyzing business performance.
Descriptive analytics summarizes data into meaningful charts and reports, for
example, about budgets, sales, revenues, or cost. This process allows managers to obtain
standard and customized reports and then drill down into the data and make queries to understand
the impact of an advertising campaign, for example, review business performance to find problems
or areas of opportunity, and identify patterns and trends in data. Typical questions that descriptive
analytics helps answer are “How much did we sell in each region?” “What was our revenue and
profit last quarter?” “How many and what types of complaints did we resolve?” “Which factory has
the lowest productivity?” Descriptive analytics also helps companies to classify customers into
different segments, which enables them to develop specific marketing campaigns and advertising
strategies.
2. Predictive analytics. Predictive analytics seeks to predict the future by examining
historical data, detecting patterns or relationships in these data, and then extrapolating
these relationships forward in time. For example, a marketer might wish to predict the
response of different customer segments to an advertising campaign, a commodities
trader might wish to predict short-term movements in commodities prices, or a
skiwear manufacturer might want to predict next season’s demand for skiwear of a
specific color and size.
Predictive analytics can predict risk and find relationships in data not readily apparent
with traditional analyses. Using advanced techniques, predictive analytics can help to
detect hidden patterns in large quantities of data to segment and group data into
coherent sets to predict behavior and detect trends. For instance, a bank manager
might want to identify the most profitable customers or predict the chances that a loan
applicant will default, or alert a credit-card customer to a potential fraudulent charge.
Predictive analytics helps to answer questions such as “What will happen if demand
falls by 10% or if supplier prices go up 5%?” “What do we expect to pay for fuel over
the next several months?” “What is the risk of losing money in a new business
venture?”
3. Prescriptive analytics. Many problems, such as aircraft or employee scheduling
and supply chain design, for example, simply involve too many choices or alternatives for a
human decision maker to effectively consider.
Prescriptive analytics uses optimization to identify the best alternatives to minimize or
Maximize some objective. Prescriptive analytics is used in many areas of business,
including operations, marketing, and finance. For example, we may determine the best
pricing and advertising strategy to maximize revenue, the optimal amount of cash to
store in ATMs, or the best mix of investments in a retirement portfolio to manage risk.
The mathematical and statistical techniques of predictive
analytics can also be combined with optimization to make
decisions that take into account the uncertainty in the data.
Prescriptive analytics addresses questions such as “How much
should we produce to maximize profit?” “What is the best way
of shipping goods from our factories to minimize costs?” “Should
we change our plans if a natural disaster closes a supplier’s factory: if
so, by how much?”
Analytic Purposes and Tools
Business Problem Definition:
Analytics in Practice: Analytics in the Home Lending and Mortgage
Industry
Sometime during their lives, most Americans will
receive a mortgage loan for a house or
• How many loan apps were taken each of the
past
condominium. The process starts with an
12 months?
application. The application contains all pertinent
• What was the total cycle time from app to close?
information about the borrower that the lender will
• What was the distribution of loan profitability by
need. The bank or mortgage company then initiates
credit score and loan-to-value (LTV), which is the
a process that leads to a loan decision.
mortgage amount divided by the appraised value
It is here that key information about the borrower is
of the property.
provided by third-party providers. This information
Predictive Analytics—Predictive modeling use
includes a credit report, verification of income,
mathematical,
verification of assets, verification of employment,
spreadsheet, and statistical models,
and an appraisal of the property among others. The
and address questions such as:
result of the processing function is a complete loan
• What impact on loan volume will a given
file that contains all the information and documents
marketing
needed to underwrite the loan, which is the next
program have?
step in the process.
• How many processors or underwriters are needed
Underwriting is where the loan application is
for a given loan volume?
evaluated for its risk.
• Will a given process change reduce cycle time?
Underwriters evaluate whether the borrower can
Prescriptive Analytics—This involves the use of
make payments on time, can afford to pay back the
simulation
loan, and has sufficient collateral in the property to
or optimization to drive decisions. Typical
back up the loan. In the event the borrower defaults
questions
on their loan, the lender can sell the property to
include:
recover the amount of the loan. But, if the amount
• What is the optimal staffing to achieve a given
of the loan is greater than the value of the property,
profitability constrained by a fixed cycle time?
then the lender cannot recoup their money. If the
• What is the optimal product mix to maximize
underwriting process indicates that the borrower is
profit
creditworthy, has the capacity to repay
constrained by fixed staffing?
the loan, and the value of the property in question
The mortgage market has become much more
is greater than the loan amount, then the loan is
dynamic in recent years due to rising home values,
approved and will move to closing. Closing is the
falling interest rates, new loan products, and an
step where the borrower signs all the appropriate
increased desire by home owners to utilize the
papers agreeing to the terms of the loan.
equity in their homes as a financial resource. This
In reality, lenders have a lot of other work to
has increased the complexity and variability of the
do.
mortgage process and created an opportunity for
 First, they must perform a quality control review on lenders to proactively use the data that are available
a sample of the loan files that involves a manual to them as a tool for managing their business.
examination of all the documents and information To ensure that the process is efficient, effective and
gathered. This process is designed to identify any performed with quality, data and analytics are used
mistakes that may have been made or information every day to track what is done, who is doing it,
that is missing from the loan file. Because lenders and how long it takes.
do not have unlimited money to lend to borrowers,
they frequently sell the loan to a third party so that
they have fresh capital to lend to others.
This occurs in what is called the secondary market.
Freddie Mac and Fannie Mae are the two largest
purchasers of mortgages in the secondary market.
The final step in the process is servicing. Servicing
includes all the activities associated with providing
the customer service on the loan like processing
payments, managing property taxes held in escrow,
and answering questions about the loan.
In addition, the institution collects various
operational data on the process to track its
performance and efficiency, including the number
of applications, loan types and amounts, cycle
times (time to close the loan), bottlenecks in the
process, and so on.
Many different types of analytics are used:
Descriptive Analytics—This focuses on historical
reporting, addressing such questions as:

Data for Business Analytics:

Data: numerical or textual facts and figures that
are collected through some type of measurement
process.

Information: result of analyzing data; that is,

extracting meaning from data to support
evaluation and decision making

Examples for Data Sources and Uses:

Data Collection

Data collection is the methodological process of gathering information about a specific subject.
In general, there are three types of consumer data:

 First-party data, which is collected directly from users by your organization

 Second-party data, which is data shared by another organization about its customers (or its first-
party data)
 Third-party data, which is data that’s been aggregated and rented or sold by organizations that
don’t have a connection to your company or users

Although there are use cases for second- and third-party data, first-party data (data you’ve collected
yourself) is more valuable because you receive information about how your audience behaves, thinks, and
feels—all from a trusted source.

Data can be qualitative (meaning contextual in nature) or quantitative (meaning numeric in nature). Many
data collection methods apply to either type, but some are better suited to one over the other.

In the data life cycle, data collection is the second step. After data is generated, it must be collected to be of
use to your team. After that, it can be processed, stored, managed, analyzed, and visualized to aid in your
organization’s decision-making.

Before collecting data, there are several factors you need to define:

 The question you aim to answer

 The data subject(s) you need to collect data from
 The collection timeframe
 The data collection method(s) best suited to your needs
The data collection method you select should be based on the question you want to answer, the type of data
you need, your timeframe, and your company’s budget.

7 Data Collection Methods Used in Business Analytics

1. Surveys

Surveys are physical or digital questionnaires that gather both qualitative and quantitative
data from subjects. One situation in which you might conduct a survey is gathering attendee
feedback after an event. This can provide a sense of what attendees enjoyed, what they wish
was different, and areas you can improve or save money on during your next event for a
similar audience.

Because they can be sent out physically or digitally, surveys present the opportunity for
distribution at scale. They can also be inexpensive; running a survey can cost nothing if you
use a free tool. If you wish to target a specific group of people, partnering with a market
research firm to get the survey in the hands of that demographic may be worth the money.

Something to watch out for when crafting and running surveys is the effect of bias,
including:

 Collection bias: It can be easy to accidentally write survey questions with a biased
lean. Watch out for this when creating questions to ensure your subjects answer
honestly and aren’t swayed by your wording.
 Subject bias: Because your subjects know their responses will be read by you, their
answers may be biased toward what seems socially acceptable. For this reason,
consider pairing survey data with behavioral data from other collection methods to
get the full picture.

2. Transactional Tracking

Each time your customers make a purchase, tracking that data can allow you to make
decisions about targeted marketing efforts and understand your customer base better.

Often, e-commerce and point-of-sale platforms allow you to store data as soon as it’s
generated, making this a seamless data collection method that can pay off in the form of
customer insights.

3. Interviews and Focus Groups

Interviews and focus groups consist of talking to subjects face-to-face about a specific topic
or issue. Interviews tend to be one-on-one, and focus groups are typically made up of
several people. You can use both to gather qualitative and quantitative data.

Through interviews and focus groups, you can gather feedback from people in your target
audience about new product features. Seeing them interact with your product in real-time
and recording their reactions and responses to questions can provide valuable data about
which product features to pursue.
As is the case with surveys, these collection methods allow you to ask subjects anything you
want about their opinions, motivations, and feelings regarding your product or brand. It also
introduces the potential for bias. Aim to craft questions that don’t lead them in one particular
direction.

One downside of interviewing and conducting focus groups is they can be time-consuming
and expensive. If you plan to conduct them yourself, it can be a lengthy process. To avoid
this, you can hire a market research facilitator to organize and conduct interviews on your
behalf.

4. Observation

Observing people interacting with your website or product can be useful for data collection
because of the candor it offers. If your user experience is confusing or difficult, you can
witness it in real-time.

Yet, setting up observation sessions can be difficult. You can use a third-party tool to record
users’ journeys through your site or observe a user’s interaction with a beta version of your
site or product.

While less accessible than other data collection methods, observations enable you to see
firsthand how users interact with your product or site. You can leverage the qualitative and
quantitative data gleaned from this to make improvements and double down on points of
success.

5. Online Tracking

To gather behavioral data, you can implement pixels and cookies. These are both tools that
track users’ online behavior across websites and provide insight into what content they’re
interested in and typically engage with.

You can also track users’ behavior on your company’s website, including which parts are of
the highest interest, whether users are confused when using it, and how long they spend on
product pages. This can enable you to improve the website’s design and help users navigate
to their destination.

Inserting a pixel is often free and relatively easy to set up. Implementing cookies may come
with a fee but could be worth it for the quality of data you’ll receive. Once pixels and
cookies are set, they gather data on their own and don’t need much maintenance, if any.

It’s important to note: Tracking online behavior can have legal and ethical privacy
implications. Before tracking users’ online behavior, ensure you’re in compliance with local
and industry data privacy standards.

6. Forms

Online forms are beneficial for gathering qualitative data about users,
specifically demographic data or contact information. They’re relatively
inexpensive and simple to set up, and you can use them to gate content or
registrations, such as webinars and email newsletters.

You can then use this data to contact people who may be interested in your
product, build out demographic profiles of existing customers, and in
remarketing efforts, such as email workflows and content recommendations.

7. Social Media Monitoring

Monitoring your company’s social media channels for follower engagement is an accessible
way to track data about your audience’s interests and motivations. Many social media
platforms have analytics built in, but there are also third-party social platforms that give
more detailed, organized insights pulled from multiple channels.

You can use data collected from social media to determine which issues are most important
to your followers. For instance, you may notice that the number of engagements
dramatically increases when your company posts about its sustainability efforts.

Data Preparation:
What is data preparation in business analytics?
Data preparation is the process of gathering, combining, structuring and organizing
data so it can be used in business intelligence (BI), analytics and data visualization
applications.

Data preparation, also sometimes called “pre-processing,” is the act of cleaning

and consolidating raw data prior to using it for business analysis. It might not be
the most celebrated of tasks, but careful data preparation is a key component of
successful data analysis.

Doing the work to properly validate, clean, and augment raw data is essential to
draw accurate, meaningful insights from it. The validity and power of any
business analysis produced is only as good as the data preparation done in the
early stages.
Data preparation work is done by information technology (IT), BI and data management
teams as they integrate data sets to load into a data warehouse, NoSQL database or data lake
repository, and then when new analytics applications are developed with those data sets. In
addition, data scientists, data engineers, other data analysts and business users increasingly
use self-service data preparation tools to collect and prepare data themselves.

Data preparation is often referred to informally as data prep. It's also known as data
wrangling, although some practitioners use that term in a narrower sense to refer to
cleansing, structuring and transforming data; that usage distinguishes data wrangling from
the data preprocessing stage.

Purposes of data preparation

One of the primary purposes of data preparation is to ensure that raw data being readied for
processing and analysis is accurate and consistent so the results of BI and analytics
applications will be valid. Data is commonly created with missing values, inaccuracies or
other errors, and separate data sets often have different formats that need to be reconciled
when they're combined. Correcting data errors, validating data quality and consolidating
data sets are big parts of data preparation projects.

Data preparation also involves finding relevant data to ensure that analytics applications
deliver meaningful information and actionable insights for business decision-making. The
data often is enriched and optimized to make it more informative and useful -- for example,
by blending internal and external data sets, creating new data fields, eliminating outlier
values and addressing imbalanced data sets that could skew analytics results.

In addition, BI and data management teams use the data preparation process to curate data
sets for business users to analyze. Doing so helps streamline and guide self-service BI
applications for business analysts, executives and workers.

What are the benefits of data preparation?

Data scientists often complain that they spend most of their time gathering,
cleansing and structuring data instead of analyzing it. A big benefit of an
effective data preparation process is that they and other end users can focus more
on data mining and data analysis -- the parts of their job that generate business
value. For example, data preparation can be done more quickly, and prepared
data can automatically be fed to users for recurring analytics applications.

Done properly, data preparation also helps an organization do the following:

 ensure the data used in analytics applications produces reliable results;

 identify and fix data issues that otherwise might not be detected;
 enable more informed decision-making by business executives and
operational workers;
 reduce data management and analytics costs;
 avoid duplication of effort in preparing data for use in multiple
applications; and
 get a higher ROI from BI and analytics initiatives.

Effective data preparation is particularly beneficial in big data environments that

store a combination of structured, semi structured and unstructured data, often in
raw form until it's needed for specific analytics uses. Those uses include
predictive analytics, machine learning (ML) and other forms of advanced
analytics that typically involve large amounts of data to prepare.
Steps in the data preparation process

Data preparation is done in a series of steps. There's some variation in the data preparation
steps listed by different data professionals and software vendors, but the process typically
involves the following tasks:

1. Data collection. Relevant data is gathered from operational systems, data

warehouses, data lakes and other data sources. During this step, data scientists,
members of the BI team, other data professionals and end users who collect data
should confirm that it's a good fit for the objectives of the planned analytics
applications.
2. Data discovery and profiling. The next step is to explore the collected data to better
understand what it contains and what needs to be done to prepare it for the intended
uses. To help with that, data profiling identifies patterns, relationships and other
attributes in the data, as well as inconsistencies, anomalies, missing values and other
issues so they can be addressed.
3. Data cleansing. Next, the identified data errors and issues are corrected to create
complete and accurate data sets. For example, as part of cleansing data sets, faulty
data is removed or fixed, missing values are filled in and inconsistent entries are
harmonized.
4. Data structuring. At this point, the data needs to be modeled and organized to meet
the analytics requirements. For example, data stored in comma-separated values
(CSV) files or other file formats has to be converted into tables to make it accessible
to BI and analytics tools.
5. Data transformation and enrichment. In addition to being structured, the data
typically must be transformed into a unified and usable format. For example, data
transformation may involve creating new fields or columns that aggregate values from
existing ones. Data enrichment further enhances and optimizes data sets as needed,
through measures such as augmenting and adding data.
6. Data validation and publishing. In this last step, automated routines are run against
the data to validate its consistency, completeness and accuracy. The prepared data is
then stored in a data warehouse, a data lake or another repository and either used
directly by whoever prepared it or made available for other users to access.

Data preparation can also incorporate or feed into data curation work that creates
and oversees ready-to-use data sets for BI and analytics. Data curation involves
tasks such as indexing, cataloging and maintaining data sets and their associated
metadata to help users find and access the data. In some organizations, data
curator is a formal role that works collaboratively with data scientists, business
analysts, other users and the IT and data management teams. In others, data may
be curated by data stewards, data engineers, database administrators or data
scientists and business users themselves.
What are the challenges of data preparation ?

Data preparation is inherently complicated. Data sets pulled together from different source
systems are highly likely to have numerous data quality, accuracy and consistency issues to
resolve. The data also must be manipulated to make it usable, and irrelevant data needs to be
weeded out. As noted above, it's a time-consuming process: The 80/20 rule is often applied
to analytics applications, with about 80% of the work said to be devoted to collecting and
preparing data and only 20% to analyzing it.

 Inadequate or nonexistent data profiling. If data isn't properly profiled, errors,

anomalies and other problems might not be identified, which can result in flawed
analytics.
  Missing or incomplete data. Data sets often have missing values and other forms of
incomplete data; such issues need to be assessed as possible errors and addressed if
so.
 Invalid data values. Misspellings, other typos and wrong numbers are examples of
invalid entries that frequently occur in data and must be fixed to ensure analytics
accuracy.
 Name and address standardization. Names and addresses may be inconsistent in
data from different systems, with variations that can affect views of customers and
other entities.
 Inconsistent data across enterprise systems. Other inconsistencies in data sets
drawn from multiple source systems, such as different terminology and unique
identifiers, are also a pervasive issue in data preparation efforts.
 Data enrichment. Deciding how to enrich a data set - for example, what to add to it -
is a complex task that requires a strong understanding of business needs and analytics
goals.
 Maintaining and expanding data prep processes. Data preparation work often
becomes a recurring process that needs to be sustained and enhanced on an ongoing
basis.

How to get started on data preparation

1. Think of data preparation as part of data analysis. Data preparation and analysis are
"two sides of the same coin," Farmer wrote. Data, he said, can't be properly prepared
without knowing what analytics use it needs to fit.

2. Define what data preparation success means. Desired data accuracy levels and other data
quality metrics should be set as goals, balanced against projected costs to create a data prep
plan that's appropriate to each use case.

3. Prioritize data sources based on the application. Resolving differences in data from
multiple source systems is an important element of data preparation that also should be
based on the planned analytics use case.

4. Use the right tools for the job and your skill level. Self-service data preparation tools
aren't the only option available -- other tools and technologies can also be used, depending
on your skills and data needs.

5. Be prepared for failures when preparing data. Error-handling capabilities need to be

built into the data preparation process to prevent it from going awry or getting bogged down
when problems occur.

6. Keep an eye on data preparation costs. The cost of software licenses, processing and
storage resources, and the people involved in preparing data should be watched closely to
ensure that they don't get out of hand.
Hypothesis Generation

-Refer Notes (Written)

Validation and Evaluation

DATA VALIADATION:

What is data validation?

Data validation is the practice of checking the integrity, accuracy and structure
of data before it is used for a business operation. Data validation operation
results can provide data used for data analytics, business intelligence or training
a machine learning model. It can also be used to ensure the integrity of data for
financial accounting or regulatory compliance.

Data can be examined as part of a validation process in a variety of ways,

including data type, constraint, structured, consistency and code validation. Each
type of data validation is designed to make sure the data meets the requirements
to be useful.
Data validation is related to data quality. Data validation can be a component to
measure data quality, which ensures that a given data set is supplied with
information sources that are of the highest quality, authoritative and accurate.

Data validation is also used as part of application workflows, including spell

checking and rules for strong password creation.

Why validate data?

For data scientists, data analysts and others working with data, validating it is
very important. The output of any given system can only be as good as the data
the operation is based on. These operations can include machine learning or
artificial intelligence models, data analytics reports and business intelligence
dashboards. Validating the data ensures that the data is accurate, which means all
systems relying on a validated given data set will be as well.

Data validation is also important for data to be useful for an organization or for a
specific application operation. For example, if data is not in the right format to
be consumed by a system, then the data can't be used easily, if at all.

As data moves from one location to another, different needs for the data arise
based on the context for how the data is being used. Data validation ensures that
the data is correct for specific contexts. The right type of data validation makes
the data useful.
What are the different types of data validation?

Multiple types of data validation are available to ensure that the right data is being used. The
most common types of data validation include the following:

 Data type validation is common and confirms that the data in each field,
column, list, range or file matches a specified data type and format.
 Constraint validation checks to see if a given data field input fits a
specified requirement within certain ranges. For example, it verifies that a
data field has a minimum or maximum number of characters.
 Structured validation ensures that data is compliant with a specified data
format, structure or schema.
 Consistency validation makes sure data styles are consistent. For
example, it confirms that all values are listed to two decimal points.
 Code validation is similar to a consistency check and confirms that codes
used for different data inputs are correct. For example, it checks a country
code or North American Industry Classification System (NAICS) codes.
 Performing data validation

Among the most basic and common ways that data is used is within a
spreadsheet program such as Microsoft Excel or Google Sheets. In both Excel
and Sheets, the data validation process is a straightforward, integrated feature.
Excel and Sheets both have a menu item listed as Data > Data Validation. By
selecting the Data Validation menu, a user can choose the specific data type or
constraint validation required for a given file or data range.

ETL (Extract, Transform and Load) and data integration tools typically integrate
data validation policies to be executed as data is extracted from one source and
then loaded into another. Popular open source tools, such as dbt, also include
data validation options and are commonly used for data transformation.

Data validation can also be done programmatically in an application context for

an input value. For example, as an input variable is sent, such as a password, it
can be checked by a script to make sure it meets constraint validation for the
right length.
Interpretation

What Is Data Interpretation?

Data interpretation refers to the process of using diverse analytical methods to

review data and arrive at relevant conclusions. The interpretation of data helps
researchers to categorize, manipulate, and summarize the information in order to
answer critical questions.

The importance of data interpretation is evident and this is why it needs to be done
properly. Data is very likely to arrive from multiple sources and has a tendency to
enter the analysis process with haphazard ordering. Data analysis tends to be
extremely subjective. That is to say, the nature and goal of interpretation will vary
from business to business, likely correlating to the type of data being analyzed.
While there are several different types of processes that are implemented based on
individual data nature, the two broadest and most common categories are
“quantitative analysis” and “qualitative analysis”.

Yet, before any serious data interpretation inquiry can begin, it should be understood that visual
presentations of data findings are irrelevant unless a sound decision is made regarding scales of
measurement. Before any serious data analysis can begin, the scale of measurement must be
decided for the data as this will have a long-term impact on data interpretation ROI. The varying
scales include:

 Nominal Scale: non-numeric categories that cannot be ranked or compared

quantitatively. Variables are exclusive and exhaustive.
 Ordinal Scale: exclusive categories that are exclusive and exhaustive but with a logical
order. Quality ratings and agreement ratings are examples of ordinal scales (i.e., good,
very good, fair, etc., OR agree, strongly agree, disagree, etc.).
 Interval: a measurement scale where data is grouped into categories with orderly and
equal distances between the categories. There is always an arbitrary zero point.
 Ratio: contains features of all three.

For a more in-depth review of scales of measurement, read our article on data analysis questions.
Once scales of measurement have been selected, it is time to select which of the two broad
interpretation processes will best suit your data needs. Let’s take a closer look at those specific
data interpretation methods and possible data interpretation problems.

How To Interpret Data?

When interpreting data, an analyst must try to discern the differences between correlation,
causation, and coincidences, as well as much other bias – but he also has to consider all the
factors involved that may have led to a result. There are various data interpretation
methods one can use.

The interpretation of data is designed to help people make sense of numerical data that
has been collected, analyzed, and presented. Having a baseline method (or methods) for
interpreting data will provide your analyst teams with a structure and consistent
foundation. Indeed, if several departments have different approaches to interpret the same
data while sharing the same goals, some mismatched objectives can result. Disparate
methods will lead to duplicated efforts, inconsistent solutions, wasted energy, and
inevitably – time and money. In this part, we will look at the two main methods of
interpretation of data: a qualitative and quantitative analysis.

When interpreting data, an analyst must try to discern the differences between correlation,
causation, and coincidences, as well as much other bias – but he also has to consider all
the factors involved that may have led to a result. There are various data interpretation
methods one can use.

The interpretation of data is designed to help people make sense of numerical data that
has been collected, analyzed, and presented. Having a baseline method (or methods) for
interpreting data will provide your analyst teams with a structure and consistent
foundation. Indeed, if several departments have different approaches to interpret the same
data while sharing the same goals, some mismatched objectives can result. Disparate
methods will lead to duplicated efforts, inconsistent solutions, wasted energy, and
inevitably – time and money. In this part, we will look at the two main methods of
interpretation of data: a qualitative and quantitative analysis.

Qualitative Data Interpretation

Qualitative data analysis can be summed up in one word – categorical. With qualitative
analysis, data is not described through numerical values or patterns, but through the use
of descriptive context (i.e., text). Typically, narrative data is gathered by employing a wide
variety of person-to-person techniques. These techniques include:

 Observations: detailing behavioral patterns that occur within an observation group. These
patterns could be the amount of time spent in an activity, the type of activity, and the method of
communication employed.
 Focus groups: Group people and ask them relevant questions to generate a collaborative
discussion about a research topic.
 Secondary Research: much like how patterns of behavior can be observed, different types of
documentation resources can be coded and divided based on the type of material they contain.
 Interviews: one of the best collection methods for narrative data. Inquiry responses can be
grouped by theme, topic, or category. The interview approach allows for highly-focused data
segmentation.

A key difference between qualitative and quantitative analysis is clearly noticeable in the
interpretation stage. Qualitative data, as it is widely open to interpretation, must be
“coded” so as to facilitate the grouping and labeling of data into identifiable themes. As
person-to-person data collection techniques can often result in disputes pertaining to
proper analysis, qualitative data analysis is often summarized through three basic
principles: notice things, collect things, think about things.

Quantitative Data Interpretation

If quantitative data interpretation could be summed up in one word (and it really can’t) that
word would be “numerical.” There are few certainties when it comes to data analysis, but
you can be sure that if the research you are engaging in has no numbers involved, it is not
quantitative research. Quantitative analysis refers to a set of processes by which
numerical data is analyzed. More often than not, it involves the use of statistical modeling
such as standard deviation, mean and median. Let’s quickly review the most common
statistical terms:

 Mean: a mean represents a numerical average for a set of responses. When dealing with a data
set (or multiple data sets), a mean will represent a central value of a specific set of numbers. It is
the sum of the values divided by the number of values within the data set. Other terms that can be
used to describe the concept are arithmetic mean, average and mathematical expectation.
 Standard deviation: this is another statistical term commonly appearing in quantitative analysis.
Standard deviation reveals the distribution of the responses around the mean. It describes the
degree of consistency within the responses; together with the mean, it provides insight into data
sets.
 Frequency distribution: this is a measurement gauging the rate of a response appearance
within a data set. When using a survey, for example, frequency distribution has the capability of
determining the number of times a specific ordinal scale response appears (i.e., agree, strongly
agree, disagree, etc.). Frequency distribution is extremely keen in determining the degree of
consensus among data points.

Typically, quantitative data is measured by visually presenting correlation tests between

two or more variables of significance. Different processes can be used together or
separately, and comparisons can be made to ultimately arrive at a conclusion. Other
signature interpretation processes of quantitative data include:

 Regression analysis: Essentially, regression analysis uses historical data to understand the
relationship between a dependent variable and one or more independent variables. Knowing
which variables are related and how they developed in the past allows you to anticipate possible
outcomes and make better decisions going forward. For example, if you want to predict your sales
for next month you can use regression analysis to understand what factors will affect them such
as products on sale, the launch of a new campaign, among many others.

 Cohort analysis: This method identifies groups of users who share common characteristics
during a particular time period. In a business scenario, cohort analysis is commonly used to
understand different customer behaviors. For example, a cohort could be all users who have
signed up for a free trial on a given day. An analysis would be carried out to see how these users
behave, what actions they carry out, and how their behavior differs from other user groups.
 Predictive analysis: As its name suggests, the predictive analysis method aims to predict future
developments by analyzing historical and current data. Powered by technologies such as artificial
intelligence and machine learning, predictive analytics practices enable businesses to spot trends
or potential issues and plan informed strategies in advance.

 Prescriptive analysis: Also powered by predictions, the prescriptive analysis method uses
techniques such as graph analysis, complex event processing, neural networks, among others, to
try to unravel the effect that future decisions will have in order to adjust them before they are
actually made. This helps businesses to develop responsive, practical business strategies.

 Conjoint analysis: Typically applied to survey analysis, the conjoint approach is used to analyze
how individuals value different attributes of a product or service. This helps researchers and
businesses to define pricing, product features, packaging, and many other attributes. A common
use is menu-based conjoint analysis in which individuals are given a “menu” of options from which
they can build their ideal concept or product. Like this analysts can understand which attributes
they would pick above others and drive conclusions.

 Cluster analysis: Last but not least, cluster analysis is a method used to group objects into
categories. Since there is no target variable when using cluster analysis, it is a useful method to
find hidden trends and patterns in the data. In a business context clustering is used for audience
segmentation to create targeted experiences, and in market research, it is often used to
identify age groups, geographical information, earnings, among others.

Now that we have seen how to interpret data, let's move on and ask ourselves some
questions: what are some data interpretation benefits? Why do all industries engage in
data research and analysis? These are basic questions, but they often don’t receive
adequate attention

Why Data Interpretation Is Important

The purpose of collection and interpretation is to acquire useful and usable information and
to make the most informed decisions possible. From businesses to newlyweds
researching their first home, data collection and interpretation provides limitless
benefits for a wide range of institutions and individuals.

Data analysis and interpretation, regardless of the method and

qualitative/quantitative status, may include the following characteristics:

 Data identification and explanation

 Comparing and contrasting of data
 Identification of data outliers
 Future predictions

Data analysis and interpretation, in the end, help improve processes and identify
problems. It is difficult to grow and make dependable improvements without, at
the very least, minimal data collection and interpretation. What is the keyword?
Dependable. Vague ideas regarding performance enhancement exist within all
institutions and industries. Yet, without proper research and analysis, an idea is
likely to remain in a stagnant state forever (i.e., minimal growth). So… what are
a few of the business benefits of digital age data analysis and interpretation?
Let’s take a look!

1) Informed decision-making: A decision is only as good as the knowledge that

formed it. Informed data decision-making has the potential to set industry
leaders apart from the rest of the market pack. Studies have shown that
companies in the top third of their industries are, on average, 5% more
productive and 6% more profitable when implementing informed data decision-
making processes. Most decisive actions will arise only after a problem has been
identified or a goal defined. Data analysis should include identification, thesis
development, and data collection followed by data communication.

If institutions only follow that simple order, one that we should all be familiar
with from grade school science fairs, then they will be able to solve issues as
they emerge in real-time. Informed decision-making has a tendency to be
cyclical. This means there is really no end, and eventually, new questions and
conditions arise within the process that needs to be studied further. The
monitoring of data results will inevitably return the process to the start with new
data and sights.

2) Anticipating needs with trends identification: data insights provide

knowledge, and knowledge is power. The insights obtained from market and
consumer data analyses have the ability to set trends for peers within similar
market segments. A perfect example of how data analysis can impact trend
prediction can be evidenced in the music identification application, Shazam. The
application allows users to upload an audio clip of a song they like, but can’t
seem to identify. Users make 15 million song identifications a day. With this
data, Shazam has been instrumental in predicting future popular artists.

When industry trends are identified, they can then serve a greater industry
purpose. For example, the insights from Shazam’s monitoring benefits not only
Shazam in understanding how to meet consumer needs, but it grants music
executives and record label companies an insight into the pop-culture scene of
the day. Data gathering and interpretation processes can allow for industry-wide
climate prediction and result in greater revenue streams across the market. For
this reason, all institutions should follow the basic data cycle of collection,
interpretation, decision making, and monitoring.

3) Cost efficiency: Proper implementation of data analysis processes can

provide businesses with profound cost advantages within their industries. A
recent data study performed by Deloitte vividly demonstrates this in finding that
data analysis ROI is driven by efficient cost reductions. Often, this benefit is
overlooked because making money is typically viewed as “sexier” than saving
 money. Yet, sound data analyses have the ability to alert management to cost-
reduction opportunities without any significant exertion of effort on the part of
human capital.

A great example of the potential for cost efficiency through data analysis is Intel.
Prior to 2012, Intel would conduct over 19,000 manufacturing function tests on
their chips before they could be deemed acceptable for release. To cut costs and
reduce test time, Intel implemented predictive data analyses. By using historic
and current data, Intel now avoids testing each chip 19,000 times by focusing on
specific and individual chip tests. After its implementation in 2012, Intel saved
over $3 million in manufacturing costs. Cost reduction may not be as “sexy” as
data profit, but as Intel proves, it is a benefit of data analysis that should not be
neglected.

4) Clear foresight: companies that collect and analyze their data gain better
knowledge about themselves, their processes, and performance. They can
identify performance challenges when they arise and take action to overcome
them. Data interpretation through visual representations lets them process their
findings faster and make better-informed decisions on the future of the company.

Common Data Analysis And Interpretation Problems

The oft-repeated mantra of those who fear data advancements in the digital age
is “big data equals big trouble.” While that statement is not accurate, it is safe to
say that certain data interpretation problems or “pitfalls” exist and can occur
when analyzing data, especially at the speed of thought. Let’s identify some of
the most common data misinterpretation risks and shed some light on how they
can be avoided:

1) Correlation mistaken for causation: our first misinterpretation of data refers

to the tendency of data analysts to mix the cause of a phenomenon with
correlation. It is the assumption that because two actions occurred together, one
caused the other. This is not accurate as actions can occur together absent a
cause and effect relationship.

 Digital age example: assuming that increased revenue is the result of increased
social media followers… there might be a definitive correlation between the two,
especially with today’s multi-channel purchasing experiences. But, that does not
mean an increase in followers is the direct cause of increased revenue. There
could be both a common cause or an indirect causality.
 Remedy: attempt to eliminate the variable you believe to be causing the
phenomenon.
 2) Confirmation bias: our second data interpretation problem occurs when you
have a theory or hypothesis in mind but are intent on only discovering data
patterns that provide support to it while rejecting those that do not.

 Digital age example: your boss asks you to analyze the success of a recent multi-
platform social media marketing campaign. While analyzing the potential data
variables from the campaign (one that you ran and believe performed well), you
see that the share rate for Facebook posts was great, while the share rate for
Twitter Tweets was not. Using only the Facebook posts to prove your hypothesis
that the campaign was successful would be a perfect manifestation of
confirmation bias.
 Remedy: as this pitfall is often based on subjective desires, one remedy would be
to analyze data with a team of objective individuals. If this is not possible,
another solution is to resist the urge to make a conclusion before data exploration
has been completed. Remember to always try to disprove a hypothesis, not prove
it.

3) Irrelevant data: the third data misinterpretation pitfall is especially important

in the digital age. As large data is no longer centrally stored, and as it continues
to be analyzed at the speed of thought, it is inevitable that analysts will focus on
data that is irrelevant to the problem they are trying to correct.

 Digital age example: in attempting to gauge the success of an email lead

generation campaign, you notice that the number of homepage views directly
resulting from the campaign increased, but the number of monthly newsletter
subscribers did not. Based on the number of homepage views, you decide the
campaign was a success when really it generated zero leads.
 Remedy: proactively and clearly frame any data analysis variables and KPIs
prior to engaging in a data review. If the metric you are using to measure the
success of a lead generation campaign is newsletter subscribers, there is no need
to review the number of homepage visits. Be sure to focus on the data variable
that answers your question or solves your problem and not on irrelevant data.

4) Truncating an Axes: When creating a graph to start interpreting the results of

your analysis it is important to keep the axes truthful and avoid generating
misleading visualizations. Starting the axes in a value that doesn’t portray the
actual truth about the data can lead to false conclusions.

 Digital age example: In the image below we can see a graph from Fox News in
which the Y-axes start at 34%, making it seem that the difference between 35%
and 39.6% is way higher than it actually is. This could lead to a misinterpretation
of the tax rate changes.
 5) (Small) sample size: Another common data analysis and interpretation
problem is the use of a small sample size. Logically, the bigger the sample size
the most accurate and reliable are the results. However, this also depends on the
size of the effect of the study. For example, the sample size in a survey about the
quality of education will not be the same as for one about people doing outdoor
sports in a specific area.

 Digital age example: Imagine you ask 30 people a question and 29 answer “yes” resulting in
95% of the total. Now imagine you ask the same question to 1000 and 950 of them answer
“yes”, which is again 95%. While these percentages might look the same, they certainly do
not mean the same thing as a 30 people sample size is not a significant number to establish a
truthful conclusion.
 Remedy: Researchers say that in order to determine the correct sample size to get truthful
and meaningful results it is necessary to define a margin of error that will represent the
maximum amount they want the results to deviate from the statistical mean. Paired to this,
they need to define a confidence level that should be between 90 and 99%. With these two
values in hand, researchers can calculate an accurate sample size for their studies.

6) Reliability, subjectivity, and generalizability: When performing qualitative

analysis, researchers must consider practical and theoretical limitations when
interpreting the data. In some cases, qualitative research can be considered
unreliable because of uncontrolled factors that might or might not affect the
results. This is paired with the fact that the researcher has a primary role in the
interpretation process, meaning he or she decides what is relevant and what is
not, and as we know, interpretations can be very subjective.

Generalizability is also an issue that researchers face when dealing with

qualitative analysis. As mentioned in the point about small sample size, it is
difficult to draw conclusions that are 100% representative because the results
might be biased or unrepresentative of a wider population.

While these factors are mostly present in qualitative research, they can also
affect quantitative analysis. For example, when choosing which KPIs to portray
and how to portray them, analysts can also be biased and represent them in a
way that benefits their analysis.

 Digital age example: Biased questions in a survey are a great example of reliability and
subjectivity issues. Imagine you are sending a survey to your clients to see how satisfied
they are with your customer service with this question: “how amazing was your experience
with our customer service team?”. Here we can see that this question is clearly influencing
the response of the individual by putting the word “amazing” on it.
 Remedy: A solution to avoid these issues is to keep your research honest and neutral. Keep
the wording of the questions as objective as possible. For example: “on a scale of 1-10 how
satisfied were you with our customer service team”. This is not leading the respondent to
any specific answer, meaning the results of your survey will be reliable.

Data Interpretation Techniques and Methods

A Guide To The Methods, Benefits & Problems of The Interpretation of

Data
By Bernardita Calzon in Data Analysis, Jan 6th 2022

Table of Contents

1) What Is Data Interpretation?

2) How To Interpret Data?
 3) Why Data Interpretation Is Important?
4) Data Analysis & Interpretation Problems
5) Data Interpretation Techniques & Methods
6) The Use of Dashboards For Data Interpretation
Data analysis and interpretation have now taken center stage with the advent of the digital
age… and the sheer amount of data can be frightening. In fact, a Digital Universe study
found that the total data supply in 2012 was 2.8 trillion gigabytes! Based on that amount of
data alone, it is clear the calling card of any successful enterprise in today’s global world
will be the ability to analyze complex data, produce actionable insights and adapt to new
market needs… all at the speed of thought.

Business dashboards are the digital age tools for big data. Capable of displaying key
performance indicators (KPIs) for both quantitative and qualitative data analyses, they are
ideal for making the fast-paced and data-driven market decisions that push today’s
industry leaders to sustainable success. Through the art of streamlined visual
communication, data dashboards permit businesses to engage in real-time and informed
decision-making and are key instruments in data interpretation. First of all, let’s find a
definition to understand what lies behind data interpretation meaning.

What Is Data Interpretation?

Data interpretation refers to the process of using diverse analytical methods to review data
and arrive at relevant conclusions. The interpretation of data helps researchers to
categorize, manipulate, and summarize the information in order to answer critical
questions.

The importance of data interpretation is evident and this is why it needs to be done
properly. Data is very likely to arrive from multiple sources and has a tendency to enter the
analysis process with haphazard ordering. Data analysis tends to be extremely subjective.
That is to say, the nature and goal of interpretation will vary from business to business,
likely correlating to the type of data being analyzed. While there are several different types
of processes that are implemented based on individual data nature, the two broadest and
most common categories are “quantitative analysis” and “qualitative analysis”.

Yet, before any serious data interpretation inquiry can begin, it should be understood that
visual presentations of data findings are irrelevant unless a sound decision is made
regarding scales of measurement. Before any serious data analysis can begin, the scale
of measurement must be decided for the data as this will have a long-term impact on data
interpretation ROI. The varying scales include:

 Nominal Scale: non-numeric categories that cannot be ranked or compared quantitatively.

Variables are exclusive and exhaustive.
 Ordinal Scale: exclusive categories that are exclusive and exhaustive but with a logical order.
Quality ratings and agreement ratings are examples of ordinal scales (i.e., good, very good, fair,
etc., OR agree, strongly agree, disagree, etc.).
 Interval: a measurement scale where data is grouped into categories with orderly and equal
distances between the categories. There is always an arbitrary zero point.
 Ratio: contains features of all three.
For a more in-depth review of scales of measurement, read our article on data analysis
questions. Once scales of measurement have been selected, it is time to select which of
the two broad interpretation processes will best suit your data needs. Let’s take a closer
look at those specific data interpretation methods and possible data interpretation
problems.

How To Interpret Data?

When interpreting data, an analyst must try to discern the differences between correlation,
causation, and coincidences, as well as much other bias – but he also has to consider all
the factors involved that may have led to a result. There are various data interpretation
methods one can use.

The interpretation of data is designed to help people make sense of numerical data that
has been collected, analyzed, and presented. Having a baseline method (or methods) for
interpreting data will provide your analyst teams with a structure and consistent
foundation. Indeed, if several departments have different approaches to interpret the same
data while sharing the same goals, some mismatched objectives can result. Disparate
methods will lead to duplicated efforts, inconsistent solutions, wasted energy, and
inevitably – time and money. In this part, we will look at the two main methods of
interpretation of data: a qualitative and quantitative analysis.

Qualitative Data Interpretation

Qualitative data analysis can be summed up in one word – categorical. With qualitative
analysis, data is not described through numerical values or patterns, but through the use
of descriptive context (i.e., text). Typically, narrative data is gathered by employing a wide
variety of person-to-person techniques. These techniques include:
 Observations: detailing behavioral patterns that occur within an observation group. These
patterns could be the amount of time spent in an activity, the type of activity, and the method of
communication employed.
 Focus groups: Group people and ask them relevant questions to generate a collaborative
discussion about a research topic.
 Secondary Research: much like how patterns of behavior can be observed, different types of
documentation resources can be coded and divided based on the type of material they contain.
 Interviews: one of the best collection methods for narrative data. Inquiry responses can be
grouped by theme, topic, or category. The interview approach allows for highly-focused data
segmentation.

A key difference between qualitative and quantitative analysis is clearly noticeable in the
interpretation stage. Qualitative data, as it is widely open to interpretation, must be
“coded” so as to facilitate the grouping and labeling of data into identifiable themes. As
person-to-person data collection techniques can often result in disputes pertaining to
proper analysis, qualitative data analysis is often summarized through three basic
principles: notice things, collect things, think about things.

Quantitative Data Interpretation

If quantitative data interpretation could be summed up in one word (and it really can’t) that
word would be “numerical.” There are few certainties when it comes to data analysis, but
you can be sure that if the research you are engaging in has no numbers involved, it is not
quantitative research. Quantitative analysis refers to a set of processes by which
numerical data is analyzed. More often than not, it involves the use of statistical modeling
such as standard deviation, mean and median. Let’s quickly review the most common
statistical terms:

 Mean: a mean represents a numerical average for a set of responses. When dealing with a data
set (or multiple data sets), a mean will represent a central value of a specific set of numbers. It is
the sum of the values divided by the number of values within the data set. Other terms that can be
used to describe the concept are arithmetic mean, average and mathematical expectation.
 Standard deviation: this is another statistical term commonly appearing in quantitative analysis.
Standard deviation reveals the distribution of the responses around the mean. It describes the
degree of consistency within the responses; together with the mean, it provides insight into data
sets.
 Frequency distribution: this is a measurement gauging the rate of a response appearance
within a data set. When using a survey, for example, frequency distribution has the capability of
determining the number of times a specific ordinal scale response appears (i.e., agree, strongly
agree, disagree, etc.). Frequency distribution is extremely keen in determining the degree of
consensus among data points.

Typically, quantitative data is measured by visually presenting correlation tests between

two or more variables of significance. Different processes can be used together or
separately, and comparisons can be made to ultimately arrive at a conclusion. Other
signature interpretation processes of quantitative data include:

 Regression analysis: Essentially, regression analysis uses historical data to understand the
relationship between a dependent variable and one or more independent variables. Knowing
which variables are related and how they developed in the past allows you to anticipate possible
outcomes and make better decisions going forward. For example, if you want to predict your sales
for next month you can use regression analysis to understand what factors will affect them such
as products on sale, the launch of a new campaign, among many others.
 Cohort analysis: This method identifies groups of users who share common characteristics
during a particular time period. In a business scenario, cohort analysis is commonly used to
understand different customer behaviors. For example, a cohort could be all users who have
signed up for a free trial on a given day. An analysis would be carried out to see how these users
behave, what actions they carry out, and how their behavior differs from other user groups.

 Predictive analysis: As its name suggests, the predictive analysis method aims to predict future
developments by analyzing historical and current data. Powered by technologies such as artificial
intelligence and machine learning, predictive analytics practices enable businesses to spot trends
or potential issues and plan informed strategies in advance.

 Prescriptive analysis: Also powered by predictions, the prescriptive analysis method uses
techniques such as graph analysis, complex event processing, neural networks, among others, to
try to unravel the effect that future decisions will have in order to adjust them before they are
actually made. This helps businesses to develop responsive, practical business strategies.

 Conjoint analysis: Typically applied to survey analysis, the conjoint approach is used to analyze
how individuals value different attributes of a product or service. This helps researchers and
businesses to define pricing, product features, packaging, and many other attributes. A common
use is menu-based conjoint analysis in which individuals are given a “menu” of options from which
they can build their ideal concept or product. Like this analysts can understand which attributes
they would pick above others and drive conclusions.

 Cluster analysis: Last but not least, cluster analysis is a method used to group objects into
categories. Since there is no target variable when using cluster analysis, it is a useful method to
find hidden trends and patterns in the data. In a business context clustering is used for audience
segmentation to create targeted experiences, and in market research, it is often used to identify
age groups, geographical information, earnings, among others.

Now that we have seen how to interpret data, let's move on and ask ourselves some
questions: what are some data interpretation benefits? Why do all industries engage in
data research and analysis? These are basic questions, but they often don’t receive
adequate attention.

Why Data Interpretation Is Important

The purpose of collection and interpretation is to acquire useful and usable information and to
make the most informed decisions possible. From businesses to newlyweds researching their first
home, data collection and interpretation provides limitless benefits for a wide range of institutions
and individuals.
 Data analysis and interpretation, regardless of the method and qualitative/quantitative status, may
include the following characteristics:

 Data identification and explanation

 Comparing and contrasting of data
 Identification of data outliers
 Future predictions

Data analysis and interpretation, in the end, help improve processes and identify problems. It is
difficult to grow and make dependable improvements without, at the very least, minimal data
collection and interpretation. What is the keyword? Dependable. Vague ideas regarding
performance enhancement exist within all institutions and industries. Yet, without proper research
and analysis, an idea is likely to remain in a stagnant state forever (i.e., minimal growth). So…
what are a few of the business benefits of digital age data analysis and interpretation? Let’s take
a look!

1) Informed decision-making: A decision is only as good as the knowledge that formed it.
Informed data decision-making has the potential to set industry leaders apart from the rest of the
market pack. Studies have shown that companies in the top third of their industries are, on
average, 5% more productive and 6% more profitable when implementing informed data decision-
making processes. Most decisive actions will arise only after a problem has been identified or a
goal defined. Data analysis should include identification, thesis development, and data collection
followed by data communication.

If institutions only follow that simple order, one that we should all be familiar with from grade
school science fairs, then they will be able to solve issues as they emerge in real-time. Informed
decision-making has a tendency to be cyclical. This means there is really no end, and eventually,
new questions and conditions arise within the process that needs to be studied further. The
monitoring of data results will inevitably return the process to the start with new data and sights.

2) Anticipating needs with trends identification: data insights provide knowledge, and
knowledge is power. The insights obtained from market and consumer data analyses have the
ability to set trends for peers within similar market segments. A perfect example of how data
analysis can impact trend prediction can be evidenced in the music identification application

When industry trends are identified, they can then serve a greater industry purpose.

3) Cost efficiency: Proper implementation of data analysis processes can provide businesses
with profound cost advantages within their industries. A recent data study performed by Deloitte
vividly demonstrates this in finding that data analysis ROI is driven by efficient cost reductions.
Often, this benefit is overlooked because making money is typically viewed as “sexier” than
saving money. Yet, sound data analyses have the ability to alert management to cost-reduction
opportunities without any significant exertion of effort on the part of human capital.

A great example of the potential for cost efficiency through data analysis is Intel. Prior to 2012,
Intel would conduct over 19,000 manufacturing function tests on their chips before they could be
deemed acceptable for release. To cut costs and reduce test time, Intel implemented predictive
data analyses. By using historic and current data, Intel now avoids testing each chip 19,000 times
by focusing on specific and individual chip tests. After its implementation in 2012, Intel saved
over $3 million in manufacturing costs. Cost reduction may not be as “sexy” as data profit, but as
Intel proves, it is a benefit of data analysis that should not be neglected.

4) Clear foresight: companies that collect and analyze their data gain better knowledge about
themselves, their processes, and performance. They can identify performance challenges when
they arise and take action to overcome them. Data interpretation through visual representations
 lets them process their findings faster and make better-informed decisions on the future of the
company.

Common Data Analysis And Interpretation Problems

The oft-repeated mantra of those who fear data advancements in the digital age is “big data
equals big trouble.” While that statement is not accurate, it is safe to say that certain data
interpretation problems or “pitfalls” exist and can occur when analyzing data, especially at the
speed of thought. Let’s identify some of the most common data misinterpretation risks and shed
some light on how they can be avoided:

1) Correlation mistaken for causation: our first misinterpretation of data refers to the tendency
of data analysts to mix the cause of a phenomenon with correlation. It is the assumption that
because two actions occurred together, one caused the other. This is not accurate as actions can
occur together absent a cause and effect relationship.

 Digital age example: assuming that increased revenue is the result of increased social media followers…
there might be a definitive correlation between the two, especially with today’s multi-channel purchasing
experiences. But, that does not mean an increase in followers is the direct cause of increased revenue.
There could be both a common cause or an indirect causality.
 Remedy: attempt to eliminate the variable you believe to be causing the phenomenon.

2) Confirmation bias: our second data interpretation problem occurs when you have a theory or
hypothesis in mind but are intent on only discovering data patterns that provide support to it while
rejecting those that do not.

 Digital age example: your boss asks you to analyze the success of a recent multi-platform social media
marketing campaign. While analyzing the potential data variables from the campaign (one that you ran and
believe performed well), you see that the share rate for Facebook posts was great, while the share rate for
Twitter Tweets was not. Using only the Facebook posts to prove your hypothesis that the campaign was
successful would be a perfect manifestation of confirmation bias.
 Remedy: as this pitfall is often based on subjective desires, one remedy would be to analyze data with a
team of objective individuals. If this is not possible, another solution is to resist the urge to make a
conclusion before data exploration has been completed. Remember to always try to disprove a hypothesis,
not prove it.
 3) Irrelevant data: the third data misinterpretation pitfall is especially important in the digital age.
As large data is no longer centrally stored, and as it continues to be analyzed at the speed of
thought, it is inevitable that analysts will focus on data that is irrelevant to the problem they are
trying to correct.

 Digital age example: in attempting to gauge the success of an email lead generation campaign, you notice
that the number of homepage views directly resulting from the campaign increased, but the number of
monthly newsletter subscribers did not. Based on the number of homepage views, you decide the
campaign was a success when really it generated zero leads.
 Remedy: proactively and clearly frame any data analysis variables and KPIs prior to engaging in a data
review. If the metric you are using to measure the success of a lead generation campaign is newsletter
subscribers, there is no need to review the number of homepage visits. Be sure to focus on the data
variable that answers your question or solves your problem and not on irrelevant data.

4) Truncating an Axes: When creating a graph to start interpreting the results of your analysis it
is important to keep the axes truthful and avoid generating misleading visualizations. Starting
the axes in a value that doesn’t portray the actual truth about the data can lead to false
conclusions.

 Digital age example: In the image below we can see a graph from Fox News in which the Y-axes start at
34%, making it seem that the difference between 35% and 39.6% is way higher than it actually is. This
could lead to a misinterpretation of the tax rate changes.

 Remedy: Be careful with the way your data is visualized. Be respectful and realistic with axes to avoid
misinterpretation of your data. See below how the Fox News chart looks when using the correct axes
values.

5) (Small) sample size: Another common data analysis and interpretation problem is the use of a
small sample size. Logically, the bigger the sample size the most accurate and reliable are the
results. However, this also depends on the size of the effect of the study. For example, the
sample size in a survey about the quality of education will not be the same as for one about
people doing outdoor sports in a specific area.
 Digital age example: Imagine you ask 30 people a question and 29 answer “yes” resulting in 95% of the
total. Now imagine you ask the same question to 1000 and 950 of them answer “yes”, which is again 95%.
While these percentages might look the same, they certainly do not mean the same thing as a 30 people
sample size is not a significant number to establish a truthful conclusion.
 Remedy: Researchers say that in order to determine the correct sample size to get truthful and meaningful
results it is necessary to define a margin of error that will represent the maximum amount they want the
results to deviate from the statistical mean. Paired to this, they need to define a confidence level that
should be between 90 and 99%. With these two values in hand, researchers can calculate an accurate
sample size for their studies.

6) Reliability, subjectivity, and generalizability: When performing qualitative analysis,

researchers must consider practical and theoretical limitations when interpreting the data. In some
cases, qualitative research can be considered unreliable because of uncontrolled factors that
might or might not affect the results. This is paired with the fact that the researcher has a primary
role in the interpretation process, meaning he or she decides what is relevant and what is not, and
as we know, interpretations can be very subjective.

Generalizability is also an issue that researchers face when dealing with qualitative analysis. As
mentioned in the point about small sample size, it is difficult to draw conclusions that are 100%
representative because the results might be biased or unrepresentative of a wider population.

While these factors are mostly present in qualitative research, they can also affect quantitative
analysis. For example, when choosing which KPIs to portray and how to portray them, analysts
can also be biased and represent them in a way that benefits their analysis.

 Digital age example: Biased questions in a survey are a great example of reliability and subjectivity issues.
Imagine you are sending a survey to your clients to see how satisfied they are with your customer service
with this question: “how amazing was your experience with our customer service team?”. Here we can see
that this question is clearly influencing the response of the individual by putting the word “amazing” on it.
 Remedy: A solution to avoid these issues is to keep your research honest and neutral. Keep the wording of
the questions as objective as possible. For example: “on a scale of 1-10 how satisfied were you with our
customer service team”. This is not leading the respondent to any specific answer, meaning the results of
your survey will be reliable.

Data Interpretation Techniques and Methods

Data analysis and interpretation are critical to developing sound conclusions and making better-
informed decisions. As we have seen with this article, there is an art and science to the
 interpretation of data. To help you with this purpose here we will list a few relevant data
interpretation techniques, methods, and tricks you can implement for a successful data
management process.

As mentioned at the beginning of this post, the first step to interpret data in a successful way is to
identify the type of analysis you will perform and apply the methods respectively. Clearly
differentiate between qualitative analysis (observe, document, and interview notice, collect and
think about things) and quantitative analysis (you lead research with a lot of numerical data to be
analyzed through various statistical methods).

1) Ask the right data interpretation questions

The first data interpretation technique is to define a clear baseline for your work. This can be done
by answering some critical questions that will serve as a useful guideline to start. Some of them
include: what are the goals and objectives from my analysis? What type of data interpretation
method will I use? Who will use this data in the future? And most importantly, what general
question am I trying to answer?

Once all this information has been defined, you will be ready to collect your data. As mentioned at
the beginning of the post, your methods for data collection will vary depending on what type of
analysis you use (qualitative or quantitative). With all the needed information in hand, you are
ready to start the interpretation process, but first, you need to visualize your data.

2) Use the right data visualization type

Data visualizations such as business graphs, charts, and tables are fundamental to successfully
interpreting data. This is because the visualization of data via interactive charts and graphs
makes the information more understandable and accessible. As you might be aware, there are
different types of visualizations you can use but not all of them are suitable for any analysis
purpose. Using the wrong graph can lead to misinterpretation of your data so it’s very important to
carefully pick the right visual for it. Let’s look at some use cases of common data visualizations.

 Bar chart: One of the most used chart types, the bar chart uses rectangular bars to show the relationship
between 2 or more variables. There are different types of bar charts for different interpretations this
includes the horizontal bar chart, column bar chart, and stacked bar chart.
 Line chart: Most commonly used to show trends, acceleration or decelerations, and volatility, the line
chart aims to show how data changes over a period of time for example sales over a year. A few tips to
keep this chart ready for interpretation is to not use many variables that can overcrowd the graph and keep
your axis scale close to the highest data point to avoid making the information hard to read.
 Pie chart: Although it doesn’t do a lot in terms of analysis due to its uncomplex nature, pie charts are
widely used to show the proportional composition of a variable. Visually speaking, showing a percentage in
a bar chart is way more complicated than showing it in a pie chart. However, this also depends on the
number of variables you are comparing. If your pie chart would need to be divided into 10 portions then it is
better to use a bar chart instead.
 Tables: While they are not a specific type of chart, tables are wildly used when interpreting data. Tables
are especially useful when you want to portray data in its raw format. They give you the freedom to easily
look up or compare individual values while also displaying grand totals.

With the use of data visualizations becoming more and more critical for businesses’ analytical
success, many tools have emerged to help users visualize their data in a cohesive and interactive
way. One of the most popular ones is the use of BI dashboards. These visual tools provide a
centralized view of various graphs and charts that paint a bigger picture about a topic. We will
discuss more the power of dashboards for an efficient data interpretation practice in the next
portion of this post. If you want to learn more about different types of data visualizations take a
look at our complete guide on the topic.
3) Keep your interpretation objective
As mentioned above, keeping your interpretation objective is a fundamental part of the process.
Being the person closest to the investigation, it is easy to become subjective when looking for
answers in the data. Some good ways to stay objective is to show the information to other people
related to the study, for example, research partners or even the people that will use your findings
once they are done. This can help avoid confirmation bias and any reliability issues with your
interpretation.

4) Mark your findings and draw conclusions

Findings are the observations you extracted out of your data. They are the facts that will
help you drive deeper conclusions about your research. For example, findings can be trends
and patterns that you found during your interpretation process. To put your findings into
perspective you can compare them with other resources that used similar methods and use
them as benchmarks.

Correlation versus causation, subjective bias, false information, and inaccurate data, etc.
Once you are comfortable with your interpretation of the data you will be ready to develop
conclusions, see if your initial question were answered, and suggest recommendations based
on them.

Interpretation of Data: The Use of Dashboards Bridging the Gap

As we have seen, quantitative and qualitative methods are distinct types of data analyses. Both
offer a varying degree of return on investment (ROI) regarding data investigation, testing, and
decision-making. Because of their differences, it is important to understand how dashboards can
be implemented to bridge the quantitative and qualitative information gap. How are digital data
dashboard solutions playing a key role in merging the data disconnect? Here are a few of the
ways:

1) Connecting and blending data. With today’s pace of innovation, it is no longer feasible (nor
desirable) to have bulk data centrally located. As businesses continue to globalize and borders
continue to dissolve, it will become increasingly important for businesses to possess the capability
to run diverse data analyses absent the limitations of location. Data dashboards decentralize data
without compromising on the necessary speed of thought while blending both quantitative and
qualitative data. Whether you want to measure customer trends or organizational performance,
you now have the capability to do both without the need for a singular selection.

2) Mobile Data. Related to the notion of “connected and blended data” is that of mobile data. In
today’s digital world, employees are spending less time at their desks and simultaneously
increasing production. This is made possible by the fact that mobile solutions for analytical tools
are no longer standalone. Today, mobile analysis applications seamlessly integrate with everyday
business tools. In turn, both quantitative and qualitative data are now available on-demand where
they’re needed, when they’re needed, and how they’re needed via interactive online dashboards.

3) Visualization. Data dashboards are merging the data gap between qualitative and quantitative
methods of interpretation of data, through the science of visualization. Dashboard solutions come
“out of the box” well-equipped to create easy-to-understand data demonstrations. Modern online
data visualization tools provide a variety of color and filter patterns, encourage user interaction,
and are engineered to help enhance future trend predictability. All of these visual characteristics
make for an easy transition among data methods – you only need to find the right types of data
visualization to tell your data story the best way possible.
To give you an idea of how a market research dashboard fulfills the need of bridging quantitative
and qualitative analysis and helps in understanding how to interpret data in research thanks to
visualization, have a look at the following one. It brings together both qualitative and quantitative
data knowledgeably analyzed and visualizes it in a meaningful way that everyone can
understand, thus empowering any viewer to interpret it:

Deployment and Iteration:

Iterative Development Overview
DSDM(Dynamics System Development Method)

Iterative development is a process in which the Evolving Solution, or a part of it,

evolves from a high-level concept to something with acknowledged business
value.

Each cycle of the process is intended to bring the part of the solution being
worked on closer to completion and is always a collaborative process, typically
involving two or more members of the Solution Development Team.

Each cycle should:

 Be as short as possible, typically taking a day or two, with several cycles

happening within a Timebox
 Be only as formal as it needs to be - in most cases limited to an informal
cycle of Thought, Action and Conversation
 Involve the appropriate members of the Solution Development Team
relevant to the work being done. At its simplest, this could be, for example,
a Solution Developer and a Business Ambassador working together, or it
could need involvement from the whole Solution Development Team
including several Business Advisors

Each cycle begins and ends with a conversation (in accordance with DSDM’s
Principles collaborate and communicate continuously and clearly). The initial
conversation is focussed on the detail of what needs to be done. The cycle
continues with thought - a consideration of how the need will be addressed. At
its most formal, this may be a collaborative planning event, but in most cases
thought will be limited to a period of reflection and very informal planning.
Action then refines the Evolving Solution or feature of it. Where appropriate,
action will be collaborative. Once work is completed to the extent that it can
sensibly be reviewed, the cycle concludes with a return to conversation to decide
whether what has been produced is good enough or whether another cycle is
needed. Dependent on the organization and the nature of the work being
undertaken, this conversation could range from an informal agreement to a
formally documented demonstration, or a “show and tell” review with a wider
group of stakeholders.
As Iterative Development proceeds, it is important to keep the agreed acceptance
criteria for the solution, or the feature of it, in clear focus in order to ensure that
the required quality is achieved without the solution becoming over-engineered.
An agreed timescale for a cycle of evolution may also help maintain focus,
promote collaboration and reduce risk of wasted effort.

11.2 Planning Iterative Development

During Foundations, it is very important to decide on a strategy for development

that encompasses how the potentially large problem of evolving a solution can
be broken down into manageable chunks for delivery in Timeboxes. Two ways
in which this may be considered. The first describes a requirement focus, the
second describes a solution focus. It is important to note that neither is better
than the other and equally valid alternative ways of approaching Iterative
Development may be available; the key is to find the most appropriate
development strategy for the project through discussion. DSDM does not dictate
how the strategy should be formed and agreed, just that there should be a
strategy for development.
11.2.1 Requirement focus

DSDM identifies that the requirements can be of three different types:

 Functional
 Usability
 Non-functional

(Note that traditionally, usability is considered to be a type of non-functional

requirement but is elevated into a class of its own because of its importance to
the business user of the final solution and to facilitate business interaction with
the rest of the Solution Development Team in this important aspect of solution
design)

An individual cycle of Iterative Development or even the work of a Timebox

may focus on evolving the solution to meet one or more of these requirement
types.

On a simple feature, a cycle may encompass all three perspectives at the same
time. However, where iterative development of a feature involves many cycles,
involving several different people, the team may decide to focus a cycle on one
or perhaps two specific perspectives rather than covering all of them at the same
time .
 For example:

The team may decide to focus early cycles on the functional perspective
of a requirement – ensuring, and demonstrating, that the detail of the
requirement is properly understood and agreed. This may be followed by
cycles focussed on usability – ensuring interaction with the solution is
effective and efficient. Later cycles may then focus on ensuring the
required non-functional quality (e.g. performance or robustness) is
achieved, that all tests pass as expected, and all acceptance criteria for
the feature are met.

11.2.2 Solution focus

DSDM identifies that the solution may be considered to have a
number of architectural layers. The example used here relates to a
business system development but the concept can also be applied to a
non-IT project, such as a marketing campaign.

Iterative Development may follow an approach in which Timeboxes

deliver horizontal slices of the solution, vertical slices or a combination
of the two.

Horizontal Approach

The horizontal approach considers the solution layer by layer

with each Timebox incrementally delivering increased
complexity of business process or layers of
complexity/completeness as regards technology.

The advantage of the horizontal approach is that it allows an

initial sight of the full breadth of the solution very early on.
The disadvantage is that nothing works fully until the last
horizontal slice is delivered.
Therefore no business benefit can accrue until that point.

For example:

Projects where a horizontal approach may be appropriate:

 Project H1 is delivering a corporate intranet. At the end of the

first Timebox, the business want to see a thin but complete
first cut of the solution that demonstrates the full breadth of
what will be delivered. Subsequent Timeboxes deliver
additional layers of more detailed information
 Project H2 is using a mixture of technologies . It is agreed to
develop the user interface first and then to add business logic
and finally to hook the solution into a custom-designed
database.
Vertical Approach

The vertical approach slices through multiple layers of the

solution with each Timebox delivering one or more fully
functional features.
The advantage of the vertical approach is that delivery of prioritised
features may enable Solution Increments to be more quickly and
frequently deployed into live use. The disadvantage is that the full
breadth of the solution is not clear until late in the project.

For example:
 Projects where a vertical approach may be appropriate:

 Project V1 is delivering a simple ecommerce solution and wants to

have a fully working group of requirements available at the end of
each Timebox ready for deployment. For example, a first timebox
making items available for purchase visible to the user, a second
Timebox allowing items to be selected and put in a basket etc
 Project V2 is enhancing an existing system by adding new features
with each new feature being fully built and tested within the
boundaries of a single Timebox and then immediately deployed
into live use.

Combined Approach

The combined approach starts by focusing one or two Timeboxes on

(thin) horizontal slices of the solution in order to fully understand the
breadth of the solution before reverting to a vertical approach to
incrementally deliver fully functional features.

The advantages of the combined approach are that there is an initial

view of the overall solution early in the project, and incremental
delivery of business value into live use is also achieved. There are no
obvious disadvantages.

For example:
 Projects where a combined approach may be appropriate:
 Project C1 is a business change project in the
operations department of a company that is planning to
simultaneously implement new business processes
supported by new technology. Early Timeboxes focus
on the design of the end-to-end business process that,
whilst subject to change at the detail level, will act as
an anchor for the development of the supporting
technology. Subsequent Timeboxes deliver the
technology to support the process and the overall
solution (new process and new technology) is deployed
into live use.
 Project C2 is automating an existing business process.
The first Timebox prototypes simple user screens to
show how the new system will support the end-to-end
process. Having got this accepted, subsequent
timeboxes can deliver working components against the
context of the agreed framework.

11.3 Controlling Iterative Development

Each Iterative Development cycle is intended to result in an incremental

change to the Evolving Solution that brings it, or more probably a
feature of it, progressively closer to being complete or “done”.

It is quite possible, however, that the review of such an incremental

change may reveal that the solution has evolved in a way that is not
right. Under such circumstances it is important, wherever possible, to
have the option to revert to the previously agreed version and undertake
another cycle based on what has been learnt and on any new insight
arising from the review. Configuration Management of the solution is
therefore an important consideration.
11.4 Quality in Iterative Development

One of the defining principles of DSDM is to never compromise quality. To achieve this, we
should (amongst other things) define a level of quality in the early lifecycle phases and then
ensure that this is achieved. The challenge then is how to meaningfully define quality and
then measure it in an iterative context.

11.4.1 Quality criteria

Quality criteria need to deal with required characteristics of the product/feature with these
being driven by the context in which the product/feature is going to be used.

For example:

For a house brick

 Its physical dimensions and ability to bear load are clearly important
characteristics
 If the dimensions are inconsistent (too long or too short) then a uniform wall
cannot be built from a random sample of bricks
 If the load-bearing capacity is not sufficient then the building will be unsafe
 If the load-bearing capacity exceeds the need then it is likely that the bricks
will cost more to make In terms of context, bricks for use in colder climates
need to be thicker than those used in temperate ones.

For example:

For a software application

 A key characteristic may be the ability to carry out business processes

within specified time constraints (e.g. a 3-second response time) and
perform calculations to a high level of accuracy (e.g. +/- 1% accuracy).
 In terms of context, web sites selling tickets suffer spikes of extreme load
when a major event goes on sale and need to be extra resilient because of
this.

11.4.2 Acceptance criteria

By the time Iterative Development of the solution starts, the main deliverables will already
have some acceptance criteria associated with them from the Foundations phase. Although it
might not be practical or even appropriate to get to this level of detail during Foundations,
by the time development of a particular feature starts acceptance criteria should be objective
and quantified (rather than subjective).

For example:

An ‘on-line purchase’ feature would need a defined set of inputs (e.g. product
codes and purchase volumes), a planned response (e.g. calculating and
displaying the item and total costs whilst separately calculating tax) and the
 context within which this is happening (e.g. checking the stock levels needed to
fulfil the request).

If the acceptance criteria are vague or subjective (as may be the case at the end of
Foundations) then more conversation is needed to agree on the specific details.

Note that this information informs both what needs to be built and how it will be assessed,
so it is essential it is done before work starts.

Thought is applied to both how the solution is built and how to verify
that it has been built correctly. Where the DSDM structured timebox is
used (Chapter 13.3), the detailed work on acceptance criteria takes
place primarily in the Investigation step. Where a less formal structure
is used, it takes place as a first step in addressing a particular
requirement once it has been selected, whenever that occurs within the
Timebox.
11.4.3 Validation and verification

Validation asks ‘Are we building the right thing?’ whilst verification asks ‘Are we building
the thing right?’

In an Iterative Development context, validation does not need to be a separate activity as the
process of collaborative design of the solution with direct involvement of Business
Ambassador and Business Advisor roles means this happens naturally. However,
verification activity still needs to be explicitly considered, to ensure it is fully integrated in
the Iterative Development cycle. How this will be achieved is part of the development
approach and should be described in the Development Approach Definition if appropriate.

Having agreed how the quality will be verified action ensures verification is carried out
effectively and efficiently. The person responsible for producing the product will naturally
carry out his or her own assessments as part of that development activity. Simultaneously, a
separate person (in most cases the Solution Tester) needs to prepare for the independent
verification activity. This can be just as time-consuming as making the product (in some
cases more time-consuming).

11.4.4 Static and dynamic verification (reviews and testing)

There are two broad classes of verification – static and dynamic. Static verification involves
inspecting a deliverable against its acceptance criteria or agreed good practice. The
advantage of this type of verification is that it is based on observation alone and so could not
cause harm to the product being inspected.

Some items can only be statically verified.

For example:
 Documents, or the dimension of a house brick

Other items might be verified statically and/or dynamically.

For example:

The components of an engine can be reviewed against the

blue print to statically test them. Running the engine allows
it to be dynamically tested.

As static methods present no risk that the deliverable will be damaged by the process
(presuming non-invasive methods), there are potential advantages to inspection even when
an item could be tested dynamically.

Static verification - reviews

Reviews can range from informal peer reviews through to highly structured and formal
reviews involving experts or perhaps groups of people. The level of formality is often driven
by the nature of the product and by corporate or regulatory standards.

Some types of static testing can be automated.

For example:

On an IT project, code scanners can check that agreed

coding standards have been followed. This can be
particularly useful for initially checking security aspects of a
solution.

The Technical Coordinator is responsible for the technical quality of the solution by
“ensuring adherence to appropriate standards of technical best practice” and so should
ensure that:

 Such practices are appropriate and understood by all

 An appropriate regime of peer review, with an appropriate level of formality, is part
of the teams working practice
 Appropriate, contemporary evidence of review activity is captured as required

Whilst the rigour of a review can vary, mostly they share certain key qualities:

1. There are key roles that need to be fulfilled:

 The producer(s) of the item being reviewed (author(s) in the case of a document)
 The reviewers
 A review moderator, where appropriate, for very formal reviews

2. All reviews require time to be carried out. A simple, informal review may be peers
gathering at a desk and reviewing the item together, whereas a formal review needs to
properly planned, preferably as a Facilitated Workshop
3. Every review involves assessing the product against criteria, which may be specific to that
item (defined as acceptance criteria) or general to an item of that kind (general standards or
good practice such as those defined by the development approach). The criteria need to be
agreed (and probably documented) in advance to gain the most benefit, but they can of
course also evolve over time to accommodate the current situation and appropriate
innovation in working practice

4. Every review must reach a conclusion. Commonly there are three potential results:

 The item is fit for purpose and no fur ther action is required
 The item needs minor amendments to make it fit for purpose. In this case, the review
group might nominate one individual to check that required changes are made
 The item needs major amendments before it is fit for purpose. In this case the item
typically needs to be fully reviewed again after being reworked

Reviews may occasionally result in no clear outcome. In this case the people involved need
to collaborate and if necessary bring other people in to the discussion in order to reach an
agreed outcome.

11.4.5 Are we ‘done’ yet?

After verification has been under taken (whether statically or dynamically) the key question
is whether or not the acceptance criteria have been met in a meaningful way. It may be
obvious that the criteria have clearly been met or have clearly failed. In other cases more
conversation is needed to decide whether the team are confident that the solution is fit for
purpose or not, based on what has been observed. This could imply that the acceptance
criteria were not sufficiently understood or defined. Alternatively, even though all the
‘Must’ criteria may have been met, the product may have failed against so many other,
lesser criteria that it is unlikely it will actually deliver the benefit needed from it.

If the solution has unconditionally met all the acceptance criteria, then it is ‘done’. Where
only some of the criteria have been met, then the product may need to be evolved further to
ensure that more criteria are met next time it is validated. More conversation will be needed,
followed in due course by more thought and action to implement what has been agreed
upon.

Once an item is agreed as ‘done’ it is advisable to record any discretionary acceptance

criteria which remained unfulfilled. This information is valuable as:

 It may have a knock-on effect on other parts of the solution or work yet to be
done
 The team may choose to spend time later on fixing these criteria rather than
implementing lower-value features

Remember that an accumulation of less serious defects may eventually have an impact on
the Business Case which is not clearly shown by any one failed criterion in isolation.
11.5 Summary

Iterative Development in a project context needs up-front thought. However this is not about
big design up-front (BDUF) or detailed planning. It is more a consideration of the strategy
for development. The DSDM philosophy states that projects deliver frequently so the ‘big
picture’ needs to be broken down into small manageable chunks to facilitate this frequent
delivery.

The principles: focus on the business need, deliver on time, collaborate and never
compromise quality must also be considered as these drive how the Solution Development
Team works. How work is planned at the detailed level, ensuring the right people are
involved at the right time and assuring quality as development progresses, requires the
whole team to be bought in to a sensible strategy for development that they help shape
during the Foundations phase. Where appropriate, this will be documented in the
Development Approach Definition.

For small, simple projects delivering conceptually simple solutions, consideration of these
issues may take an hour or two and be based on simple conversation and ‘round the table’
agreement. However, as a general rule, the need for a more formal and more carefully
considered Iterative Development strategy increases with the size and complexity of the
project organization and the solution to be created.

Quality assurance is a key part of delivering a solution that is fit for business purpose.
However, the formality and rigour of testing will depend very much on the nature of the
project and the standards laid down by the organisation

UNIT II BUSINESS INTELLIGENCE 9

Data Warehouses and Data Mart - Knowledge Management – Types of Decisions -
Decision Making Process - Decision Support Systems – Business Intelligence –
OLAP –, Analytic functions
BUSINESS INTELLIGENCE
Business Intelligence (BI):
(BI) comprises wide variety of applications for analysing, gathering, storing and making data easily
accessible to help users to make better business processes. A good Business Intelligence (BI) definition
must encompass both business purpose and technical functionality. Business Intelligence (BI) tools that are
widely used are Data Warehouse, Data Mining, Extraction Transformation Load (ETL) and On-Line
Analytical Processing (OLAP) (See Figure 1).
Business Layer- Program Management
This component of the business layer ensures that people, projects, and priorities work in a
manner in which individual processes are compatible with each other so as to ensure
seamless integration and smooth functioning of the entire program. It should attend to each
of the following:
• Business priorities
• Mission and goals
• Strategies and risks• Strategies and risks
• Multiple projects
• Dependencies
• Cost and value
• Business rules
• Infrastructure

Business Layer- Development

The process of development consists of

• database/data-warehouse development (consisting of ETL, data profiling, data
cleansing and database tools),
• data integration system development (consists of data integration tools and data
quality tools)
• business analytics development (about processes and various technologies
• Business analytics development (about processes and various technologies
used).

Data Warehouse and Data Marts:

Data Warehouse and Data Marts

Organizations are mainly focused on maintaining records of their current operations in the
form Relational Database Management System like Oracle, SQL Server, MySQL etc. but
with the progress of analytics and competition, managers are required to be highly proactive
and take decisions in real time. So, they need to be fully aware of not only current
operations but also of past trends and patterns followed by the organizations. Thus, they
need to manage Data Warehouse and /or Data Marts in their organizations, which are data
mined to give unknown facts of the past and help not only in current operations and
decisions but also in future plans and decisions. Data Mart is a repository of past databases
of an organization maintained department wise. So there is HR Mart, Sales Mart, Production
Mart etc. which have past years information of respective departments. These help managers
to dig out data of their respective departments and use for current.
There are three types of Data Marts-
 Dependent- has data from internal and/or external sources.
 Independent- makes no use of Data Warehouse for creation.
 Hybrid-can use data from operational databases or Data Warehouse

However, Data Warehouse is a past years database repository of entire organization that
helps managers in knowing past patterns of the entire company in a small time frame.
Data Warehouses are created in following two approaches-
 Top-Down Approach- It is a complex, time consuming and costly approach in which
databases of pat years are merged to give Data Warehouse.

 Bottom-up approach- in this approach first Data Marts are created and then are
gradually merged to give Data Warehouse. It is fast, simple and cheaper method.
Need for Data Warehouse:

Definition of Data Warehouse

Data Mining
It is a technique used by Analysts, Consultants and Managers to dig data from databases
repositories of organizations stored in form of Data Marts and/or Data Warehouses. It gives
reports in the form of summary reports, visualizations, data charts, patterns and findings
which are hidden and not known. These unknown facts add on to the knowledge of
managers and help them in taking better decisions not only for the present situations but also
for future plans. There are many It involves statistical methods along with machine learning,
Artificial Intelligence and IoT. A number of softwares are available as Data Miners in the
market that organizations use as per their need. Data Miners find clusters of information.
Some of them are Data Lake Storage on Amazon Web Service.

5.1. Data Mining Process

REFER NOTES:
KNOWLEDGE MANAGEMENT:

Knowledge Management (KM) is the collection of processes that govern the creation,
dissemination, and utilization of knowledge. (KM) this is, as the word entails the power to
handle “knowledge” and right knowledge available to the right people. It is about making
sure that an organization can learn, and that it will be able to retrieve and use its knowledge
assets in current applications as they are needed. In Peter Drucker paper, he defines
Knowledge Management (KM) as the coordination and exploitation of organizational
knowledge resources, in order to create benefit and competitive advantage. Knowledge
Management (KM) is not always about technology, but also about understanding how the
people work, brainstorming, identify groups of people who work together and how they can
share and learn from each other and in the end the organization learning about their workers
experience and about the leadership the organization (Arora, 2011).
In Rao and Kumar paper, they explain that Knowledge Management (KM) is the practice to
add actionable value to the information from the tacit to the explicit knowledge using
storing, filtering retrieving and disseminating explicit knowledge and also by testing and
creating new knowledge. "Knowledge Management (KM) will deliver outstanding
collaboration and partnership working. It will ensure the region maximizes the value of its
information and knowledge assets and it will help its citizens to use their creativity and
skills better, leading to improved effectiveness and greater innovation", West Midlands
Regional Observatory, UK.
 Kerschberg presents a conceptual model for knowledge
management systems composed of three layers (Figure 4). The first
layer is marked as Knowledge Presentation, the second – Knowledge
Management, and the third one is referred to as Data Sources layer.
The Presentation layer enables the workers to communicate,
collaborate and share knowledge. They obtain information through a
defined Knowledge Portal, which can be customized for every
employee.
 The Knowledge Management layer consists of a Knowledge
Repository, a process used for acquiring, cleansing, distributing and
representing knowledge and data integration services(Kerschberg).
 Kerschberg represents the Data Warehouse within the Data
Knowledge Management (Figure4), in the segment he refers to as
Knowledge Integration Services. Together with other services,
notably Data Mining, Knowledge Integration Services form a
knowledge repository which becomes available to employees in the
enterprise through various knowledge portals.
 White (2005) argues that the development of information
technologies has significantly contributed to the fact that knowledge
management, through knowledge management systems, has
become a significant resource of any organisation, and business
intelligence has acquired a highly important role in knowledge
management projects.
  Business intelligence applications provide the analysis of business
information, producing information that enables users to enhance
and optimise business operations. Information used in the decision-
making support domain can be obtained directly from the
transaction system (Figure 5), or more frequently, information is
obtained by processing data from the Data Warehouse database.
The Data Warehouse data repository is managed by the database
management system, using languages such as SQL (Structured
Query Language) for data access and manipulation White (2005).
 In his works, White (White, 2005) states that business intelligence
plays a central role in knowledge management (Figure 5). The same
author also places knowledge management in the context of
business process enhancement as well. For the traditional business
intelligence system to support knowledge management, what is
essential is their integrated functioning with business process
management software, planning and collaboration software, portals,
content management systems and other systems with similar
purpose, thereby enabling provision of timely information for
individual management levels.
 As one of the shortcomings of Data Warehouse databases, many authors
mention the existence of unnecessary, redundant, inaccurate or
incomplete data, and sometimes even incorrect data. Albescu, Pugna,
and Paraschiv (2008) claim, however, that despite these shortcomings, it
must be borne in mind that each piece of data, similar to chips in a jigsaw
puzzle, may be helpful, if they are integrated so as to get the complete
picture. Even if it is established that some of the data are missing, this
can often result in the idea of what the picture actually represents. These
authors refer to this procedure as knowledge creation. Further on,
Alabescu et al. (2008) write of two approaches to knowledge creation:

Successful knowledge management requires integration of databases, information systems

and knowledge-based systems, where Data Warehouse features as the integrating
component of these three elements (Erdmann). Knowledge retrieved by way of OLAP, Data
Mining and other techniques can be very useful if applied appropriately
Difference between business intelligence and knowledge management

The major difference between business intelligence and knowledge management is the
scope of activities involved in each area. Business intelligence focuses solely on capturing
data, manipulating the data and analyzing the data. Whereas knowledge management would
perform business intelligence activities while also pursuing the creation of new knowledge.
Decision Making Process & Types of Decisions:

Types of decisions
Decisions are part of the manager's remit. Difficult choices may have to be made for the
common good of the organisation. There are three types of decision in business:

 strategic
 tactical
 operational

Strategic decisions

Strategic decisions comprise the highest level of organizational business decisions and are
usually less frequent and made by the organization’s executives. Yet, their impact is
enormous and far-reaching.

Some types of strategic decisions include selecting a particular market to penetrate, a

company to acquire, or whether to hire additional staff.

Decisions made at this level usually involve significant expenditure. However, they are
generally non-repetitive in nature and are taken only after careful analysis and evaluation of
many alternatives.

Tactical decisions

Tactical decisions (or semistructured decisions) occur with greater frequency (e.g., weekly
or monthly) and fall into the mid-management level. Often, they relate to the
implementation of strategic decisions.

Examples of tactical decisions include product price changes, work schedules, departmental
reorganization, and similar activities.

The impact of these types of decisions is medium regarding risk to the organization and
impact on profitability.
Operational decisions

Operational decisions (or structured decisions) usually happen frequently (e.g., daily or
hourly), relate to day-to-day operations of the enterprise, and have a lesser impact on the
organization. Operational decisions determine the day-to-day profitability of the business,
how effectively it retains customers, or how well it manages risk.

Answering a sales inquiry, approving a quotation, or calculating employee bonuses may be

examples of this decision type.

You can summarize these types of decisions in business intelligence this way:

o Strategic: Long-term, complex, made by senior managers

o Tactical: Medium-term, less complex, made by mid-level managers
o Operational: Day-to-day, simple, routine, made by junior managers

Changing Business Environments and Computerized Decision Support

The business pressures‐responses‐support model

– The business environment
– Organizational responses: be reactive, anticipative,
adaptive, and proactive
– Computerized support

Managerial Decision Making

The nature of managers’ work

– Interpersonal
– Informational
– Decisional
Introduction to Decision Models

Decision modeling refers to the use of mathematical or scientific methods to determine an

allocation of scarce resources which improves or optimizes the performance of a system.
The terms operations research and management science are also used to refer to decision
modeling

Overview of Decision Models