UNIT-II

Data Science Process: Overview

Data Science Process

Data science process consists of six stages :

1. Discovery or Setting the research goal

2. Retrieving data

3. Data preparation

4. Data exploration

5. Data modeling

6. Presentation and automation

• Fig. 1.3.1 shows data science design process.

• Step 1: Discovery or Defining research goal

This step involves acquiring data from all the identified

internal and external sources, which helps to answer the business
question.

• Step 2: Retrieving data

It collection of data which required for project. This is the

process of gaining a business understanding of the data user have
and deciphering what each piece of data means. This could entail
determining exactly what data is required and the best methods for
obtaining it. This also entails determining what each of the data
points means in terms of the company. If we have given a data set
from a client, for example, we shall need to know what each column
and row represents.

• Step 3: Data preparation

Data can have many inconsistencies like missing values,

blank columns, an incorrect data format, which needs to be cleaned.
We need to process, explore and condition data before modeling.
The cleandata, gives the better predictions.

• Step 4: Data exploration

Data exploration is related to deeper understanding of data.

Try to understand how variables interact with each other, the
distribution of the data and whether there are outliers. To achieve
this use descriptive statistics, visual techniques and simple
modeling. This steps is also called as Exploratory Data Analysis.
• Step 5: Data modeling

In this step, the actual model building process starts. Here,

Data scientist distributes datasets for training and testing.
Techniques like association, classification and clustering are applied
to the training data set. The model, once prepared, is tested against
the "testing" dataset.

• Step 6: Presentation and automation

Deliver the final baselined model with reports, code and

technical documents in this stage. Model is deployed into a real-time
production environment after thorough testing. In this stage, the key
findings are communicated to all stakeholders. This helps to decide
if the project results are a success or a failure based on the inputs
from the model.

Defining Research Goals

• Goal of first phase will be the answer of these three questions.

• In this phase, the data science team must learn and investigate the
problem, develop context and understanding and learn about the
data sources needed and available for the project.

1. Learning the business domain:

• Understanding the domain area of the problem is essential. In

many cases, data scientists will have deep computational and
quantitative knowledge that can be broadly applied across many
disciplines.
 • Data scientists have deep knowledge of the methods,
techniques and ways for applying heuristics to a variety of business
and conceptual problems.

2. Resources:

• As part of the discovery phase, the team needs to assess the

resources available to support the project. In this context, resources
include technology, tools, systems, data and people.

3. Frame the problem:

• Framing is the process of stating the analytics problem to be

solved. At this point, it is a best practice to write down the problem
statement and share it with the key stakeholders.

• Each team member may hear slightly different things related

to the needs and the problem and have somewhat different ideas of
possible solutions.

4. Identifying key stakeholders:

• The team can identify the success criteria, key risks and
stakeholders, which should include anyone who will benefit from
the project or will be significantly impacted by the project.

• When interviewing stakeholders, learn about the domain

area and any relevant history from similar analytics projects.

5. Interviewing the analytics sponsor:

• The team should plan to collaborate with the stakeholders to

clarify and frame the analytics problem.
 • At the outset, project sponsors may have a predetermined
solution that may not necessarily realize the desired outcome.

• In these cases, the team must use its knowledge and expertise
to identify the true underlying problem and appropriate solution.

• When interviewing the main stakeholders, the team needs to

take time to thoroughly interview the project sponsor, who tends to
be the one funding the project or providing the high-level
requirements.

• This person understands the problem and usually has an idea

of a potential working solution.

6. Developing initial hypotheses:

• This step involves forming ideas that the team can test with
data. Generally, it is best to come up with a few primary hypotheses
to test and then be creative about developing several more.

• These Initial Hypotheses form the basis of the analytical tests

the team will use in later phases and serve as the foundation for the
findings in phase.

7. Identifying potential data sources:

• Consider the volume, type and time span of the data needed
to test the hypotheses. Ensure that the team can access more than
simply aggregated data. In most cases, the team will need the raw
data to avoid introducing bias for the downstream analysis.
Retrieving Data
• Retrieving required data is second phase of data science
project. Sometimes Data scientists need to go into the field and
design a data collection process. Many companies will have already
collected and stored the data and what they don't have can often be
bought from third parties.

• Most of the high quality data is freely available for public and
commercial use. Data can be stored in various format. It is in text file
format and tables in database. Data may be internal or external.

1. Start working on internal data, i.e. data stored within the

company

• First step of data scientists is to verify the internal data.

Assess the relevance and quality of the data that's readily in
company. Most companies have a program for maintaining key data,
so much of the cleaning work may already be done. This data can be
stored in official data repositories such as databases, data marts,
data warehouses and data lakes maintained by a team of IT
professionals.

• Data repository is also known as a data library or data

archive. This is a general term to refer to a data set isolated to be
mined for data reporting and analysis. The data repository is a large
database infrastructure, several databases that collect, manage and
store data sets for data analysis, sharing and reporting.

• Data repository can be used to describe several ways to

collect and store data:
 a) Data warehouse is a large data repository that aggregates
data usually from multiple sources or segments of a business,
without the data being necessarily related.

b) Data lake is a large data repository that stores unstructured

data that is classified and tagged with metadata.

c) Data marts are subsets of the data repository. These data

marts are more targeted to what the data user needs and easier to
use.

d) Metadata repositories store data about data and databases.

The metadata explains where the data source, how it was captured
and what it represents.

e) Data cubes are lists of data with three or more dimensions stored
as a table.

Advantages of data repositories:

i. Data is preserved and archived.

ii. Data isolation allows for easier and faster data reporting.

iii. Database administrators have easier time tracking problems.

iv. There is value to storing and analyzing data.

Disadvantages of data repositories :

i. Growing data sets could slow down systems.

ii. A system crash could affect all the data.

iii. Unauthorized users can access all sensitive data more easily than
if it was distributed across several locations.

2. Do not be afraid to shop around

• If required data is not available within the company, take the

help of other company, which provides such types of database. For
example, Nielsen and GFK are provides data for retail industry. Data
scientists also take help of Twitter, LinkedIn and Facebook.

• Government's organizations share their data for free with the

world. This data can be of excellent quality; it depends on the
institution that creates and manages it. The information they share
covers a broad range of topics such as the number of accidents or
amount of drug abuse in a certain region and its demographics.

3. Perform data quality checks to avoid later problem

• Allocate or spend some time for data correction and data

cleaning. Collecting suitable, error free data is success of the data
science project.

• Most of the errors encounter during the data gathering phase

are easy to spot, but being too careless will make data scientists
spend many hours solving data issues that could have been
prevented during data import.

• Data scientists must investigate the data during the import,

data preparation and exploratory phases. The difference is in the
goal and the depth of the investigation.
 • In data retrieval process, verify whether the data is right data
type and data is same as in the source document.

• With data preparation process, more elaborate checks

performed. Check any shortcut method is used. For example, check
time and data format.

• During the exploratory phase, Data scientists focus shifts to

what he/she can learn from the data. Now Data scientists assume
the data to be clean and look at the statistical properties such as
distributions, correlations and outliers.

Exploratory Data Analysis

• Exploratory Data Analysis (EDA) is a general approach to
exploring datasets by means of simple summary statistics and
graphic visualizations in order to gain a deeper understanding of
data.

• EDA is used by data scientists to analyze and investigate data

sets and summarize their main characteristics, often employing data
visualization methods. It helps determine how best to manipulate
data sources to get the answers user need, making it easier for data
scientists to discover patterns, spot anomalies, test a hypothesis or
check assumptions.

• EDA is an approach/philosophy for data analysis that

employs a variety of techniques to:

1. Maximize insight into a data set;

2. Uncover underlying structure;

 3. Extract important variables;

4. Detect outliers and anomalies;

5. Test underlying assumptions;

6. Develop parsimonious models; and

7. Determine optimal factor settings.

• With EDA, following functions are performed:

1. Describe of user data

2. Closely explore data distributions

3. Understand the relations between variables

4. Notice unusual or unexpected situations

5. Place the data into groups

6. Notice unexpected patterns within groups

7. Take note of group differences

• Box plots are an excellent tool for conveying location and

variation information in data sets, particularly for detecting and
illustrating location and variation changes between different groups
of data.

• Exploratory data analysis is majorly performed using the

following methods:
 1. Univariate analysis: Provides summary statistics for each
field in the raw data set (or) summary only on one variable. Ex :
CDF,PDF,Box plot

2. Bivariate analysis is performed to find the relationship

between each variable in the dataset and the target variable of
interest (or) using two variables and finding relationship between
them. Ex: Boxplot, Violin plot.

3. Multivariate analysis is performed to understand

interactions between different fields in the dataset (or) finding
interactions between variables more than 2.

• A box plot is a type of chart often used in explanatory data

analysis to visually show the distribution of numerical data and
skewness through displaying the data quartiles or percentile and
averages.

1. Minimum score: The lowest score, excluding outliers.

2. Lower quartile: 25% of scores fall below the lower quartile value.

3. Median: The median marks the mid-point of the data and is shown
by the line that divides the box into two parts.
4. Upper quartile: 75 % of the scores fall below the upper quartiel
value.

5. Maximum score: The highest score, excluding outliers.

6. Whiskers: The upper and lower whiskers represent scores

outside the middle 50%.

7. The interquartile range: This is the box plot showing the middle
50% of scores.

• Boxplots are also extremely useful for visually checking

group differences. Suppose we have four groups of scores and we
want to compare them by teaching method. Teaching method is our
categorical grouping variable and score is the continuous outcomes
variable that the researchers measured.
Data Transformation

Data transformation is an important step in data analysis

process that involves the conversion, cleaning, and organizing of
data into accessible formats. It ensures that the information is
accessible, consistent, secure, and finally recognized by the intended
business users. This process is undertaken by organizations to
utilize their data to generate timely business insights and support
decision-making processes.

Data Transformation

The transformations can be divided into two categories:

1. Simple Data Transformations include straightforward

procedures including data cleansing, standardization,
aggregation, and filtering. These transformations are often
carried out utilizing simple data manipulation methods and are
frequently used to prepare data for analysis or reporting.

2. Complex Data Transformations include more advanced

processes such data integration, migration, replication, and
enrichment. These transformations often need complex data
manipulation methods like as data modeling, mapping, and
 validation, and are commonly used to prepare data for
advanced analytics, machine learning, or data warehousing
applications.

Importance of Data Transformation

Data transformation is important because it improves data quality,

compatibility, and utility. The procedure is critical for companies
and organizations that depend on data to make informed decisions
because it assures the data's accuracy, reliability, and accessibility
across many systems and applications.

1. Improved Data Quality: Data transformation eliminates

mistakes, inserts in missing information, and standardizes
formats, resulting in higher-quality, more dependable, and
accurate data.

2. Enhanced Compatibility: By converting data into a suitable

format, companies may avoid possible compatibility difficulties
when integrating data from many sources or systems.

3. Simplified Data Management: Data transformation is the

process of evaluating and modifying data to maximize storage
and discoverability, making it simpler to manage and maintain.

4. Broader Application: Transformed data is more useable and

applicable in a larger variety of scenarios, allowing enterprises
to get the most out of their data.

5. Faster Queries: By standardizing data and appropriately

storing it in a warehouse, query performance and BI tools may
be enhanced, resulting in less friction during analysis.
Data Transformation Techniques and Tools

There are several ways to alter data, including:

1. Programmatic Transformation: automating the

transformation operations via the use of scripts or computer
languages such as Python, R, or SQL.

2. ETL Tools: Tools for extracting, transforming, and loading data

(ETL) are made to address complicated data transformation
requirements in large-scale settings. After transforming the
data to meet operational requirements, they extract it from
several sources and load it into a destination like a database or
data warehouse.

3. Normalization/Standardization: Scikit-learn in Python

provides functions for normalization and standardization such
as MinMaxScaler and StandardScaler.

4. Encoding Categorical variables: Pandas library in python

provides get_dummies function employed for one-hot
encoding. For label encoding LabelEncoder is provided
by Scikit-learn.

5. Imputation: Missing values in the dataset are filled using

statistical methods like fillna method in Pandas
Library. Additionally, missing data can be imputed using mean,
median, or mode using scikit-learn's SimpleImputer.

6. Feature Engineering: To improve model performance, new

features are developed by combining old ones. Pandas, a
Python library, is often used to execute feature engineering
 tasks. Functions such as apply, map, and
transform are used to generate new features.

7. Aggregation and grouping: Pandas group by function is used

to group data and execute aggregation operations such as sum,
mean, and count.

8. Text preprocessing: Textual data is preprocessed by

tokenizing, stemming, and eliminating stop words using NLTK
and SpaCy Python libraries.

9. Dimensional Reduction: The technique involves reducing the

amount of characteristics while retaining vital
information. Scikit-learn in Python provides techniques such
as PCA for Principal Component Analysis and TruncatedSVD
for Dimensionality Reduction.

Advantages of Data Transformation

1. Enhanced Data Quality: Data transformation aids in the

organization and cleaning of data, improving its quality.

2. Compatibility: It guarantees data consistency between many

platforms and systems, which is necessary for integrated
business environments.

3. Improved Analysis: Analytical results that are more accurate

and perceptive are frequently the outcome of transformed
data.
 4. Increases Data Security: Data transformation can be used to
mask sensitive data, or to remove sensitive information from
the data, which can help to increase data security.

5. Enhances Data Mining Algorithm Performance: Data

transformation can improve the performance of data mining
algorithms by reducing the dimensionality of the data and
scaling the data to a common range of values.

Disadvantages of Data Transformation

1. Time-consuming: Data transformation can be a time-

consuming process, especially when dealing with large datasets.

2. Complexity: Data transformation can be a complex process,

requiring specialized skills and knowledge to implement and
interpret the results.

3. Data Loss: Data transformation can result in data loss, such as

when discretizing continuous data or when removing attributes
or features from the data.

Build the Models

• To build the model, data should be clean and understand the
content properly. The components of model building are as follows:

a) Selection of model and variable

b) Execution of model

c) Model diagnostic and model comparison

• Building a model is an iterative process. Most models consist

of the following main steps:
 1. Selection of a modeling technique and variables to enter in
the model

2. Execution of the model

3. Diagnosis and model comparison

Model and Variable Selection

For this phase, consider model performance and whether

project meets all the requirements to use model, as well as other
factors:

1. Must the model be moved to a production environment and,

if so, would it be easy to implement?

2. How difficult is the maintenance on the model: how long

will it remain relevant if left untouched?

3. Does the model need to be easy to explain?

Model Execution

• Various programming language is used for implementing the

model. For model execution, Python provides libraries like
StatsModels or Scikit-learn. These packages use several of the most
popular techniques.

• Coding a model is a nontrivial task in most cases, so having

these libraries available can speed up the process. Following are
the remarks on output:

a) Model fit: R-squared or adjusted R-squared is used.

b) Predictor variables have a coefficient: For a linear model

this is easy to interpret.
 c) Predictor significance: Coefficients are great, but
sometimes not enough evidence exists to show that the influence is
there.

• Linear regression works if we want to predict a value, but

for classify something, classification models are used. The k-
nearest neighbor’s method is one of the best methods.

• Following commercial tools are used:

1. SAS enterprise miner: This tool allows users to run

predictive and descriptive models based on large volumes of data
from across the enterprise.

2. SPSS modeler: It offers methods to explore and analyze

data through a GUI.

3. Matlab: Provides a high-level language for performing a

variety of data analytics, algorithms and data exploration.

4. Alpine miner: This tool provides a GUI front end for users
to develop analytic workflows and interact with Big Data tools and
platforms on the back end.

• Open Source tools:

1. R and PL/R: PL/R is a procedural language for PostgreSQL

with R.

2. Octave: A free software programming language for

computational modeling has some of the functionality of Matlab.

3. WEKA: It is a free data mining software package with an

analytic workbench. The functions created in WEKA can be
executed within Java code.
 4. Python is a programming language that provides toolkits
for machine learning and analysis.

5. SQL in-database implementations, such as MADlib provide

an alternative to in memory desktop analytical tools.