Introduction to Data
analytics and life cycle
What is Data Analysis?
• Data analysis examines, processes, and interprets raw data to
discover useful insights and conclude and support decision-making.
• It includes arranging and translating data into an understandable
format, displaying data using graphs and charts, and applying
statistical techniques to spot patterns and linkages.
Key Roles for a Successful Analytics Project
1. Business User: Someone who understands the domain area and usually benefits from the
results. Usually a business analyst, line manager, or deep subject matter expert in the project
domain fulfills this role.
2. Project Sponsor: Responsible for the genesis of the project. Provides requirements for the project
and defines the core business problem
3. Business Intelligence Analyst: Provides business domain expertise based on a deep
understanding of the data, Business Intelligence Analysts generally create dashboards and reports
and have knowledge of the data feeds and sources
4. Database Administrator (DBA): Provisions and configures the database environment to support
the analytics needs of the working team
5. Data Engineer: Leverages deep technical skills to assist with tuning SQL queries for data
management
6. Data Scientist: Provides subject matter expertise for analytical techniques, data modeling, and
applying valid analytical techniques to given business problems.
Life Cycle Phases of Data Analytics
Phase 1: Discovery
1. Learning the Business Domain
• Understanding the domain area of the problem is essential.
2. Resources
• Team needs to assess the resources available to support the project.
• In this context, resources include technology, tools, systems, data,
and people(stakeholders)
3. Framing the Problem
4. Identifying the key stakeholders
5. Interviewing the Analytics sponsor
6. Developing the initial hypothesis
7. Identifying Potential Data sources
Phase 2: Data Preparation
1. Preparing the Analytic Sandbox
• Create the analytic sandbox (also called workspace)
• Allows team to explore data without interfering with live production
data
• Sandbox collects all kinds of data (expansive approach)
• The sandbox allows organizations to undertake ambitious projects
beyond traditional data analysis and BI to perform advanced
predictive analytics
• Although the concept of an analytics sandbox is relatively new, this
concept has become acceptable to data science teams and IT groups
2. Performing ETLT/ETL
• Step 1) Extraction
• Data is extracted from the source system and placed in the staging area
during extraction.
• Before moving extracted data into the Data warehouse, it can be validated in
the staging area.
• Step 2) Transformation
• The data retrieved from the source server is raw and unusable in its original
state. As a result, it must be cleaned, mapped, and transformed.
• Prepare the error free data by eliminating duplicity and integrity
• Step 3) load
• The final stage in the ETL process is to load data into the target data
warehouse database.
• A large volume of data is loaded in a relatively short period of time in a typical
data warehouse. As a result, the load process should be optimized for
performance.
3. Learning About the Data
– Determines the data available to the team early in the project
– Highlights gaps – identifies data not currently available
– Identifies data outside the organization that might be useful
Table 2.1 Sample Dataset Inventory
Data Conditioning
Survey and Visualize
Survey and Visualize
• Leverage data visualization tools to gain an overview of the data
• Review data to ensure calculations are consistent
• Does the data distribution stay consistent?
• Assess the granularity of the data, the range of values, and the level
of aggregation of the data
• Does the data represent the population of interest?
• Check time-related variables – daily, weekly, monthly? Is this good
enough?
• Is the data standardized/normalized? Scales consistent?
• For geospatial datasets, are state/country abbreviations consistent
Common Tools for Data Preparation
Phase 3:Model Planning
• The main goal is to choose an analytical technique, or several candidates, based
on the end goal of the project
• We observe events in the real world and attempt to construct models that use
this behavior with a set of rules and conditions
• Determine whether to use techniques best suited for structured data,
unstructured data, or a hybrid approach
• Teams often create initial models using statistical software packages such as R,
SAS, or Matlab
– Which may have limitations when applied to very large datasets
• The team moves to the model building phase once it has a good idea about the
type of model to try
Phase 3: Model Planning
Table : Research on Model Planning in Industry Verticals
Data Exploration and Variable Selection
Model Selection
Phase 4:Model building
Phase 5:Communication Result
Phase 5:Communication results
Phase 5:Communication results
Phase 5:Communication results
Phase 6:Operationalize
Phase 6:Operationalize
Phase 6:Operationalize
Phase 6:Operationalize