Chapter 2

Big Data Analytics Practice

1. Big Data Analytics: Definition

Big Data Analytics (BDA) is the process of examining large and complex datasets to uncover
hidden patterns, unknown correlations, market trends, customer preferences, and other
useful business information. The ultimate goal is to enable better decision-making.

2. The Core Component of Big Data Analytics- Data

Analytics quality depends not only on the tools and techniques employed, but basically on the
quality of the data processed. So, The basic attributes of data that should be considered
before analysis are as following:

Data Formats

Data formats may refer to data types, recording and content formats, file formats
and more. We will discuss a few of these here. Specifically, we would like to
provide a broad brush on data formats that form the basis for certain type of
analytics. The following list illustrates some data formats that often drive the
type of analytics performed:
1) Words
2) Numeric
3) Images / Pictures
4) Video
5) Audio / Sounds

Data Format Example of Analytics Applied

Words Text Analytics, Natural Language, Search Analytics, Web Analytics

Numeric Healthcare Analytics, Business Analytics, Risk Analytics, Financial

Analytics
Images / Image Analytics, Sports Analytics, Visual Analytics, Geospatial
Pictures Analytics
Audio / Sound Voice Recognition/Transcription, Security Analytics, Call Center
Analytics
Video Video Analytics, IoT and Smart Cities, Retail Analytics, Streaming
Analytics

There has been a major shift in the types of data used for analysis. Forty years ago,
almost all data analyzed was numerical or categorical (quantitative and qualitative)
in nature. Healthcare and businesses were full of numerical data that was being
analyzed by the adoption of computing technology. Ten to twenty years later, as
computers got much faster, there was a greater adoption of “number crunching”
1
 and data mining, and there was an explosion of text analytics. In the last year,
there has been a huge increase in image recognition, audio, video, and streaming
analytics.
File formats refer to structures that a computer can read. It allows programs to
identify and load different data formats from the list above as well as mixtures of
these data formats. Some computer file extension examples are *.csv, *.txt. *.xml,
*.mdf, *.html, *.log, and many, many more.

Data Stores

Processes generate data and we record that data. It exists in non-permanent

memory sometimes called temporary memory or volatile memory, which used
immediately and then deleted. For example, think of files on your computer that
are created, but not saved. That data exists in temporary memory (RAM) that is
only referenceable while the computer is on. If you want to access that data later,
you must store it.
Most analytics data is stored at least temporarily. All analytics development can be
thought of as using data storage. We just want to provide a quick list of data
storage locations now, as this can dictate the type of method that is used. Here
are a few examples most applicable to analytics:

1) Local files. Spreadsheets, databases, text, xml, html

2) Relational databases on premise
3) Semi-structured (non-lake) databases on premise (JSON, XML, logs, NoSQL)
4) Data lake, big data stores (Apache and Hadoop)
5) Cloud Storage options (Software as a Service, Database as a Service,
Infrastructure and Platform as a service)
6) Streaming data – This is more of a connector than storage; noted here since
streaming analytics is growing rapidly

Provisioning Data for Analytics

Some of the basic data provisioning activities for projects. You will hear similar
names for some of these activities, like data wrangling, data munging, etc. Let us
address a few activities that are common in many analytics projects.

1) Data Sourcing
a. Identification of Project Data (led by business)
b. Data Engineering (technical aspects)
2) Data Quality Assessment and Remediation
3) Data Integration
4) Data Cleaning of Merged Data

Once the project begins there will be additional activities:

5) Exploratory Data Analysis
6) Data Transformation

2
 7) Data Reduction
8) Rinse and Repeat Where Necessary

Data Sourcing

There are two parts to data sourcing.

The first part is determining the data you will need. What is the right data needed
to support an analytics project? If not, that is okay; we will revisit the initial stages
of the analytics process in the section on business understanding and project
definition. What is an important result that the business would benefit from
achieving? Is it attainable in a realistic time horizon? Who would need to be
involved in the process, which members from what groups? IT, and which business
leaders? Which technical people? What would the metrics be to support the
answer? How would you know you arrived at the destination? As Dr. Tom Hill, “How
would we know we won? How would we know we are done?”
If it helps, begin with the end in mind and work backward. What would be the
decision or the action needed to bring about the final result? Then, at the next
step backward: What analysis would deliver the information to determine that
decision or action? Then one more step backward: What data would be required
for that analysis?
Once you know what data elements are needed to support the project, you move
to the second part, which is data sourcing, connecting and returning the data to an
analytics sandbox. This sandbox could take several forms; it is normally just a
database that is separated from transactional systems. First, you must ask if you
can access the data you need? If so, who can source it? If not, is there surrogate
data that will provide a close approximation or good enough information? If not,
you might need to set this project aside for a time while you can work on getting
those pipes installed and in the meantime select another target project.

Data Quality Assessment and Remediation

Once you have data in an analytics sandbox, you need to determine the quality of
the data. Let’s examine five common issues in data quality which will form your
assessment:

a) Record Duplication
b) Sparsity
c) Outliers
d) Invariance
e) Redundancy

Duplication exists in transactional systems and when data is merged across

systems. At this point you will have not merged data, but you should do a first
pass at the sourcing and bring in the data (called staging and provisioning by data
scientists). As the old quality paradigm goes, the more you can move quality
toward the beginning of a system, the more time and money you can save.
3
 Duplicates can be exact copies appearing in multiple rows. This is very easy to fix by
the IT department. Or, it could be caused by “similar” records. An example for a
similar record could be a patient master index that has Sarah S. Jones in one record
and Sarah Sue Jones in another record, and they are actually records for the same
person. Suppose that Sarah went to an out-of-network provider and filled the form
out with Sara Sue Jones. When this record was brought in, the patient master was
not smart enough to identify it as the same Sara S. Jones so it created a new record.
Interestingly, there are systems that use machine learning to help alleviate this issue.
Sparsity is missing data. You can have row or column sparsity; this means you
are missing several elements for a record across a row. You might have several
missing values for a variable (column of data). If you imagine your data in a
spreadsheet below with 10 variables (10 columns); then the Patient ID forms a
row for the other 9 variables. In the table, patient 3333 is missing data for two
columns, for example. You must decide how to handle missing values for your
modeling projects. The easiest way to handle it is to throw out rows or columns,
but this may weaken the signal or amount of information you need for project
success. Alternatively, there are many ways to handle missing values: replacing
with averages, average frequency categories, use of an imputation model to fill
data and many more methods are options to handle these issues (good predictive
analytic software packages can do this “automatically,” with the user having the
ability to specify which type of method to handle missing data is desired).

Table 2.1 Example of Row and Column Sparsity with Patient Data

ID Readmi Age Surgeon EBL Apgar_Score HMBin60days ASA Gender WND_Class

ssion ID
1111 Yes 24 4 3300 5-6 2 female 00
2222 Yes 22 1 3300 7-8 9 male 02
3333 Yes 29 2700 2.8 2 male 03 Bad
Row
4444 Yes 21 2 2400 9-10 2.2 3 male 03
5555 Yes 41 2 2000 5-6 2 male 02
Bad Column

Outliers are another problem where a few values can throw off statistics. You can
test for these in a variety of ways and intelligently replace them with values more
tenable for your analysis needs. All of these have drawbacks: missing data has its
costs, but it does not have to derail your project unless your data is really lacking.
Invariance is when you have a categorical variable (nominal or interval) that has
either all or almost all the same values. Imagine you have a column for Gender and
99% of your data is female. This would cause problems if you used this column for
modeling. It is best to see if you can increase the male instances, throw out the
entire variable, or apply some sort of oversampling. Or, if the variable is numeric,
and it has very little wiggle (a not-so-technical term for variance), it could also be
discarded, as there is not enough variance to be of much or any value in statistics
4
and modeling. Suppose the class average is 99 with a low score of 98 and a high
score of 100 – this variable will be of little use in any analysis.

Last on the list is redundancy. There are analytic methods that do not work well if
you have highly correlated predictors. Example, suppose I want to predict annual
income with four predictor variables. These predictors are age in years, highest
degree completed, number of years of education and whether the occupation was
science/technical vs non-science/non-technical. The highest degree completed and
the number of years of education are likely to be highly correlated and contain the
same information in trying to predict annual income. Therefore they are redundant
and we would likely want to get rid of one of them or at least note it and not use
both of them in a model that might be susceptible to problems with redundancy.

Other issues can occur in the data for your project. For example, your data can be
stale, out of date. You cannot account for every possible nuance in advance, as
data discovery is a process. However, with a good team working together, you can
discover them and correct for them, thus adding value to your project.

Integrate and Repeat

Data integration is joining various data sets together. It is often needed to get
customer demographic data from one system, transactional sales data from
another system and more… Most analytics are performed on a single, monolithic
set of data – wide and deep. You will typically need to join this data together. A
data engineer, an IT data base guru or a savvy business analyst can help with this
activity. You will stage these tables in the analytic sandbox for additional
processing. Different data tables, each with different data from the same subject,
can be “joined” into one table; these joins are done such that one row will
represent a customer, a patient or an event.
Once you have this complete data set formulated, it is necessary to repeat at
minimum a couple of the data quality steps. Once you have the process formulated
it should not be difficult to repeat all steps, but it is critical to look for row
duplicates, since joining several data tables or data files together increases the
likelihood that you will have more of them.

Exploratory Data Analysis (EDA)

Once you have the data sourced and fixed many of the data quality issues (you
should know that this is a continuing process; like parenthood, you are never done),
you can start the fun stuff.
Exploratory data analysis is very akin to descriptive analytics. We compute
summary statistics, we continue to look for outliers, but now we look at them in a
different lens; we start to see data in a multidimensional way, where rather than
one data point outside its neighbors, we look if it has an unwieldy effect on other

5
variables. This can be very interesting and lead to immediate wins of the
analytics project. This should be underscored – you want to celebrate
wins from the EDA phase. This is the time where the analytics team can go
back to the business and thus demonstrate the value of analytics to the entire
enterprise – the entire membership of the company or organization. You want a
place where you record all these successes. At times, the end deliverables may not
be as stellar as hoped, but what might seem small insights can have a huge payoff.
Deloitte reported in 2019 a study that increased data capture and analysis can
likely save billions of dollars for the energy industry.
As with most of the analytics process, this is never complete. The initial pass may
be the most time consuming and provide the greatest insights, but it usually needs
to be revisited.

Here are some goals from EDA:

1) What are the relationships among the predictors and what is being predicted
(target)?
2) Are there any data points that highly influence/leverage a target outcome? If
so, investigate, and share with the business if applicable. If this leads to
business insight or action, record this as a win for the team and try to measure
the impact in a quantitative measure – money savings, money growth.
3) We need to look at interactions .
4) Are we correctly accounting for the relationships? Do we need additional
data to make a more accurate prediction?
5) Are we asking the right questions? So important, we will repeat.

Data Transformations

Data transformations are formulas applied to variables or combinations of variables. A

few examples are:
1) Indicator creation or dummy coding
2) Logarithmic or mathematical transformations
3) Normalization
4) Statistical standardization
5) Binning of continuous variable ranges (thus converted into categorical
variables)
6) Aggregation
7) Generalization
8) Masking / de-identification for privacy

Data Reduction

Data reduction refers to eliminating variables that cannot or should not be used in
the analysis. It may be discovered in the data preparation phase that certain
variables should be eliminated for:

1) Ethical reasons
6
 2) Legal reasons
3) Public relations
4) Insufficient utility for the problem being addressed
5) Future censoring – data is available historically, but not in the future
6) Expense – the data exists, but is cost prohibitive going forward

A second type of data reduction is done in the modeling or analysis phase to

pare down the data set so that models can execute more timely and the analysis of
the results is more efficient. Statistical and machine learning algorithms usually aid
these reductions.

3. Classification of Big Data Analytics

BDA can be classified into four primary types based on the questions they answer and their
complexity:

Question
Classification Description Example
Answered
Summarizes past data to understand Calculating average
Descriptive What
what occurred. This is the simplest sales figures last
Analytics happened?
form. quarter.
Analyzing server logs
Uses data to investigate causes and
Diagnostic Why did it to pinpoint the root
relationships, helping to determine
Analytics happen? cause of an
why a particular outcome occurred.
application crash.
Uses statistical models and machine Predicting future
Predictive What will
learning to forecast future outcomes stock prices or
Analytics happen?
or probabilities. customer churn rates.
Recommending the
Recommends a course of action to optimal inventory
Prescriptive What should
achieve an optimal outcome. This is level to maximize
Analytics I do about it?
the most complex form. profit based on
predicted demand.

Below we describe an analytics classifcation scheme that is generally accepted. This scheme is
ubiquitous in the analytics domain and is often attributed to Gartner. It consists of four major
categories of analytics – Descriptive Analytics, Diagnostic Analytics, Predictive Analytics, and
Prescriptive Analytics.

Gartner’s Classifcation
Gartner, Inc. was founded in 1979. Its purpose is to equip executives across the entire
enterprise to make the right decisions at the right time, and thus stay ahead of change in the
world (Gartner, Inc., 2019). It is a global company serving over 15,600 client organizations in
over 100 countries around the world.
We will explore the four major groups of analytics that Gartner uses (Gartner 2019). The
following chart (Figure 2.1) demonstrates the relationship of these four categories plus the
concept of models that optimize a process. It contains the evolution of these across three
dimensions. First, as we move from descriptive analytics to prescriptive analytics to the higher
7
levels, we require less human cognition and more machine processing. In other words,
computer processing and algorithms replace the thinking required of humans and this
requires more machine processing. As we move from description and diagnosis to prediction,
we are moving from looking in the rear view mirror (hindsight) to looking through the
windshield (foresight). And, finally, as a general rule as we move from the lower left to the
upper right the value of these projects increases.

Figure 2.1: Big Data analytics classifcation scheme

Descriptive Analytics
Descriptive analytics has a very long historical foundation. These analytics are the easiest to
implement, but many times do not offer the value of other methods, which are generally
more complex and sophisticated. Descriptive analytics typically fall into one of the following
categories:
1) Descriptive statistics and numerical summaries
2) Visual representation
According to Figure 2.1, descriptive analytics describes analytics as answering the question,
“What Happened?” However, descriptive analytics go well beyond this question. They form
the foundation for all the other analytics categories, meaning that diagnostic analytics,
predictive analytics, etc. all start with descriptive analytics.
A simple example of a descriptive statistic is the average weight of a male living in the United
States. It provides a quick summary for millions of individuals in one number. Its advantage is
that it is a single number, very easy to digest. Its disadvantage is the same. It does not provide
any idea of spread or nuances of the population. Spread and other nuances can be done with
several complimentary statistics (percentiles, variance, max, min, etc.), but this also requires
mental gymnastics to be able to understand them.
A visual representation is often better than a set of summary statistics. Humans are very good
at processing visuals. This skill was developed over millions of years of evolution. It is true that
a picture is worth a thousand words. BI is about description – tables, reports, and Visual BI is
8
graphs, dashboards, interactive displays of information. In classical statistics, descriptive
statistics is one of two major divisions, the other being inferential statistics.

Diagnostic Analytics
Diagnostic analytics is a form of advanced analytics that examines data or content to answer
the question, “Why did it happen?” It is causal and retrospective. It has causal meaning in that
a change in an attribute (a cause) will generally or specifcally result in an observable change in
a second attribute (the effect). Moreover, the cause must precede the effect. It is important
to note the difference in association or correlation with causation (see the box below on
“Causation vs Correlation”)

CAUSATION VS CORRELATION
While the confusion over causation and correlation has improved over the last few
years, you can still see many examples by journalists and other authors that mix the
meaning of the two concepts. When two variables tend to run together, we say they
are associated, they co-occur, they are correlated. When one category of a variable is
present, a second variable category is often present. Or, in the case of numerical
attributes, when one variable increases the second variable increases (positive
relationship) or decreases (negative relationship).
It is very important to understand that correlations can be spurious, meaning they can
be caused by factors outside the two variables of interest. On the other hand, they can
be sheer coincidence. There are billions of variables so at times you can think
something is correlated when in fact, it is just chance. Causation on the other hand
implies that there is some sort of physical, psychological or underlying link.
If you want some fun, we recommend that you check out Tyler Vigen’s website at
[Link] You will find a lot of interesting spurious
correlations! We recreated a sample of this in the spirit of Tyler’s website. Figure 2.2
illustrates the relationship of U.S. Spending on Science, Space and Technology vs
Suicides by Hanging, Strangulation, and Suffocation.
You can see that these two series are highly correlated, and if you found it in a
newspaper you might read that U.S. Spending is causing suicides! Yes, this happens
all the time when journalists and others confuse correlation vs causation.

Predictive Analytics

Predictive analytics answers the question, “What will happen?” Predictive

analytics is about assigning a probability or likelihood to an outcome based on
some type of analysis or model. The types of analysis or models that result in
predictive models will be presented in various chapters of this book. Many
predictive models today are generated with machine learning algorithms. These
models when given a set of inputs will generate the probability of an outcome.
For example, the University of Iowa Hospitals and Clinics used patient and
operating room data to generate predictive models to determine the likelihood of
surgical site infections developing from operations performed in the operating
room. The most interesting and innovative thing about this model was that it was
executed while the patient was still under anesthesia in the operating room. The
resulting probability in “Real Time” gave the care team information they could use
on what preventative treatment should be used – normal closure of the wound,

9
 delayed closure or negative pressure therapy (Siwicki, 2018). Relationships
between targets and predictors in predictive models may be correlated or causal in
nature.

Figure 2.2 Spurious Correlations, U.S. Spending, and Suicides.

Prescriptive Analytics

Gartner notes prescriptive analytics is a form of advanced analytics, which examines

data or content to answer the question “What should be done?” or “What can we do
to make XYZ happen?” This type of analytics requires good data and maybe additional
assumptions.
The difference between predictive analytics and prescriptive analytics is often
misunderstood. Prescriptive analytics requires causality. It is important to
understand the differences and we created a special gray box to cover these
(please see “Predictive or Prescriptive??” gray box).

Process Optimization

Optimization is a word that is overused a lot these days. I optimized my workout.

We use the word optimization in a very strict, rigorous way. In general we start
with an objective function that we explicitly state. We state whether we want to
maximize or minimize this function – this is optimization. We add constraints that
cannot be violated. We then search for solutions that maximize or minimize our

10
 function and do not violate our constraints. The solution has the highest or lowest
value of the objective function is the optimal solution.

PREDICTIVE OR PRESCRIPTIVE??
One of the most misunderstood differences is the classification of predictive vs
prescriptive analytics. We have seen experienced data scientists mix this up. There
is a big difference in these models. We will not get technical, but if you learn this –
you will be ahead of many professionals in the analytics space!
First, some similarities in the method classifcations:

1) Both predictive analytics and prescriptive analytics apply to unseen events

(most are future or forward looking)
2) They both contain a probability of an outcome (probabilistic models)
3) Both often provide highly valuable insights

Then, some dissimilarities in the method classifcations:

1) Prescriptive models are causal; predictive models may or may not be
2) Predictive models are generally easier to formulate
3) Prescriptive models require special assumptions or special data to formulate
Prescriptive models are the gold standard since they are causal, and thus
manipulating factors that cause the outcome results in a higher probability of
success. For example, in healthcare, it is known that prescribing antibiotics will
generally reduce bacterial infections.
What is an example of a predictive model that is not prescriptive? Imagine an
Internet retailer that wants to sell products or services off their website, and they
have a loyalty program called “elite rewards.” It turns out, not surprisingly, that a
customer in their elite rewards (an annual fee required) membership is a very
important variable in their predictive model to determine if they will be more likely
to respond to an email offer.
Now, a naïve marketer might think – “We need to push these customers into our
loyalty program so we can sell to them more frequently – they will respond more
receptively to our email ads.” This is the idea behind prescriptive analytics, i.e. what
should I do, what actions(s) should I take? They might incentivize existing customers
that are not in their elite program with a free account. Could they now expect the
same results,
e.g. the same likelihood of purchase as their predictive model would have
suggested? No. Why? Because they have not changed the customer’s behavior or
mindset. Customers that are willing to pay for the “elite” program are inherently
different from customers that are not. However, offering an easy path to elite
membership does not change the likelihood to respond to an offer. This is an
example of a confused marketer using a predictive model as if it were prescriptive.
Again, understanding the difference will make you well ahead of the pack in
understanding analytics.

11
4. Industry Examples of Big Data Analytics

Big Data analytics involves analyzing large, complex datasets to uncover patterns, trends, and
insights. It transforms decision-making across industries by enabling personalization,
predictive capabilities, and operational optimization, though it also presents significant
challenges related to data management, security, and talent.

Applications by Industry

Healthcare:

Improves Patient Care: Personalizes treatments, enables early disease

detection, and predicts outbreaks.

Accelerates Research: Analyzes medical records and genomic data to speed up

drug discovery and cancer research.

Enhances Public Health: Tracks population health trends and manages

resources.

Finance & Banking:

Manages Risk & Fraud: Monitors transactions in real-time to detect fraud and
assess credit risk.

Informs Trading: Powers algorithmic trading for faster, more precise

investment decisions.

Personalizes Services: Uses customer analytics for segmentation and targeted

marketing.

Retail:

Understands Customers: Analyzes behavior and preferences for personalized

marketing and segmentation.

Optimizes Operations: Manages inventory and supply chains based on market

demand predictions.

Manufacturing & Supply Chain:

Optimizes Production: Analyzes machine data to improve output and quality.

Predicts Maintenance: Prevents equipment breakdowns by predicting failures.

Manages Supply Chain: Aligns production and inventory with market demand
to reduce waste.

Transportation & Logistics:

12
 Optimizes Routes: Uses traffic and weather data to find the most efficient
delivery paths.

Tracks in Real-Time: Provides live visibility into shipment locations and

conditions (e.g., temperature for perishables).

Manages Warehouses: Monitors stock levels in real-time to prevent shortages.

Energy & Utilities:

Manages Smart Grids: Balances electricity supply and demand in real-time.

Performs Predictive Maintenance: Identifies potential equipment failures

before they occur.

Integrates Renewables: Forecasts energy generation from solar and wind to

optimize grid use.

Telecommunications:

Optimizes Networks: Analyzes performance data to prevent outages and plan

capacity.

Reduces Churn: Identifies customers likely to leave and creates targeted

retention campaigns.

Prevents Fraud: Detects suspicious activities like SIM card cloning.

Government & Public Sector:

Enables Smart Cities: Optimizes traffic, energy use, and public services.

Enhances Public Safety: Predicts crime hotspots and improves emergency

response.

Informs Policy: Uses data for evidence-based decision-making and resource

allocation.

Marketing & Advertising:

Personalizes Campaigns: Delivers targeted ads and recommendations based on

customer data.

Measures Effectiveness: Tracks campaign performance across channels for

better ROI.

Analyzes Sentiment: Monitors social media to gauge brand perception in real-

time.

Sports Analytics:

13
 Analyzes Performance: Uses player biometric and statistic data to optimize
training and strategy.

Prevents Injuries: Monitors athlete workload and health to identify injury risks.

Enhances Fan Engagement: Personalizes experiences and offers based on fan

behavior.

Common Challenges Across Industries

Data Management: Handling the enormous volume, high velocity, and wide variety of
data requires robust and scalable infrastructure.

Data Quality & Integration: Ensuring data is accurate, consistent, and can be
integrated from disparate, siloed sources is difficult but critical.

Security & Privacy: Protecting sensitive information from breaches and ensuring
compliance with data protection regulations (like General Data Protection Regulation-
GDPR) is a major concern.

Talent Shortage: There is a high demand for but short supply of skilled data scientists,
analysts, and engineers.

Cost & ROI: Implementing Big Data technologies involves significant investment,
making the demonstration of a clear return on investment essential.

Organizational Adoption: Fostering a data-driven culture and integrating insights into

daily decision-making can be a significant hurdle.

5. The Big Data Analytics Life Cycle

The process of building a Big Data analytics system is iterative and complex. For a software
engineer, it's a lifecycle akin to a specialized SDLC, focusing on data flow and processing.

Stage 1: Business Problem Definition & Feasibility

Goal: Translate a business need into a concrete, measurable data problem.

Key Questions: What decision will this analytics support? What is the Key Performance
Indicator (KPI)? Is the necessary data available and accessible?

Engineering Output: A clear problem statement and a preliminary feasibility

assessment of data sources and required processing power.

Stage 2: Data Acquisition & Ingestion

Goal: Extract data from diverse sources and load it into a system where it can be
processed.

Activities:

14
 Batch Ingestion: Scheduled, bulk movement of data (e.g., nightly ETL jobs from
transactional databases using Sqoop, or file transfers to a data lake).

Stream Ingestion: Continuous, real-time ingestion of data events (e.g., using

Apache Kafka, AWS Kinesis, or Google Pub/Sub).

Engineering Considerations: Decoupling sources from processing systems using

message queues, handling schema evolution, and ensuring data provenance.

Stage 3: Data Preprocessing & Storage

Goal: Transform raw data into a clean, usable format and store it efficiently.

Activities:

Data Wrangling: Cleaning (handling missing values, outliers), normalization,

and transformation.

Storage: Choosing the right storage solution based on access patterns.

Data Lake (e.g., S3, ADLS, HDFS): Store raw, unstructured data at scale.
Cheap storage, schema-on-read.

Data Warehouse (e.g., Snowflake, BigQuery, Redshift): Store processed,

structured data for SQL-based analytics. Optimized for fast queries.

NoSQL Databases (e.g., Cassandra, HBase, MongoDB): For specific low-

latency access patterns (key-value, document, wide-column).

Engineering Considerations: Using distributed processing frameworks like Apache

Spark for transformation, choosing file formats (Parquet, ORC) for columnar storage
and compression, and implementing data partitioning.

Stage 4: Data Analysis & Modeling

Goal: Apply statistical, machine learning, or deep learning models to discover patterns
and build predictive capabilities.

Activities:

Exploratory Data Analysis (EDA): Using tools like Jupyter notebooks to understand
data distributions and relationships.

Feature Engineering: Creating derived input variables that improve model

performance.

Model Training & Validation: Training ML models (e.g., regression, classification,

clustering) on large datasets using frameworks like Spark MLlib, TensorFlow, or
PyTorch.

15
 Engineering Considerations: Managing the lifecycle of ML models (MLOps), versioning
data and models, and ensuring reproducible experiments.

Stage 5: Data Visualization & Interpretation

Goal: Present the results of the analysis in an interpretable and actionable format for
stakeholders.

Activities: Building dashboards (with Tableau, Power BI, Grafana), generating reports,
and creating interactive visualizations.

Engineering Considerations: Designing low-latency APIs to serve data to visualization

tools and ensuring the security and governance of data access.

Stage 6: Operationalization & Maintenance

Goal: Deploy the analytics pipeline into a production environment and ensure its
continued reliability.

Activities:

Pipeline Orchestration: Using tools like Apache Airflow, Prefect, or Dagster to

schedule, monitor, and manage complex data workflows.

Monitoring & Alerting: Tracking pipeline health, data quality, and model
performance drift.

Cost & Performance Optimization: Continuously tuning the system for efficiency
and cost-effectiveness in a cloud environment.

Engineering Considerations: This is where software engineering principles—CI/CD,

monitoring, logging, fault tolerance—are paramount for building robust, production-
grade data systems.

16