What is Data Engineering?

Data engineering is a set of operations and practices focused on creating and managing the
infrastructure and systems that allow for the flow, storage, and access of data. A data
engineer's main job is to ensure that data is available, usable, and reliable for others—such as
data analysts and data scientists—to work with.

The text highlights a few key aspects of data engineering:

• It's a foundational discipline: Data engineering has existed for a long time, evolving
as companies began using data for reports, analysis, and predictions. It became a
distinct field with the rise of data science.
• It involves building data infrastructure: Data engineers are responsible for the
entire "plumbing" of an organization's data. They set up and operate the systems (like
databases and data warehouses) that store and process data.
• It's about movement, manipulation, and management: As Lewis Gavin puts it, a
core part of data engineering is handling the entire lifecycle of data—from where it
comes from, to how it's transformed, and where it ends up.
• It has different flavors: The text mentions two main types of data engineering:
o SQL-focused: This involves working with traditional relational databases and
using SQL or ETL tools to process and move data.
o Big Data-focused: This deals with massive amounts of data using
technologies like Hadoop, Spark, and Flink. It often involves programming
languages like Java, Scala, and Python.
• It's a blend of skills: Maxime Beauchemin suggests that data engineering is a
superset of business intelligence and data warehousing, but it also heavily
incorporates principles from software engineering.

Example: A Retail Company's Data Engineering Challenge

Imagine a large online retail company that sells thousands of products. They want to use data
to understand customer behavior and make better business decisions, like:

• Which products are most popular?

• What marketing campaigns are driving the most sales?
• How can they personalize product recommendations for each customer?

This is where data engineering comes in. The data is scattered across many different systems:

• Website: User clicks, product views, and cart additions are stored in a web log file.
• Sales database: Customer purchase history, transaction amounts, and payment
methods are in a relational database.
• Marketing platform: Campaign performance data (e.g., ad clicks, email opens) is in
a third-party system.
• Inventory system: Stock levels and supplier information are in another internal
database.

A data engineer would be responsible for building the system to handle all this data. Here's a
step-by-step example of what they would do:

Cisco Confiden+al
 1. Extract (E): The data engineer builds pipelines to extract data from all these different
sources. For example, they might write a script to pull daily sales data from the sales
database, and another to ingest log files from the website.
2. Transform (T): The raw data is messy and inconsistent. The data engineer must
transform it to make it usable.
o They might clean up inconsistent product names.
o They could aggregate individual clicks into a session to see a customer's full
Browse journey.
o They would combine the sales data with the marketing data to link a purchase
to a specific campaign.
3. Load (L): Once the data is cleaned and transformed, the data engineer loads it into a
centralized and structured repository, such as a data warehouse. This repository is
designed for fast analysis and reporting.
4. Manage and Maintain: The work doesn't stop there. The data engineer also:
o Monitors the pipelines to ensure they run correctly every day without errors.
o Optimizes the data warehouse so that queries from data analysts run quickly.
o Builds APIs and interfaces so that data scientists can easily access the clean
data to build machine learning models (e.g., for personalized
recommendations).
o Ensures data quality by setting up alerts for missing or corrupted data.

============

Data Engineering Defined

Data engineering is the discipline of developing, implementing, and maintaining systems

and processes that transform raw data into high-quality, consistent information. This
information is then used to support various business needs, such as analysis and machine
learning.

The text emphasizes that data engineering is a multidisciplinary field, sitting at the
intersection of:

• Security: Protecting the integrity and privacy of the data.

• Data Management: Organizing and overseeing data throughout its lifecycle.
• DataOps: Applying principles from software development to data operations to
improve efficiency and quality.
• Data Architecture: Designing the structure of the data systems and pipelines.
• Orchestration: Automating and managing the flow of data processing tasks.
• Software Engineering: Using programming and development best practices to build
robust and scalable systems.

The Role of a Data Engineer

Cisco Confiden+al
A data engineer is the professional who manages the entire data engineering lifecycle. Their
primary responsibility is to ensure that data is reliably moved from its source systems to its
final destination, where it is ready for use. This lifecycle includes:

1. Getting the Data: Extracting or ingesting raw data from various sources (e.g.,
databases, log files, APIs).
2. Storing the Data: Designing and managing data storage solutions like data
warehouses or data lakes.
3. Preparing the Data: Transforming the data to make it clean, consistent, and ready
for analysis or machine learning models.
4. Serving the Data: Making the processed data available to "downstream" users, such
as data scientists and data analysts, who will use it to generate insights and build
applications.

The Data Engineering Lifecycle:

The Data Engineering Lifecycle

The data engineering lifecycle is a conceptual framework that provides a holistic view of the
data engineer's role, shifting the focus from specific technologies to the data itself and its
ultimate purpose. It outlines the complete journey of data, from its origin to its final use.

The lifecycle is composed of five main stages:

• Generation: This is where the data is created or originates. It could be user actions on
a website, sensor data from a device, financial transactions, or any other source
system.

Cisco Confiden+al
 • Storage: Once generated, data needs to be stored somewhere. This involves designing
and managing the systems where the data resides, such as databases, data warehouses,
or data lakes. This stage supports the entire process of ingestion, transformation, and
serving.
• Ingestion: This is the process of taking the raw data from its source (generation) and
bringing it into the data ecosystem for processing and storage. It's the first step in
moving data.
• Transformation: In this stage, the raw, often messy data is cleaned, structured, and
manipulated to become high-quality, consistent, and usable information. This might
involve tasks like standardizing formats, aggregating data, or enriching it with other
data sources.
• Serving: This is the final stage where the processed, high-quality data is made
available for its intended purpose. This can include serving it to various
"downstream" use cases.

The downstream use cases mentioned in the diagram are:

• Analytics: Using the data to create reports, dashboards, and business intelligence.
• Machine Learning: Providing clean, prepared data for data scientists to build and
train machine learning models.
• Reverse ETL: This is the process of moving data from the data warehouse back into
operational business tools (like CRM or marketing platforms) to enrich them.

Undercurrents

The lifecycle also has "undercurrents"—these are critical concepts that are not a single stage
but rather flow through and influence every part of the lifecycle. They are the foundational
principles that a data engineer must apply at every step. The text and diagram identify these
as:

• Security: Protecting the data from unauthorized access throughout its entire journey.
• Data Management: Ensuring data quality, consistency, and governance.
• DataOps: Applying automation and collaboration principles to streamline the data
pipeline.
• Data Architecture: Designing the overall structure of the data systems to be efficient
and scalable.
• Orchestration: Automating and managing the execution of all the data pipeline
stages.
• Software Engineering: Using best practices from software development to build
reliable and maintainable data systems.

Cisco Confiden+al
The Evolution of the Data Engineer (in Simple Terms)

The role of the data engineer has changed dramatically over the decades, evolving from a
specialist in a single area to a master of a broad data ecosystem.

1. The Early Days (1980s-2000): The BI and Data Warehouse Engineer

• What was the focus? In the beginning, companies needed to analyze their business
data for reporting. The solution was the data warehouse, a single, massive database
designed for analysis.
• What did the engineer do? The engineer's job was to use tools to extract data from
various systems, transform it (clean it up), and load it into the data warehouse. Their
work was very focused on ETL (Extract, Transform, Load) and using SQL. They
were known as BI Engineers, ETL Developers, or Data Warehouse Engineers.
• Analogy: Think of them as a librarian for a small, well-organized library. Their job
was to take new books (data), catalog them neatly (transform), and place them on the
right shelves (load into the warehouse) so that people could find them easily.

2. The Big Data Era (2000s-2010s): The Big Data Engineer

• What was the focus? The internet exploded, and companies like Google and Amazon
had to deal with a new problem: "Big Data". This data was too massive, varied, and
fast to fit into a traditional data warehouse. New, open-source technologies like
Hadoop and Spark emerged to handle this scale.
• What did the engineer do? The engineer's role became much more complex. They
had to be software developers who could build and maintain huge clusters of servers.
They were "low-level infrastructure hackers" who focused on the nitty-gritty details
of these complex, distributed systems. The goal was simply to make the massive data
usable at all.
• Analogy: This engineer was more like a construction worker building a brand-new,
sprawling city. They had to lay the pipes, build the roads, and manage the power grid
themselves, often with new and experimental tools. The focus was on getting the
infrastructure built, no matter how complicated.

3. The Modern Era (2020s and beyond): The Data Lifecycle Engineer

• What is the focus? Today, many of the difficult "Big Data" problems have been
solved and packaged into easy-to-use, cloud-based services (like AWS, Google
Cloud, Snowflake). The technology is now more abstracted and modular. The focus is
no longer on building the low-level infrastructure but on managing the entire data
lifecycle.
• What does the engineer do? The modern data engineer is a "lifecycle engineer."
They act as a glue connecting various specialized tools like LEGO bricks. Their role
is to focus on the higher-level concerns that were once considered "enterprisey," such
as:
o Data Governance: Who can access what data?
o Data Quality: Is the data accurate and reliable?

Cisco Confiden+al
 o Security & Privacy: Are we complying with regulations like GDPR?
o Orchestration: Making sure all the different parts of the data pipeline run
smoothly and automatically.
• Analogy: The modern data engineer is now like a city planner or architect. Instead
of building every road and pipe from scratch, they use pre-built modules and services.
Their job is to design the overall city (data architecture), ensure all the different parts
connect seamlessly, manage traffic flow (orchestration), and enforce city laws and
regulations (governance and security). The tools are much easier, so their focus has
shifted to the strategic management of the data itself.

Data Engineering and Data Science

The Relationship: Upstream and Downstream

The text defines the relationship between data engineering and data science as upstream and
downstream.

• Upstream (Data Engineering): Data engineering is the work that happens first. It's
the foundational process of acquiring, storing, and preparing data. Data engineers
build and maintain the systems that create a clean, reliable, and accessible flow of
data.
• Downstream (Data Science and Analytics): Data science and analytics are the
processes that happen after the data has been prepared. Data scientists and analysts
take the high-quality data provided by the data engineers and use it to generate
insights, build models, and make predictions.

Essentially, data engineering provides the essential fuel (clean data) that data science and
analytics need to function effectively. Without the upstream work of data engineering, the
downstream work of data science would be impossible or, at the very least, incredibly
difficult and inefficient.

Example: A Ride-Sharing Company

Let's use a ride-sharing company like Uber or Lyft as an example to see how this
upstream/downstream relationship works in practice.

1. Upstream: Data Engineering

The data engineers at the company are responsible for the entire flow of data from its source.
Their tasks would include:

Cisco Confiden+al
 • Ingestion: They build data pipelines to collect massive amounts of real-time data
from various sources:
o Driver app data (GPS coordinates, ride status, driver availability)
o Rider app data (ride requests, bookings, payments)
o Customer support data (service tickets, chat logs)
o Maps and traffic data from third-party services.
• Storage: They design and manage the data lake and data warehouse where all this
raw and processed data is stored. They ensure the storage systems can handle
petabytes of data and are highly available.
• Transformation: They create processes to clean, structure, and aggregate the raw
data. This might involve:
o Combining driver and rider data to form a complete ride record.
o Calculating trip duration, distance, and cost.
o Standardizing location data.
o Ensuring all data is accurate and consistent.
• Serving: The engineers then make this high-quality, prepared data available in a user-
friendly format for downstream teams. They might serve it in a clean database table or
through an API.

2. Downstream: Data Science and Analytics

Now, the data scientists and analysts can do their work using the data provided by the data
engineers:

• Data Analysts: An analyst might use the clean ride data to create a dashboard
showing peak ride times, popular routes, and driver earnings. This helps the business
understand supply and demand.
• Data Scientists: A data scientist might use the same clean data to build a machine
learning model to:
o Predict ride demand: This model helps the company strategically place
drivers in anticipation of future demand surges.
o Optimize pricing (surge pricing): A model can be built to dynamically
adjust prices based on real-time supply and demand, which is only possible
with a consistent, real-time data feed.
o Improve estimated arrival times: The data scientist can build a model that
provides more accurate ETAs by using historical and real-time traffic data, all
of which was prepared by the data engineers.

The Data Science Hierarchy of Needs

The Data Science Hierarchy of Needs is a conceptual framework, often depicted as a

pyramid, that illustrates the foundational requirements for successful data science and
machine learning. It argues that advanced, "sexy" tasks like AI and deep learning are
impossible to achieve without first establishing a solid base of reliable data infrastructure.

The pyramid is structured into five levels, from bottom to top:

Cisco Confiden+al
 1. Collect: This is the most fundamental level, where raw data is gathered. It involves
instrumentation (adding code to applications to track events), logging data, and
collecting information from sensors or external sources. This is about getting the
data in the door.
2. Move/Store: Once collected, the data needs to be moved and stored effectively. This
level involves building reliable data flow, setting up the underlying infrastructure
and pipelines, and using ETL (Extract, Transform, Load) to place data into
structured and unstructured storage systems.
3. Explore/Transform: At this level, the raw data is made usable. This includes
cleaning the data, performing anomaly detection to find errors, and general data
preparation. This is where the data is refined and shaped.
4. Aggregate/Label: This is the first level where meaningful analysis begins. Data is
aggregated (e.g., calculating averages), segmented, and turned into metrics and
features. This is where data is prepared for modeling.
5. Learn/Optimize: This is the top of the pyramid, where the most advanced data
science tasks occur. This includes A/B testing, experimentation, building simple
ML algorithms, and ultimately, deep learning.

The Disconnect and the Role of the Data Engineer

The text highlights a critical "disconnect": many data scientists spend most of their time (70-
80%) toiling in the bottom three levels of the pyramid (Collect, Move/Store,
Explore/Transform). This is because they often lack the support of a dedicated data engineer.

An ideal scenario, as described in the text, is where a data engineer focuses on building and
maintaining the solid foundation—the bottom three levels. This allows the data scientist to
spend their time on what they are truly trained for: the top two levels of the pyramid
(Aggregate/Label and Learn/Optimize).

Cisco Confiden+al
Example: A Customer Churn Prediction Model

Imagine an e-commerce company wants to build a machine learning model to predict which
customers are likely to stop buying from them (customer churn).

• Without a Data Engineer:

o The data scientist has to start at the bottom of the pyramid.
o They must write scripts to collect log data from the website, payment history
from the database, and customer support tickets from a different system.
o They then have to build a makeshift pipeline to move/store all this data in a
single place.
o Next, they spend weeks cleaning the data, dealing with missing values, fixing
inconsistent product names, and figuring out which data is an outlier.
o Only after this difficult and time-consuming work can they begin to aggregate
and build features (like "number of purchases in the last 90 days") and finally
build the ML model. The model might take months to build because the data
preparation was so challenging.
• With a Data Engineer:
o The data engineer has already built a robust foundation. They have created
reliable, automated pipelines to collect, move, and store all the company's
data in a central, clean data warehouse.
o They have processes in place to automatically clean and prepare the data,
handling anomalies and standardizing formats.
o When the data scientist is tasked with the churn project, they can immediately
access the high-quality, pre-processed data. They don't have to worry about
how the data was collected or stored.

Cisco Confiden+al
 o The data scientist can now spend nearly all of their time on the higher levels:
aggregating the data into meaningful features and building the machine
learning model. The entire process is much faster and the final model is more
reliable because it's built on a solid data foundation.

Data Engineering: The Bridge to Value

The image and text illustrate a simple but crucial concept: data engineering is the essential
link that connects raw data to meaningful value.

• Left (Data from various sources): This represents the starting point. All companies
have data, but it exists in a scattered, raw, and often messy state. It could be in
databases, log files, spreadsheets, APIs, or various external systems. In this raw form,
it has potential but little direct business value.
• Middle (Data Engineering): This is the core of the process. The data engineer takes
this raw data and applies the skills and processes of data engineering to transform it.
This involves the full data lifecycle:
o Collecting the data from its various sources.
o Storing it in a centralized, accessible location.
o Cleaning and transforming it into a high-quality, consistent, and structured
format.
o Serving it in a way that is easy for others to use.

In doing so, the data engineer turns disorganized data into a usable asset. They are the
ones who make the data "ready for business."

• Right (Data Science and Analytics): This represents the final destination and the
point where business value is realized. The data scientists and analysts take the clean,
prepared data from the data engineer and use it to:
o Perform analytics to generate insights and create reports.
o Build machine learning models to make predictions, optimize processes, and
automate decisions.
o Ultimately, answer key business questions and drive strategic decisions.

Cisco Confiden+al
Data Engineering Skills and Activities

The Balancing Act of a Modern Data Engineer

The text describes the modern data engineer as a professional who must constantly balance
several competing priorities, or "axes," when designing and maintaining data systems. Unlike
in the past, where the focus was on mastering a few complex technologies, the modern data
engineer's role is to act as a system architect and manager, optimizing for a variety of
business goals.

The six axes that a data engineer must balance are:

1. Cost: Data engineers must be mindful of the financial implications of their choices.
For example, using expensive, high-powered cloud resources 24/7 might be a great
way to process data quickly, but it might be far more cost-effective to use cheaper
resources for a few hours a day.
o Example: Choosing a serverless, pay-as-you-go data processing service (like
AWS Lambda or Google Cloud Functions) over a dedicated, always-on cluster
to save money on compute resources.
2. Agility: This refers to the ability to quickly adapt and evolve the data infrastructure.
Data engineers must build systems that can be easily modified or extended to support
new data sources or business requirements without a complete overhaul.
o Example: Using modular, cloud-based services that can be easily swapped out
or reconfigured, rather than a monolithic, on-premise system that is difficult to
change.
3. Scalability: A data system must be able to handle a growing volume of data without a
significant drop in performance. Data engineers must design systems that can scale up
or down automatically as data needs change.
o Example: Using a distributed file system like Amazon S3 or a scalable data
warehouse like Snowflake that can handle petabytes of data, rather than a
traditional relational database that has a hard limit on its capacity.
4. Simplicity: Complexity is the enemy of maintenance and reliability. The modern data
engineer seeks to use tools and build systems that are as simple as possible to manage,
deploy, and understand. This reduces the risk of errors and makes the system easier to
maintain.
o Example: Adopting a single, integrated data pipeline orchestration tool (like
Apache Airflow) instead of a collection of different scripts and cron jobs that
are difficult to manage and debug.
5. Reuse: The data engineer should build data pipelines and components that can be
used for multiple purposes. This avoids redundant work and creates a more efficient
and consistent data ecosystem.

Cisco Confiden+al
 o Example: Creating a single, high-quality data table for customer information
that can be reused by multiple teams—for marketing analytics, sales reporting,
and machine learning—instead of each team building its own version of the
same table.
6. Interoperability: The data ecosystem is no longer dominated by a single vendor.
Data engineers must build systems that can seamlessly work with a variety of
different tools and services, from different vendors and open-source projects.
o Example: Ensuring that the chosen data warehouse can easily connect to a
variety of BI tools (like Tableau or Power BI) as well as machine learning
platforms (like SageMaker or Vertex AI).

What a Data Engineer Does (and Doesn't Do)

• What they DO: The modern data engineer's primary focus is on building and
managing the data lifecycle while optimizing for the six axes above. Their work
includes:
o Evaluating and choosing the best tools.
o Designing and implementing data architectures.
o Managing data flow and orchestration.
o Ensuring data quality, security, and governance.
• What they typically DO NOT DO: The text clarifies that a data engineer is distinct
from other data roles. While they have a functional understanding of these areas, they
are generally not responsible for:
o Building machine learning models.
o Creating reports and dashboards.
o Performing data analysis or defining KPIs.
o Building consumer-facing software applications.

Data Maturity and the Data Engineer

The Data Maturity Model Explained

The text proposes a simplified, three-stage model to describe a company's data maturity—
how effectively and widely an organization uses data. This maturity level significantly shapes
the work of a data engineer.

Cisco Confiden+al
Stage 1: Starting with Data

• Company Characteristics: The company is just beginning to use data. Goals are
fuzzy, data infrastructure is minimal, and the data team is small (often a single
person). The desire to use data exists, but the know-how is limited.
• Data Engineer's Role: The data engineer at this stage is a generalist. Their main
goal is to move quickly, get initial buy-in from management, and prove the value of
data.
o Focus:
§ Defining a basic data architecture from scratch.
§ Finding and auditing key data sources.
§ Building a minimal, functional data foundation to support initial
reports.
§ Often doing the work of a data analyst or data scientist as well to get
"quick wins."
o Example: A small e-commerce startup has sales data in one system and
customer behavior data in another. A single data engineer's job is to build a
simple pipeline to bring this data together into a single database, enabling the
CEO to run a basic report on the company's top-selling products. The
engineer's goal is to show immediate, tangible value to get more resources.

Stage 2: Scaling with Data

• Company Characteristics: The company has seen success with its initial data efforts
and is now ready to grow its data capabilities. The data team is expanding, and data
practices are becoming more formal. The challenge shifts from "can we use data?" to
"how can we use data at scale?"
• Data Engineer's Role: The data engineer becomes a specialist. Their focus moves
from just proving value to building robust, scalable, and maintainable systems.
o Focus:
§ Establishing formal data practices (like quality checks and standards).
§ Building scalable data architectures and pipelines that can handle
growing data volumes.
§ Adopting modern DevOps and DataOps principles for efficiency.
§ Building systems that specifically support the growing needs of
machine learning.
o Example: The e-commerce startup from Stage 1 is now a mid-sized company
with millions of customers. The single database is no longer sufficient. A team
of data engineers works to migrate the data to a scalable data warehouse (like
Snowflake), automate the data pipelines to run daily, and build a more reliable
infrastructure to support new analysts and data scientists joining the company.
The focus is on making the data reliable and scalable for future growth.

Cisco Confiden+al
Stage 3: Leading with Data

• Company Characteristics: The company is now truly "data-driven." Data is a core

part of its competitive strategy. Data is readily available for self-service analytics and
ML across the organization. The focus is on automation, governance, and finding new
ways to leverage data for competitive advantage.
• Data Engineer's Role: Data engineers become highly specialized and focus on
advanced, high-level challenges.
o Focus:
§ Creating automation for the seamless introduction of new data sources.
§ Building custom tools and systems that give the company a unique
competitive edge.
§ Implementing advanced data management, governance, and quality
controls.
§ Deploying tools like data catalogs and metadata management systems
to make data easily discoverable throughout the entire organization.
o Example: The now-large e-commerce company wants to personalize its
website in real time for every user. A specialized data engineer is tasked with
building a custom, low-latency streaming data pipeline that can collect user
clicks, process them instantly, and feed them to a machine learning model that
serves personalized product recommendations on the homepage within
milliseconds. The company is no longer just using data for reports; it is using
data to lead its market.

The Background of a Data Engineer

The text highlights that there is no single, formalized path to becoming a data engineer.
Unlike fields with standard university degrees, data engineering is a relatively new and
rapidly evolving discipline. As a result:

• Diverse Backgrounds: People enter data engineering from a variety of fields. The
most common and easiest transitions are from adjacent data-related fields. These
include:
o Software Engineering: These professionals already have strong programming
and software development skills, which are crucial for building robust data
pipelines.
o ETL Development or Database Administration: These roles have a strong
foundation in working with databases and data movement, which are core data
engineering tasks.
o Data Science or Data Analysis: These individuals have a deep understanding
of what is needed from data and the challenges of working with messy data,
making them well-suited to build the systems that provide clean data.
• Self-Study is Key: Because of the lack of formal training, a significant amount of
self-study is required to enter and succeed in the field.

The Core Skills of a Data Engineer

The text breaks down the necessary skills into two main categories: technical and holistic.

Cisco Confiden+al
1. Technical Skills

A data engineer must have a strong understanding of both data and technology.

• Data Management: This involves best practices for organizing, storing, and
maintaining data.
o Example: A data engineer needs to know how to design a data warehouse
schema (e.g., using star or snowflake schemas) so that data analysts can run
fast queries without performance issues.
• Technology and Tools: A data engineer must be aware of the vast ecosystem of data
tools and be able to evaluate them for a given business need.
o Example: Deciding whether to use a batch processing framework like Apache
Spark or a streaming framework like Apache Flink for a specific data pipeline,
and understanding the trade-offs in terms of latency, cost, and complexity.
• Software Engineering: This is a foundational skill set for building reliable,
maintainable, and scalable data systems.
o Example: A data engineer writes code for their pipelines, so they need to
follow best practices like version control (using Git), automated testing, and
proper code documentation to ensure their work is robust.
• DataOps and Data Architecture: These skills are about designing the overall data
system and applying operational principles to it.
o Example: A data engineer designs a data architecture where raw data is stored
in a data lake, cleaned data is stored in a data warehouse, and business-facing
data is in a separate reporting layer. They then use DataOps principles to
automate the entire process from data ingestion to reporting, ensuring
reliability.

2. Holistic Skills

Beyond the technical expertise, a successful data engineer needs a broader, more business-
oriented perspective.

• Understanding Data Consumers: A data engineer must understand the needs of the
"downstream" users—data analysts and data scientists. Their job is to provide data
that is useful and easy to consume.
o Example: Before building a new pipeline, a data engineer talks to the
analytics team to understand what specific metrics they need and in what
format, ensuring the final product directly serves their needs.
• Business and Competitive Context: The best data engineers view their work through
a business lens. They understand the company's goals and how data can provide a
competitive advantage.
o Example: Instead of just building a pipeline because it was requested, a data
engineer asks, "How does this pipeline help us serve our customers better or
make our operations more efficient?" This allows them to prioritize projects
that provide the most business value.

Cisco Confiden+al
The Business Responsibilities of a Data Engineer

The text emphasizes that a data engineer's success is not solely dependent on their technical
skills. They must also have a strong set of "macro responsibilities" that are crucial for
navigating a business environment and ensuring their technical work provides real value.

1. Communication: A successful data engineer must be able to communicate

effectively with both technical and non-technical people. This involves translating
complex technical concepts into simple, business-friendly terms.
o Example: A marketing manager asks the data team for "user data from the
website." A data engineer needs to ask clarifying questions like, "What
specific user actions are you interested in? Do you need real-time data or a
daily summary? What business decision will this data help you make?" Then,
they can explain the technical approach (e.g., "We will build a pipeline to
capture clickstream events and store them in a data warehouse") in a way the
manager can understand, building trust and ensuring the final product meets
the business need.
2. Scoping and Requirement Gathering: Data engineers need to go beyond simply
receiving a request. They must proactively work with stakeholders to understand what
needs to be built and why. This involves understanding the business and product
goals.
o Example: A data scientist asks for a "feed of all customer data." A good data
engineer doesn't just provide a giant table. They ask about the data scientist's
project: "Are you building a churn prediction model? What features do you
need? Do you need historical data or just the most recent information?" By
understanding the end goal, the engineer can scope the project correctly and
provide a specific, high-quality data set that is far more useful than a raw
dump of data.
3. Understanding Cultural Foundations (Agile, DevOps, DataOps): These are not
just technologies; they are mindsets and cultural practices. A data engineer must
understand that these practices are about collaboration, continuous improvement, and
breaking down organizational silos.
o Example: A data engineer implements an automated deployment pipeline for
their data jobs (a DevOps practice). This isn't just a technical trick; it's a
cultural shift. By automating the process, they reduce manual errors, speed up
delivery, and free up time for collaboration and innovation, which is the core
goal of DevOps. They must communicate this value to the rest of the team and
encourage its adoption.
4. Cost Control: A data engineer's choices have a direct impact on the company's
bottom line. They need to be cost-conscious and continuously monitor expenses to
ensure they are providing significant value without overspending.
o Example: A data engineer needs to decide how to process a large dataset.
They could use an expensive, always-on cluster, or they could use a more
cost-effective serverless option that only runs when needed. A good data
engineer will analyze the trade-offs and choose the option that delivers the
required value (e.g., a daily report) at the lowest cost, thereby demonstrating
their business acumen.

Cisco Confiden+al
 5. Continuous Learning: The data field changes at an incredible pace. A successful
data engineer must have a thirst for knowledge and the ability to filter out fads from
genuinely useful new technologies.
o Example: When a new tool like "Data Lakehouse" emerges, a data engineer
doesn't immediately drop everything to learn it. Instead, they evaluate it
against their company's needs. They might read about its benefits and
drawbacks, compare it to their current technology, and decide that for now,
their existing data warehouse is sufficient. They will, however, keep an eye on
the new technology, adding it to their mental toolkit for future consideration.

Technical Responsibilities and the Data Engineering Lifecycle

The text frames a data engineer's technical responsibilities around the Data Engineering
Lifecycle and its Undercurrents. The primary goal is to build architectures using various
components (both prepackaged and custom-built) that serve the stages of this lifecycle
efficiently.

• Lifecycle Stages: A data engineer must understand the technical requirements of each
stage:
o Generation: Understanding where the data comes from (e.g., from an
application's database, a web log, or a sensor).
o Storage: Deciding which storage solution is best for a given type of data (e.g.,
a data warehouse for structured data vs. a data lake for raw, unstructured data).
o Ingestion: Building pipelines to reliably move data from the source to the
storage location.
o Transformation: Writing code or using tools to clean, aggregate, and
structure the data.
o Serving: Making the processed data available for use by other teams.
• Undercurrents: The technical decisions made at each stage are guided by these
underlying principles:
o Security: Implementing access controls and encryption.
o Data Management: Ensuring data quality, governance, and lineage.
o DataOps: Automating deployments and monitoring data pipelines.
o Data Architecture: Designing the overall structure of the data system.
o Orchestration: Scheduling and managing the workflow of data tasks.
o Software Engineering: Using best practices to write code that is reliable,
testable, and maintainable.

The Importance of Coding

The text strongly asserts that a data engineer must know how to code and possess
"production-grade software engineering chops." While modern tools have simplified many
low-level tasks, the nature of a data engineer's coding has evolved:

• From Low-Level to High-Level: Instead of writing complex code to manage clusters

or storage systems, data engineers now often work with high-level abstractions and
managed services. Their coding is more focused on scripting, writing pipelines as
code, and integrating different tools.

Cisco Confiden+al
 • A Competitive Edge: A deep understanding of software engineering remains a
significant advantage. The ability to write clean, efficient, and well-tested code is
crucial for building custom solutions and debugging complex architectural issues
when a prepackaged solution falls short.

Key Programming Languages for Data Engineers

The text identifies a core set of languages essential for a data engineer, categorized as
primary languages:

• SQL (Structured Query Language):

o Why it's essential: SQL is the universal language for interacting with
databases and data lakes. It has re-emerged as the primary way to query and
manipulate data in the age of modern data warehouses and distributed query
engines.
o Use case: Writing a query to transform and aggregate sales data before
loading it into a reporting table.
• Python:
o Why it's essential: Python acts as a "bridge language" that connects data
engineering with data science. Its extensive ecosystem of libraries (pandas,
NumPy) and its common APIs for popular data tools (PySpark, Airflow) make
it a crucial language for scripting, automation, and data manipulation.
o Use case: Writing a Python script to call an API, process the JSON data, and
load it into a data warehouse.
• JVM Languages (Java or Scala):
o Why they're essential: These languages are the backbone of many popular
open-source "big data" frameworks like Apache Spark. They are generally
more performant than Python for data processing at scale and offer access to
lower-level features of these frameworks.
o Use case: Writing a custom data processing job in Scala or Java to perform
complex, high-performance transformations on massive datasets using Apache
Spark.
• Bash:
o Why it's essential: As the command-line interface for Linux, Bash is crucial
for scripting automation tasks, managing files, and running system commands.
It's often used within orchestration frameworks to execute various parts of a
data pipeline.
o Use case: Writing a Bash script to automate the backup of a directory of log
files before they are ingested by a data pipeline.

=====================================

Cisco Confiden+al
The Continuum of Data Engineering Roles

The text proposes a useful distinction between two types of data engineers, borrowing from a
similar concept in data science. This model helps to clarify the different kinds of work data
engineers do and avoids the "unicorn" myth that every data engineer does everything.

1. Type A Data Engineers (A for Abstraction)

• Focus: A Type A data engineer focuses on building data solutions by avoiding

custom work and instead using existing, pre-built solutions. They use off-the-shelf
products, managed cloud services, and tools to create a data architecture that is as
simple and abstract as possible.
• Skills: They excel at integrating different tools and services, understanding their
APIs, and using them to manage the data engineering lifecycle. They are experts in
choosing the right tool for the job from a vast landscape of options.
• Where they work: They are commonly found at companies of all sizes and data
maturity levels, from startups to large enterprises. Their role is to lay a solid, low-
effort foundation that gets the company started with data quickly.
• Analogy: A Type A data engineer is like a professional who builds a house using
high-quality, pre-fabricated components (walls, windows, etc.). They are skilled at
assembly, integration, and ensuring all the parts fit together perfectly without having
to build anything from scratch.

2. Type B Data Engineers (B for Build)

• Focus: A Type B data engineer's role is to build custom data tools and systems
from the ground up. This is necessary when a company's data needs are unique,
highly specific, or so critical to its core business that off-the-shelf solutions are not
sufficient.
• Skills: They have deep software engineering expertise and an understanding of low-
level systems. They can build scalable data processing frameworks, custom databases,
or specialized tools that provide a unique competitive advantage.
• Where they work: They are more common at companies with a higher level of data
maturity (Stage 2 and 3) or at tech giants where data is the core product.
• Analogy: A Type B data engineer is like an architect and builder who designs and
constructs a custom-made building from scratch, often using their own specialized
methods and materials to create something unique and perfectly suited to a specific
purpose.

Example: A Music Streaming Company

Let's use a music streaming service to see how these two types of data engineers might work
together.

Scenario 1: Early Stage Startup (Type A)

• The company is just starting. Their main goal is to collect user listening data to create
a basic playlist and analytics dashboard.

Cisco Confiden+al
 • A Type A data engineer would be hired. They would not build a custom data
pipeline. Instead, they would use a combination of managed cloud services:
o They might use a third-party service to ingest user listening data from the
application.
o They would load this raw data into a managed cloud data warehouse like
Snowflake or BigQuery.
o They would use a managed orchestration tool like Airflow to schedule the
daily data cleaning and aggregation tasks.
• The entire system is built using pre-packaged, abstracted services, allowing the
company to quickly get a functional data platform without a massive engineering
effort.

Scenario 2: Mature Tech Giant (Type B)

• The company is a tech leader with hundreds of millions of users. They want to create
a real-time, personalized music recommendation engine that is better than anything
else on the market.
• A Type B data engineer would be hired. An off-the-shelf tool wouldn't be fast or
customizable enough for this unique, competitive need. The engineer's work would be
to:
o Build a custom streaming data processing framework (likely using
technologies like Kafka and Flink) to process millions of listening events per
second.
o Develop a proprietary data storage system optimized for lightning-fast
lookups by the recommendation engine.
o Create custom APIs and tools to serve the real-time recommendations, all built
and managed in-house to gain a competitive edge.
• In this case, the custom-built infrastructure is the core of the business and cannot be
replicated with generic, off-the-shelf tools.

Data Engineers Inside an Organization: A Central Hub

The text and the accompanying diagram (Figure 1-9) position the data engineer as a central
hub, interacting with users both inside and outside the company. The nature of a data
engineer's work, and who they primarily serve, can be divided into two main categories:
internal-facing and external-facing.

Cisco Confiden+al
Internal-Facing Data Engineer

An internal-facing data engineer primarily serves users within their own organization. These
users are typically the downstream consumers of the data pipeline, as shown in Figure 1-6
(Data science and analytics) and the Data Science Hierarchy of Needs (Figure 1-5).

• Who they serve: Data scientists, data analysts, business intelligence (BI) teams,
product managers, and other software engineers.
• What they do: Their main goal is to build and maintain the data infrastructure that
empowers these internal teams to do their jobs effectively. Their work is a direct
application of the data engineering lifecycle (Figure 1-1).
o They build pipelines to ingest data from various internal sources (e.g., sales
databases, user logs).
o They transform this data to make it clean, consistent, and ready for analysis.
o They serve this high-quality data to a central data warehouse or lake, where
analysts can create dashboards and data scientists can build machine learning
models.
• Example: A data engineer at a large retail company builds and maintains the data
warehouse. Their primary users are the marketing analysts who need clean data to
measure campaign performance and the data scientists who use the data to train a
model for customer segmentation. The engineer's success is measured by the
reliability and quality of the data they provide to these internal teams.

External-Facing Data Engineer

An external-facing data engineer, on the other hand, serves users or systems outside of their
organization. Their work is often directly related to the company's products and services, and
the data they manage is a key part of what the company offers to its customers.

• Who they serve: The company's own customers, partners, or other external systems.
• What they do: Their responsibilities involve building data products, APIs, or systems
that directly interact with people or services outside the company.
o They might build data pipelines that power a feature within a customer-facing
application.
o They could create and maintain a public-facing API that allows partners to
access specific data.
o They might be responsible for data pipelines that are part of a core data
product sold by the company.
• Example: A data engineer at a weather forecasting company builds the data pipelines
that collect raw weather data from various sources. Their main responsibility is to
process this data and make it available in a clean, consistent format through a public
API that other companies (the "external users") can pay to use in their own
applications. The engineer's success is directly tied to the performance, reliability, and
security of this external-facing data product.

Cisco Confiden+al
The Role of an External-Facing Data Engineer

An external-facing data engineer is responsible for the data systems that directly support a
company's customer-facing applications or products. Their work is characterized by a
"feedback loop" (as shown in Figure 1-10) where data from an application used by an
external user is processed by the data engineer's systems and then sent back to the application
to provide a better user experience.

This role is distinct from an internal-facing data engineer, who primarily serves the needs of
internal teams like data analysts and data scientists.

Key Responsibilities and Challenges

The unique nature of serving external users presents a specific set of challenges that external-
facing data engineers must address:

1. Handling High Concurrency: External applications like social media or e-commerce

platforms can have millions of users interacting at the same time. The data systems
must be designed to handle a much larger number of simultaneous queries and data
writes than a typical internal reporting system.
o Example: A gaming company's data engineer needs to build a leaderboard
system. The system must be able to handle thousands of players checking their
rank simultaneously after a match, so the query engine needs to be extremely
performant and scalable to avoid slow load times or crashes.
2. Query Limitations: To protect the infrastructure from being overwhelmed, external-
facing systems often need to place strict limits on what users can do.
o Example: A data engineer at a company that provides a public-facing
analytics dashboard for its clients must implement query limits. They might
restrict the time range of queries (e.g., only allow data from the last 90 days),
limit the number of data points returned, or prevent complex joins to ensure
one user's query doesn't slow down the system for everyone else.
3. Complex Security: Security is a much more sensitive issue when dealing with
external data, especially with multitenant data (where data from many different
customers is stored together). The engineer must ensure that customers can only
access their own data.

Cisco Confiden+al
 o Example: A data engineer for a SaaS (Software as a Service) platform stores
all customer data in a single, large table. They must implement a robust
security layer that ensures that when Customer A queries their own data, they
can never accidentally or maliciously see Customer B's data. This requires
careful architectural design and strict access controls at every step of the data
pipeline.
4. Real-time Feedback Loop: The systems built by external-facing data engineers often
operate in a tight feedback loop with the application.
o Example: On an e-commerce site, a user adds an item to their cart. The data
engineer's systems immediately process this event, and the application's
recommendation engine uses this real-time data to instantly suggest related
products on the same page. The data is processed and sent back to the
application in real time to create a better user experience.

The Role of an Internal-Facing Data Engineer

An internal-facing data engineer's primary responsibility is to serve the data needs of internal
stakeholders within their organization. As depicted in Figure 1-11, the work is a
straightforward, one-way flow: data moves from the company's internal source systems, is
processed by the data engineers, and is then used by internal teams for analytics and ML.

This is in contrast to the external-facing role, which has a feedback loop with customer-
facing applications. The work of the internal-facing data engineer is foundational and is often
a prerequisite for more advanced data work, whether internal or external.

Key Activities and Focus Areas

An internal-facing data engineer's work aligns closely with the foundational levels of the
Data Science Hierarchy of Needs (Figure 1-5). Their main goal is to create a solid, reliable
data foundation that empowers the downstream analytics and machine learning teams.

• Building and Maintaining Data Pipelines: This is a core activity. They build the
pipelines that extract, transform, and load (ETL) data from various internal source
systems (e.g., customer relationship management (CRM) software, enterprise resource
planning (ERP) systems, internal databases).
• Creating and Managing Data Warehouses: They design and manage the central
data repositories, such as a data warehouse, that are specifically structured for internal
analysis and reporting.
• Supporting BI and Reporting: A significant part of their job is to ensure that the
data is clean, consistent, and readily available for the business intelligence team to
create dashboards and reports that inform business decisions.

Cisco Confiden+al
 • Enabling Data Science and ML: They prepare and curate the data that data scientists
and machine learning engineers need to build their models. This involves tasks like
feature engineering, data cleaning, and creating a reliable source of training data.

The Relationship Between Internal- and External-Facing Roles

The text highlights a key relationship between the two types of roles:

• Internal as a Prerequisite: Internal-facing data is typically a prerequisite for

external-facing data. A company must first have its own internal data systems in order
to build robust external-facing data products. For example, a company cannot offer a
data product to its customers (external-facing) without first collecting, cleaning, and
managing that data in its internal systems.
• Different Requirements: The two roles have very different requirements. Internal-
facing systems generally do not need to handle the same level of query concurrency as
external-facing systems. However, they are still critical for the business's day-to-day
operations and strategic decisions.

Data Engineers and Other Technical Roles:

Data Engineers as a Central Hub

The text and Figure 1-12 illustrate that a data engineer is a central "hub" connecting various
technical roles within an organization. They sit between a group of upstream data
producers and a group of downstream data consumers. This position requires them to be
skilled at collaboration and communication with a diverse range of technical professionals.

Upstream Stakeholders (Data Producers)

These are the people who generate the data that the data engineer will ultimately work with.
A data engineer must understand how these stakeholders produce data to ensure it can be
effectively used in the data engineering lifecycle.

Cisco Confiden+al
 • Software Engineers: They build the applications and systems that form the backbone
of the business. The data engineer collaborates with them to understand the data that
these systems generate, such as event data and logs. The goal is to design applications
in a way that the data they produce is easy to collect and process.
• Data Architects: They operate at a higher level of abstraction, designing the overall
blueprint for the company's data ecosystem. While data architects focus on the "big
picture," data engineers work hand-in-hand with them to implement these designs,
especially in cloud environments where the lines between architecture and
implementation are blurred.
• DevOps and Site-Reliability Engineers (SREs): These professionals manage the
operational infrastructure. They produce data through operational monitoring and may
also be downstream consumers of data for their dashboards. Data engineers interact
with them to ensure the smooth operation and monitoring of data systems.

Downstream Stakeholders (Data Consumers)

These are the people who use the clean, prepared data that the data engineer provides to
create business value. The data engineer's success is often measured by how well they serve
these roles.

• Data Scientists: They build predictive models and make recommendations.

According to the text, in ideal scenarios, a data scientist should not be spending the
majority of their time on data preparation. A data engineer's job is to automate this
work, providing clean, scalable data so that data scientists can focus on model
building and analysis. This collaboration is crucial for moving models from
development to production.
• Data Analysts: They focus on understanding past and present business trends by
running queries and creating reports and dashboards. Data engineers work with data
analysts to provide them with the new data sources they need and to collaborate on
improving data quality, as analysts are often domain experts in their data.
• Machine Learning Engineers and AI Researchers: These roles overlap
significantly with data engineering and data science. ML engineers focus on
developing, training, and deploying ML models at a production scale. Data engineers
support them by providing the necessary infrastructure and data pipelines, allowing
the ML engineers to focus on the advanced aspects of their work, like MLOps and
model deployment. AI researchers, on the other hand, are highly specialized and often
focus on developing new AI techniques, relying on ML engineers and other support
teams for their engineering needs.

Data Engineers and Business Leadership:

Data Engineers as Strategic Partners

The text highlights a crucial shift: data engineers are no longer just technical implementers.
As data has become a core business asset, data engineers are now key players in strategic
planning and major initiatives. They act as the "glue" that connects business vision with
technical reality.

Cisco Confiden+al
Interaction with the C-Suite

Data engineers provide critical technical insights to C-level executives, helping to shape the
company's data strategy.

• CEO (Chief Executive Officer): The CEO sets the overall vision for the company.
Data engineers provide the CEO with a "window into what's possible with data,"
informing them about the company's data assets and the feasibility of major initiatives
like a cloud migration or the adoption of new technologies.
• CIO (Chief Information Officer): This is the senior executive responsible for
internal-facing IT. A CIO works closely with data engineering leadership to define
and execute major data initiatives, such as implementing a new CRM or data system.
They help shape the data culture within the organization.
• CTO (Chief Technology Officer): A CTO is similar to a CIO but focuses on
external-facing technology and products. Data engineers often report to the CTO and
provide the technical expertise needed to support external-facing applications (e.g.,
mobile apps, IoT devices) that are critical data sources.
• CDO (Chief Data Officer) / CAO (Chief Analytics Officer): These roles are
specifically focused on data as a business asset. The CDO oversees the company's
data strategy and assets, while the CAO focuses more on the analytics and decision-
making side. Data engineers are essential for both, as they build the infrastructure that
the CDO defines and that the CAO uses to drive business value.
• Chief Algorithms Officer (CAO-2): This is a highly technical, recent C-suite role
focused on data science and machine learning. A CAO-2 relies on data engineers to
provide the data, automation, and scale necessary for their advanced ML initiatives.

Interaction with Management Roles

Data engineers also work closely with project and product managers to ensure their technical
work aligns with business needs and is delivered efficiently.

• Project Managers: These managers are responsible for keeping large initiatives on
track (e.g., a multi-year cloud migration). Data engineers collaborate with them in an
Agile or Scrum framework, providing updates on progress, identifying roadblocks,
and helping to plan sprints. The project manager's role is to balance the technical
team's cadence with the ever-changing needs of the business.
• Product Managers: In the context of data, product managers oversee "data
products." They balance the needs of the customer and the business with the technical
work being done by data engineers. A data engineer works with a product manager to
understand the customer's needs and build data systems that directly support the
company's data products.

Cisco Confiden+al