4 Data Engineering

Fundamentals You Need

to Be Successful

[Link]
[Link] 1
4 Data Engineering Fundamentals You Need to Be Successful

Contents
Introduction: 3 Part Three:
How To Become a Successful Data Engineer 15
Part One:
Evolution of the Data Landscape 4 Part Four:
The Need for an End-to-End Data
Part Two: Management Platform 19
The 4 Fundamentals of Data Engineering 7
1. Data Discovery and Lineage 8 About Informatica® 21
2. Data Ingestion 10
3. Data Processing 11
4. Data Quality 13

[Link] 2
4 Data Engineering Fundamentals You Need to Be Successful

Introduction
Adam is a lead data engineer on the data and analytics team in To manage such complex data problems, data engineers need an easy-
his organization. He supports all major business units across sales, to-use tool that can handle complex data patterns. This makes it easier to
marketing, research and development, finance and operations. integrate with any source or targets, whether in the cloud or on-premises.

It’s Tuesday, 11:30 p.m. and Adam — exhausted and stressed — is trying This eBook is designed for early-stage professionals and aspiring data
to keep his eyes open to finish a big project. By tomorrow morning, the head engineers: Those who want to solve the most complex data processing
of sales needs some key reports, which sounds like an easy enough request. problems using innovative tools and technologies. It explains the
However, recent changes in the organization’s CRM data structure have cornerstones of data engineering and outlines what it takes to become a
caused some challenges with pulling accurate data. successful data engineer in today’s market by:

After hours of investigating, Adam resolved the problem. The culprit? • Providing a brief history of the data landscape
New data structures introduced in the CRM system. While happy he fixed • Detailing the four fundamentals of data engineering (data discovery and
the issue, Adam knows the only way to prevent this from happening again lineage, data ingestion, data processing and data quality)
is to automate the overall data processing — including schema drift.
• Outlining the key traits and skillsets of a successful data engineer
This is a common problem for many data engineers like Adam who aspire
to solve tough, complicated data problems. This high-stress job can lead Data engineers move, shape and transform data from the source to the tools
to burnout, which can inadvertently introduce technical debt1 or lead to quick that extract insights — ensuring that data can be trusted. In many ways, data
fixes. This can, unfortunately, have a larger impact once the products engineers are the heroes in data-driven projects. They are the experts who
or solutions mature. Rather than building innovative designs for intelligent preserve the credibility of business intelligence (BI) and advanced analytics,
data enterprises, data engineers oftentimes spend more time fixing and support artificial intelligence (AI) and machine learning (ML) models.2
existing data pipelines and workflows. Or worse, working on manual tasks
and complex code fixes. So, if you’re a data engineer, dust off that cape — your superpowers
are needed!
1
What Is Technical Debt? | Definition and Examples ([Link])
2
[Link]
3
4 Data Engineering Fundamentals You Need to Be Successful

Part 1

Evolution of the Data Landscape

The evolution of the data landscape plays
Figure 1: Evolution of the data landscape.
a critical role in understanding modern
data engineering. Setting the DeLorean to • Compute is cheaper

travel back to the 1970s and ‘80s (for you • Storage is cheaper
• Engineering is cheaper
“Back to the Future” movie fans), we would see
• Compute is cheaper • Cost optimization: FinOps
mainframes and midrange machines storing • Storage is cheaper
• Compute is cheaper
most enterprise data. As time progressed into • Engineering is cheaper 4
• Storage is cheaper
Data Fabric
the 1990s, much of this shifted into distributed • Engineering is Metadata Maturity,
• Compute is expensive VERY expensive
3 Automation and
applications like ERP, SCM, CRM and other Cloud-Native Operationalization
• Storage is expensive
systems. Each of these applications was 2
Data Warehouses
• Engineering is expensive
designed to store its own data and presented (but worth it) On-Premises CLOUD LAKEHOUSES
Data Marts
different access challenges. and Data CLOUD LAKEHOUSES Unified Data Access
1 Warehouses and Management
Cloud-Native
On-Premises
Data Lakes
Moving into the 2000s, as illustrated in Figure Data Marts
and Data
On-Premises
Spark-based
1, there were on-premises data marts and data Warehouses Data Lakes

warehouses. There was furious debate about

the best way to structure the warehouses —
Kimball or Inmon. Regardless of the right answer, Early 2000s Mid-2000s–Mid-2010s Mid-2010s–Mid-2020s 2020s 2020s

there were some common truths. Compute and

storage were expensive!

[Link] 4
4 Data Engineering Fundamentals You Need to Be Successful

Part 1

Evolution of the Data Landscape (continued)

But the value of the data warehouse was worth the expense. In fact, Hadoop Map-Reduce wasn’t well suited for data management,
data warehouses delivered so much value the world moved towards so Spark soon made its way into the architecture. With strong in-memory
purpose-built data warehouse appliances, and those were very expensive. processing, it opened the door to near real-time analytics and efficient
So expensive that data modelers and data engineers were tasked with big data management.
optimizing the systems and reducing the operational costs. Those data
modelers and engineers were also expensive, but again, they were worth it. Although compute and storage were inexpensive, Hadoop was painfully
complex, and engineers who really knew how to make it sing were extremely
Around 2006, Hadoop came along, and it looked like Big Data was going hard to find. And if you were able to find skilled Hadoop engineers, they were
to take over the world. As we know, that didn’t quite pan out, but even so, extremely pricey.
Hadoop had a massive impact on data management: The notion that
compute and storage are expensive got flipped on its head. Storage and Technology again evolved and here we are today rushing to the cloud. And
compute, relatively speaking, now became cheap. why not? With cloud, storage is cheap and compute is cheap because it is
consumption based. And the final nail in Hadoop’s coffin? Engineering is
More importantly, Hadoop made it OK to say, “Throw more horsepower at it.” handled by the cloud ecosystems.
Prior to Hadoop, uttering those words was very likely a career limiting move.
If you had Teradata, Netezza, Exadata, etc. and you said to your manager: This all leaves us in a state of mixed architectures. Most organizations
“Performance is a little slow. Let’s buy more processing…” Well, that was very today are in some stage of architecture modernization. Whether it’s cloud
likely a seven- or even eight-figure suggestion and data engineers would get data lake and warehouse, data fabric or data mesh, implementing a
shown the door. data science practice or something else, there are significant challenges
to overcome. Mainframes, distributed applications, relational databases,
cloud ecosystems, batch, change and real- time latencies, evolving
technology and expense profiles and self-service data management —
all while still delivering business requirements and meeting service level
agreements (SLAs)? Who can successfully pull all these things together?
It’s the data engineer, of course.
[Link] 5
4 Data Engineering Fundamentals You Need to Be Successful

Part 1

Evolution of the Data Landscape (continued)

The Role of the Data Engineer and predicting. Data engineers work in the And the responsibilities are clear: Data engineers
background to help answer a specific question. build and maintain the systems that store and
It is estimated that there will be around 200
The more data the company processes, the organize data. Data scientists analyze data to
zettabytes of data by 2025, with 100 zettabytes
more time is spent on analyzing it. predict trends and answer questions that help make
of them stored in the cloud.3 Storing zettabytes
meaningful business decisions. The data engineer
of data is challenging on its own, but it can be
Data engineers design and implement the engineers the data for the scientist to work on —
even more difficult to gain value from such
architectures necessary for data scientists to a little bit like a lab technician and a scientist.4
a huge amount of information. The data that’s
be successful. After all, without clean, trusted
collected will have security and governance
data, what’s the point of running analytics? The below pyramid illustrates how data engineering
requirements that are mandatory to protect.
You can see why data engineers are viewed as assists in data science operations.
Poor data quality causes misinformed business
the backbone of any data team.
decisions, which can lead to pricy mistakes.
The data that is collected not only needs Data
Data Engineering Versus Data Science AI and Science
to be secure, but it must also be clean and deep learning

consistent. This is where data engineering The amount of time spent on data preparation Analytics, metrics,
segments, aggregates
comes into play. versus data analytics is disproportionate. and features
Based on our experience, less than 20% of time
Data quality
The role of the data engineering team is to take is spent analyzing data, while over 80% is spent (cleaning, anamoly detection)
and data preparation
data in a raw and unusable format and transform collectively on searching for, preparing and Data
Engineering
it into a clean state so it can be used by business governing the appropriate data. Reliable data flow, infrastructure pipelines,
ETL, structured and unstructured data storage
leaders and data science teams for forecasting

Cataloging and data discovery

Figure 2: How data engineering supports data science projects.

3
How Much Data Is Created Every Day in 2022? [NEW Stats] ([Link])
[Link]
6
4
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering

While the data landscape and its impact
Figure 3: Data engineering supporting data analytics.
on data engineers are constantly changing,
the core fundamentals of the data Data Governance & Protection
engineering processes listed below have
Data Engineering Data Analytics
not changed much:
Source 1. Data Discovery
Enterprise
Data & Lineage Ad-hoc Analysis
Analytics
1. Data discovery and lineage
2. Data ingestion
2. Data Ingestion
3. Data processing Machine Self-Service
Learning Analytics
4. Data quality
3. Data Processing

Over time, they have been enriched with Business Data Quality
4. Data Quality Intelligence Reports
the advancement of cloud storage, computing
Parse | Cleanse | Enhance
resource optimization and emerging
data architectures. Figure 3 shows how DevOps-CI/CD | DataOps | MLOps | Orchestration | Data Security | Data Observability
these processes support data analytics and Data Platform and Operations
machine learning.
2

[Link] 7
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

1 Data Discovery and Lineage

Before a data engineer starts building pipelines The main steps in the data discovery • Data discovery and lineage: Data discovery
to clean the data for business needs, one of the process include: begins with scanning for data across
fundamental steps in a data lake architecture your organization’s landscape. This can
• Connect to data assets: It is very important
is data discovery. This includes solving modern- include on-premises or cloud-based sources
to have the right connectors to scan and
day data challenges, such as enterprise-wide from data warehouses to extract, transform
profile data assets as part of data discovery.
data democratization, privacy and trust and load (ETL) data and BI tools, SaaS
Manually gathering data assets and
assessments for compliance, and greater applications and more.
information takes longer, so out-of-the-box
data insights to ensure digital transformation connectors to these data assets are critical in
success. Data discovery enables organizations Once you have located your data, you can
understanding existing data and anomalies.
to identify, catalog and classify business- discover further information about it,
• Curate data preparation: Once you
critical and sensitive data, so you can govern such as its structure, content and
establish connections to the required data
it for meaningful purposes with increased relationships. This information can be used
assets, the next step is creating, organizing
transparency. to catalog data, enrich its metadata and
and maintaining data sets so they can be
provide context to help you understand what
accessed and used by people looking for
data exists, its source, its lineage and how
information. It involves collecting, structuring,
it’s related to other data.
indexing and cataloging data for users in an
organization or group or for the general public.

[Link] 8
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

The ability to map and verify how data • Measure and optimize data value: And it makes sense. Measuring data assets
has been accessed and changed is key Unless we derive meaningful information helps answer critical questions, such as:
to generating a detailed record of where for business decisions from an underlying
• Who is using what asset, what feature and when?
specific data originated, how it changed and dataset, the data itself is worthless.
how it gets used. This is valuable for finding Hence it is very important to constantly • What is the catalog adoption rate?
and fixing gaps in necessary data prior to measure and optimize data value through • What are the most accessed assets?
data pipeline building. It is also important for intuitive, real-time data insights. A recent poll
• Who are the top contributors and collaborators?
responding to reporting requirements and we conducted with webinar participants
audit requests for regulatory compliance. indicates that the top four data asset-related
Measuring and optimizing data value help
But how do you keep up with data lineage information organizations are most interested
you to define data as an asset. The definition
given the volume and scale of data today? in analyzing are:
of an asset is something owned or controlled,
[ Data inventory exchangeable for cash, and that generates
You need an AI-powered data lineage
a benefit.
solution that includes a data catalog [ Catalog adoption
with advanced scanning and discovery [ Data usage Data catalog solutions such as an enterprise
capabilities to ensure you capture all the
[ Data value data catalog and cloud data governance help
relevant metadata from your data sources.
you measure and optimize data value by defining
The solution also needs to provide detailed,
it as an asset. To learn more, check out this data
end-to-end data lineage across the cloud and
sheet on data asset analytics and hear from
on-premises.
industry experts in this on-demand webinar.

[Link] 9
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

2 Data Ingestion

Once data discovery is complete, the next step in data processing is There are many ways to ingest data. The organization’s data strategy,
to ingest the data into a data lake. Data engineers use data ingestion business requirements and when the data is needed determine the
pipelines to handle the scale and complexity of business demands for appropriate data ingestion method. Below are the three most common
data. Data ingestion has also become a key component of self-service data ingestion techniques:
platforms for analysts and data scientists to access data for real-time
analytics, ML and AI workloads. Data ingestion is a process that extracts • Batch processing: Collects data from sources incrementally and
data from the source where it was created or originally stored and loads it sends batches to the application or system where the data is to be
into a destination or staging area. A simple data ingestion pipeline might used or stored.
apply one or more light transformations, enriching or filtering data before • Real-time processing: The data is loaded as soon as it is recognized
writing it to a set of destinations, a data store or a message queue. by the ingestion layer and is processed as an individual object.
• Micro batching: The data is divided into groups and ingested in
smaller increments. This makes it more suitable for applications
that require data in real time.

[Link] 10
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

3 Data Processing

Data processing engines take data Constructing data pipelines with complex There are also different types of data processing
processing pipelines, abstract the business transformation and business logic is the core techniques based on the source data:
logic (either simple or complex) and process the responsibility of data engineering. It requires
data on frameworks such as Apache Spark, advanced skills to design a program for • Transactional processing is deployed in
in a streaming or batch mode, on-premises or continuous and automated data exchange. mission-critical situations. These are situations
in the cloud. These data pipelines are commonly used for: which, if disrupted, will adversely affect
business operations. An example is bank
More complex transformations such as joins, • Preparing data into a single location to transactions.
aggregates and sorts for specific analytics, streamline ML projects • Real-time processing is like transaction
applications and reporting systems can be done • Integrating data from various Internet of processing, where output is expected in
on the data inside the data lake using additional Things (IoT) systems and connected devices real time. However, this differs from
pipelines. From there, the final data is loaded transactional processing in terms of how
into a cloud data warehouse for analytics.
• Moving the prepared data into a cloud
data warehouse data loss is handled. Real-time processing
This process is typically done using (ETL) computes incoming data as quickly as
or extract, load, transform (ELT) techniques, • Bringing data into the data warehouse to
possible. If it encounters an error in incoming
which is often used for BI, analytics, reporting make informed business decisions
data, it ignores the error and moves to the
and AI/ML. next chunk of incoming data. GPS-tracking
applications are the most common example
of real-time data processing.

[Link] 11
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

• Batch processing is when chunks of data, You need a single platform that provides the databases or applications
stored over a period of time, are analyzed following data processing techniques and can: • Infer schema and detecting column
together or in batches. Batch processing changes for any data source and file formats
is required when a large volume of data • Support just about any integration patterns,
• Cleanse data including data quality rules, data
needs to be analyzed for detailed insights. ingestion and ETL/ELT with thousands of
masking and data domain rules with out-of-
For example, a company’s sales figures metadata-aware connectors
order (OOO) transformations
over a period are typically processed in • Support just about any data processing
batches because there is a large volume of patterns, such as batch, real-time and near
data involved, and the system will take time real-time through various engines
to process it.
“With Informatica, we
• Develop pipelines on day one with achieved our key business
• Distributed processing breaks down large zero infrastructure footprint with
datasets and stores them across multiple
objective of democratizing
serverless processing
machines or servers. Distributed processing self-service access to the
• Achieve unlimited scale with auto scaling and cloud data platform, which
can be immensely cost-effective since
auto tuning
businesses no longer need to build expensive is responsible for delivering
computers and invest in their maintenance. • Ensure business continuity with high standardized, accurate,
availability and tenant isolation
on-time data to various core
• Build once and run mapping in ETL business units.”
or ELT mode
– Shashank Kulkarni,
• Process data incrementally and efficiently Data Engineer, Nutanix
as it arrives from files or streaming sources or

[Link] 12
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

4 Data Quality

Data quality is a critical component of only used when the second integer is 7, in which • Shared metadata: Typical collaboration
any architecture. It is also unique because case it indicates that customer has opted out of usually involves multiple rounds of back-and-
data quality is not the sole responsibility of data being contacted? forth discussion where business and IT are
engineers. Done properly, data quality bridges not exactly on the same page. What typically
the gap between business and IT. Business In the example above, the business user happens is the business user tries to describe
users understand the rules and context of data understands the rule to implement but does not the requirements and the data engineer tries
but do not usually have the technical ability know how to implement it enterprise-wide. This to explain the results. By sharing metadata, the
to implement data quality rules in a production is where the business and IT must collaborate business user and data engineer are always
process. On the other hand, data engineers to design and implement the data quality rules. aligned because they both see the exact same
can implement rules but do not always have The keys to doing this properly are role-specific thing, but through their role-specific lens.
the business understanding necessary to interfaces, shared metadata and loosely coupled • Loosely coupled architecture: Because
create the rules. architecture: data quality rules are represented as logical
metadata, they should be loosely coupled
For example, we may have a customer identifier • Role-specific interfaces: The business from their sources and deployed via different
formatted as XX999 — two characters followed user should have a thin client interface mechanisms. Data quality is not specific to
by three integers. Creating a data quality rule to designed around ease of creating and testing a database, SaaS application, mainframe or
check for this format is straightforward for the rules. Rule creation should be graphically other sources. Data quality rules must also
data engineer, but rules are often more nuanced driven rather than requiring scripting be easily applied to batch data, real-time data
than simple format checks. What if those first knowledge. Rule testing must then be or exposed via an API to enable upstream,
two characters mean something specific and integrated into the data quality workstream. proactive data quality. Only a metadata-driven
only the letters A, R and Q are allowed but Q is The data engineer’s interface is more technical solution supports the same logic regardless of
in nature with a full palette of both data quality
data source or deployment mechanism.
and data integration functions.
[Link] 13
4 Data Engineering Fundamentals You Need to Be Successful

Part 2

The 4 Fundamentals of Data Engineering (continued)

Data quality ensures your data is fit for purpose. Understanding these four data engineering
To help you achieve this, you need a quality processes — data discovery and lineage, data
solution for multi-cloud and on-premises data ingestion, data processing and data quality —
“Thanks to Informatica
with key features like: will help you accelerate your data engineering [cloud solutions] and
journey. To maximize these key fundamentals, Google Cloud, we now have
• Discovery, search and profiling data engineers should leverage out-of-the-box much more capacity for
• Data enrichment capabilities whenever possible and automate advanced analytics, giving us
data pipelines using AI/ML- based engines to the insights we need to
• Role-based capabilities identify data changes faster.
compete in the fast-changing
• A rich set of transformations
solar power industry.”
• Reusable rules and accelerators
• Exception management – Harish Ramachandraiah,
• AI-driven insights to automate data Director, Engineering & Analytics, Sunrun
quality rules

[Link] 14
4 Data Engineering Fundamentals You Need to Be Successful

Part 3

How To Become a Successful Data Engineer

We have covered the fundamentals of data The bottom line: Your skillset is in demand.
engineering. Now let’s look at what it takes According to The New York Times, U.S.
“The expert in anything was
to become a successful data engineer. From unemployment rates for high-tech jobs range
from slim to nonexistent. On average, each
once a beginner.”
aspiring to early-stage data engineers to data
and analytics leaders who plan to build data tech worker looking for a job is considering
– Anonymous
engineering teams, this chapter is for you! more than two employment offers.6

Data engineering is a rapidly growing What Exactly Does a Data Engineer Do?
Data engineers find trends or inconsistencies
profession. From large public cloud companies Data engineers enable data-driven decision
in data sets and develop algorithms to help
to innovators, data engineers are in high making by acquiring/ingesting, transforming
make raw data more useful to the enterprise.
demand. There are over 220,000 job listings for and publishing data. A data engineer discovers,
Along with technical skills, data engineers
a data engineer in the U.S. on LinkedIn. In fact, designs, builds, operationalizes, secures
also convey data trends, quality issues and
data engineering is the fastest growing tech and monitors data processing systems.
patterns to help the business make meaningful
job, beating data science hands down, and the They do this by focusing on security and
use of the data it collects.
demand has only increased since 2020.5 compliance; scalability and efficiency; reliability
and governance; and flexibility and portability.
The role is very outcome orientated. A data
A data engineer is also responsible for
engineer is a superhero of sorts because you
leveraging, deploying and training pre-existing
can bring all this data to life.7
machine learning models.

5
[Link]
6
The New York Times, OnTech with Shira Ovide, June 14, 2022
7
[Link]

[Link] 15
4 Data Engineering Fundamentals You Need to Be Successful

Part 3

How To Become a Successful Data Engineer (continued)

Some examples of data projects you may be Data Engineer Qualifications: • Cloud computing skills in one or more
involved in include: What You Need cloud service providers (e.g., Amazon
Web Services, Microsoft Azure, Google
Data engineers wear many hats throughout
• Analytical and visualization projects, which Cloud Platform, etc.)
the various phases of the data lifecycle, so
require knowledge of how to share data with • Basic understanding of machine learning
must have a diverse background that goes
data visualization and BI tools such as: algorithms, statistical models and some
beyond education. While a degree in computer
[ Data aggregation science, engineering, applied mathematics, mathematical functions
[ Website monitoring statistics or related IT area is critical, here are • Knowledge of data discovery and
[ Real-time data analytics key technical skills that every data engineer profiling through data cataloging and
should have: data quality tools
[ Event data analysis

• Data science and ML-focused projects, which • Deep understanding of data management
concepts focusing on data cataloging,
require knowledge of how to train ML models “A fundamental reality of the data
continuously with the right set of cleansed data ingestion, data integration (e.g. ETL,
engineer job market is that demand
data, etc. Some examples include: ELT) and data quality
far exceeds supply. There are too
Smart IoT infrastructure • Experience in database management
[
many companies looking for too
(relational/non-relational database
[ Shipping and distribution
management system concepts), data
few data engineers.” 8
demand forecasting
warehouse and data lake concepts
[ Virtual chatbots
• Proficiency in scripting/coding languages
[ Loan prediction such as SQL, R, Python, Java, etc.

8
[Link]

[Link] 16
4 Data Engineering Fundamentals You Need to Be Successful

Part 3

How To Become a Successful Data Engineer (continued)

Intermediate and advanced data engineers Traits of a Successful Data Engineer 1. Curiosity. Data engineers must keep
should have the following skillsets, which can up with the latest trends surrounding
Being a great engineer goes beyond
be learned by shadowing or creating your own technology, tools, datasets and its usage.
technical skills and advanced degrees.
test project by downloading test datasets: Things change fast and you need to be able
Having the right personality is just as
to quickly understand, evaluate and learn
important. A career in data engineering can
• Emerging modern data architecture new tools. You should be eager to learn,
be rewarding and amazing. It can also
frameworks like data fabric, data mesh and grow and always ask, “Why?”
be overwhelming, demanding and stressful.
modern data stack
Here are five key traits of a data engineer 2. Flexibility. There is constant change in the
• API and real-time streaming who is poised for success: data industry. Data engineers should be able
• Data governance concepts such as to go with the flow and be comfortable with
data sharing, data access and data pivoting strategies, changing priorities and
adjusting timelines.
asset analytics “Engineering is easy. It’s the people
• Data compliance and security knowledge problems that are hard.” 3. Problem-solver. Data engineers are
responsible for testing and maintaining
• Operationalizing data processing, data
— Bill Coughran, the data architecture that they design and
observability and DataOps
Former Google Senior Vice looking for ways to improve data processes.
President of Engineering
This requires a mind for creative problem
solving and thinking outside of the box.

[Link] 17
4 Data Engineering Fundamentals You Need to Be Successful

Part 3

How To Become a Successful Data Engineer (continued)

4. Multi-tasker. Not all data engineers come from a computer science or

data science background. Other fields include IT, statistics, math and “Utilizing Informatica
computer engineering. It helps to be well-versed in all facets of data and
[cloud solutions] to integrate
proficient in tools and technologies focused on automation.
the warehouse information
5. Strong communicator. Data engineers are an integral part of the data
from an on-premises operation
team and must present and explain concepts to non-technical and
data store (ODS) into Google
technical stakeholders ranging from peers to executive leadership. The
ability to confidently state your case will help break down silos across the
Big Query, analytic execution
organization and lead to better business decisions. time was driven down to minutes
resulting in a highly, scalable
In addition to having the above traits, being proficient in several (up to 30!) easy-to-use analytics solution.”
technologies and knowing what tool to use when is critical. You must have
– Data Engineer, Large Grocery Retailer
a strong sense of ownership. This is a job where you are literally given a
framework to work with and expected to come up with the program to run. 9

9
[Link]

[Link] 18
4 Data Engineering Fundamentals You Need to Be Successful

Part 4

The Need for an End-to-End Data

Management Platform
Four hundred and sixty-three exabytes Informatica offers a cloud-native end-to-end data management platform with the Intelligent Data
of data will be generated each day by people Management Cloud ™ (IDMC). Its capabilities meet common data processing needs and simplify a data
as of 2025.10 Clearly the need for data engineer’s job tasks through CLAIRE®, its AI-driven engine.
engineers to help transform this tsunami
DATA CONSUMERS
of data is overwhelming.
ETL Developer Data Engineer Citizen Integrator Data Scientist Data Analyst Business Users
Understanding the four fundamentals of data
engineering — data discovery and lineage, data
Intelligent Data Management Cloud™
ingestion, data processing and data quality —
is key to your success. As the architect of DISCOVER &
UNDERSTAND
ACCCESS &
INTEGRATE
CONNECT &
AUTOMATE
CLEANSE
& TRUST
MASTER
& RELATE
GOVERN
& PROTECT
SHARE &
DEMOCRATIZE
the data, you are bringing characters, numbers,
symbols, facts and stats to life that will
ultimately benefit your organization. DATA DATA� API & APP � DATA MDM & 360 GOVERNANCE DATA
CATALOG INTEGRATION INTEGRATION QUALITY APPLICATIONS & PRIVACY MARKETPLACE

As technological advancement makes overall

AI-Powered Metadata Intelligence & Automation
data processing simpler, data requirements
Connectivity
are getting more complex. The success of data Metadata System of Record

engineering must take advantage of modern

DATA SOURCES
technologies to make processes scalable,
+ +
Real-time /
SaaS Apps On-premises
reusable and adaptable. Sources
Mainframe Applications Databases
Sources
IoT Machine Data Logs
Streaming
Sources

To do this well, businesses need a solution for

cloud data warehouses and data lakes across Figure 4: Informatica IDMC is the industry’s most comprehensive solution for multi-cloud,
multiple cloud service providers and on-premises modern data management for data engineering.
to meet practically all data processing needs.

10
[Link] 19
4 Data Engineering Fundamentals You Need to Be Successful

Part 4

The Need for an End-to-End Data

Management Platform (continued)

IDMC can empower you, as a data engineer, to help your organization:

• Bring data together with data integration tools

• Build a foundation for successful data science initiatives
• Forecast trends such as customer retention, churn, fraud and profitability
• Solve problems and inform critical business decisions that
accelerate innovation

As volumes of data continue to pour in from a variety of sources and

convert data into meaningful insights, organizations need a solid data
platform that is scalable and interoperable to meet modern data
engineering needs. Informatica is a leader in data engineering. In fact,
we have helped over 5,000 organizations succeed with IDMC.

Data engineering is always evolving. Because there are always new

technologies to master, you will likely not do the same thing year after year.
The job is complex and uber-critical to business outcomes. It requires time,
dedication and effort. Data engineers are the unsung heroes of the
data world. And that is beyond rewarding.

[Link] 20
4 Data Engineering Fundamentals You Need to Be Successful

About Informatica

At Informatica (NYSE: INFA), we believe data is the soul of business transformation. That’s why we Worldwide Headquarters
help you transform it from simply binary information to extraordinary innovation with our Informatica 2100 Seaport Blvd,
Intelligent Data Management Cloud.™ Powered by AI, it’s the only cloud dedicated to managing data Redwood City, CA 94063, USA
of any type, pattern, complexity or workload across any location — all on a single platform. Whether Phone: 650.385.5000
you’re driving next-gen analytics, delivering perfectly timed customer experiences or ensuring Fax: 650.385.5500
governance and privacy, you can always know your data is accurate, your insights are actionable and Toll-free in the US: 1.800.653.3871
your possibilities are limitless. Informatica. Cloud First. Data Always.™ [Link]
[Link]/company/informatica
[Link]/Informatica

CONTACT US

IN19-1022-04359

© Copyright Informatica LLC 2022. Informatica and the Informatica logo are trademarks or registered trademarks of Informatica LLC in the United
States and other countries. A current list of Informatica trademarks is available on the web at [Link]
21
Other company and product names may be trade names or trademarks of their respective owners. The information in this documentation is
subject to change without notice and provided “AS IS” without warranty of any kind, express or implied.