Module 2:

Essentials of Data
warehousing and Data mining
 Data Management
● Data management is the creation and implementation of architectures, policies, and procedures that manage the full data
lifecycle needs of an organization.

Common elements of data management:

● Data preparation is the process of cleaning and transforming raw data to allow for accurate analysis. This critical first step
sometimes gets missed in the rush for reporting and analysis, and organizations find themselves making bad decisions with
bad data.
● Data pipelines are used to automatically transfer data from one system to another.
● Data extract, transform, load (ETL) means transforming data to load in an organization’s data warehouse. ETLs often are
automated processes once they are built and usually require preparation and pipeline work.
● Data catalogs help create a complete picture of the data by managing metadata and also making data easier to find and track.
● Data warehouses provide a clear route to data analysis by consolidating all data sources.
● Data governance helps define policies and procedures for maintaining data security and compliance.
● Data architecture will be the formal structure for managing data flow.
● Data security consists of the processes put in place to protect your data from unauthorized access or corruption.
 Data Governance
Data governance is, first and foremost, a data management function to ensure the quality, integrity, security, and
usability of the data collected by an organization.

Data governance needs to be in place from the time a factoid of data is collected or generated until the point in
time at which that data is destroyed or archived.

Data governance focuses on making the data available to all stakeholders in a form that they can readily access.

In addition, it must be one they can use in a manner that generates the desired business outcomes (insights,
analysis) and conforms to regulatory standards, if relevant.

These regulatory standards are often an intersection of industry (e.g., healthcare), government (e.g., privacy),
and company (e.g., nonpartisan) rules and codes of behavior.

Moreover, data governance needs to ensure that the stakeholders get a high-quality integrated view of all the
data within the enterprise.
 Data Governance

Finally, data governance needs to be in place to ensure that the data is

secure, by which we mean that:

● It is accessed only by permitted users in permitted ways

● It is auditable, meaning all accesses, including changes, are logged
● It is compliant with regulations
 Data Governance
Data governance is a key component of data management—the practice of managing how the data that is being
managed is processed through the organization. Data governance helps answer questions like:

● Who has ownership of the data?

● Who can access what data?
● What security measures are in place to protect data and privacy?
● How much of our data is compliant with new regulations?
● Which data sources are approved to use?
 Data Governance

The purpose of data governance is to enhance trust in the data. Trustworthy

data is necessary to enable users to employ enterprise data to support
decision making, risk assessment, and management using key performance
indicators (KPIs).

The principles of data governance are the same regardless of the size of the
enterprise or the quantity of data.

However, data governance practitioners will make choices with regard to tools
and implementation based on practical considerations driven by the
environment within which they operate.
 Data Governance
Enhancing Trust in Data

Ultimately, the purpose of data governance is to build trust in data. Data governance is valuable to the extent that it
adds to stakeholders’ trust in the data—specifically, in how that data is collected, analyzed, published, or used.

Ensuring trust in data requires that a data governance strategy address three key aspects: discoverability, security,
and accountability .

Discoverability itself requires data governance to make technical metadata, lineage information, and a business
glossary readily available. In addition, business critical data needs to be correct and complete.
 Data Governance

Finally, master data management is necessary to guarantee that data is finely classified and thus ensure
appropriate protection against inadvertent or malicious changes or leakage.

In terms of security, regulatory compliance, management of sensitive data (personally identifiable information,
for example), and data security and exfiltration prevention may all be important depending on the business
domain and the dataset in question.

If discoverability and security are in place, then you can start treating the data itself as a product. At that point,
accountability becomes important, and it is necessary to provide an operating model for ownership and
accountability around boundaries of data domains.
Data Governance
 Data Governance
Classification and Access control:

The human resources information includes several data elements: each employee’s name, their date of hire, past
salary payments, the bank account into which those salary payments were deposited, current salary, etc

. Each of these data elements is protected in different ways, depending on the classification level.

Potential classification levels might be public (things accessible by people not associated with the enterprise),
external (things accessible by partners and vendors with authorized access to the enterprise internal systems),
internal (things accessible by any employee of the organization), and restricted.

For example, information about each employee’s salary payments and which bank account they were deposited
into would be restricted to managers in the payroll processing group only. On the other hand, the restrictions
could be more dynamic. An employee’s current salary might be visible only to their manager, and each manager
might be able to see salary information only for their respective reports. The access control policy would
specify what users can do when they access the data—whether they can create a new record, or read, update, or
delete existing records.
 Data Governance

The governance policy is typically specified by the group that is accountable for the data (here, the human resources
department)—this group is often referred to as the governors. The policy itself might be implemented by the team that
operates the database system or application (here, the information technology department), and so changes such as
adding users to permitted groups are often carried out by the IT team—hence, members of that team are often referred to
as approvers or data stewards.

Some enterprises default to open—for example, when it comes to business data, the domain of authorized users may
involve all knowledge workers in the enterprise. Other enterprises default to closed—business data may be available only
to those with a need to know. Policies such as these are within the purview of the data governance board in the
organization—there is no uniquely correct answer on which approach is best.
 Data Governance

● Data governance is mostly focused on making data accessible, reachable, and indexed for searching across the
relevant constituents, usually the entire organization’s knowledge-worker population.
● Data governance extends data enablement into including a workflow in which data acquisition can take place.
● Users can search for data by context and description, find the relevant data stores, and ask for access, including
the desired use case as justification.
● An approver (data steward) will need to review the ask, determine whether the ask is justified and whether the
data being requested can actually serve the use case, and kick off a process through which the data can be made
accessible.
 Data Governance

● Data enablement goes further than making data accessible and discoverable; it extends into tooling
that allows rapid analysis and processing of the data to answer business-related questions: “how
much is the business spending on this topic,” “can we optimize this supply chain,” and so on.
● The topic is crucial and requires knowledge of how to work with data, as well as what the data
actually means—best addressed by including, from the get-go, metadata that describes the data and
includes value proposition, origin, lineage, and a contact person who curates and owns the data in
question, to allow for further inquiry.
 Data Governance

● Data security, which intersects with both data enablement and data governance,
is normally thought about as a set of mechanics put in place to prevent and
block unauthorized access.
● Data governance relies on data security mechanics to be in place but goes
beyond just prevention of unauthorized access and into policies about the data
itself, its transformation according to data class and the ability to prove that the
policies set to access and transform the data over time are being complied with.
 Benefits of Data Governance

● Make better, more timely decisions.

● Improve cost controls.
● Enhance regulatory compliance.
● Earn greater trust from customers and suppliers.
● Manage risk more easily.
● Allow more personnel access to more data.