1
Metadata in Data Management
Understanding the Role and Importance
Eng. Mohamed Elyas Abakar
What is Metadata?
Simple Definition: “it’s Data about data”
Metadata describes the
data itself
Examples (e.g., databases, data
Scope elements, data models), the
concepts the data represents
Metadata includes information (e.g., business processes,
The kind of information that about technical and business application systems, software
can be classified as Metadata processes, data rules and code, technology
is wide-ranging. constraints, and logical and infrastructure), and the
physical data structures. connections (relationships)
between the data and
concepts
2
What is the Importance of
Metadata?
It enables data It contributes to
Metadata helps quality the ability to
an organization assessment and process,
understand its is integral to the maintain,
data, its management of integrate,
systems, and its databases and secure, audit,
workflows other and govern
applications. other data.
3
Clarification:
To understand Metadata’s vital role in data management,
imagine a large library, with hundreds of thousands of
books and magazines, but no card catalog. Without a
card catalog, readers might not even know how to start
looking for a specific book or even a specific topic. The
card catalog not only provides the necessary information
(which books and materials the library owns and where
they are shelved) it also enables patrons to find materials
using different starting points (subject area, author, or
title). Without the catalog, finding a specific book would be
difficult if not impossible. An organization without
Metadata is like a library without a card catalog.
4
Why Metadata is essential to data
management ?
All large organizations
produce and use a lot
Metadata provides the
of data. Across an This information must
primary means of
organization, different be documented or the
Metadata is essential capturing and
individuals will have organization risks
to data management managing
different levels of data losing valuable
as well as data. organizational
knowledge, but no knowledge about
knowledge about
individual will know itself.
data.
everything about the
data.
5
Metadata is necessary
to ensure an
organization can
Metadata management
identify private or
is not only a knowledge
sensitive data and that
management
it can manage the data
challenge; it is also a
lifecycle for its own
risk management
benefit and in order to
necessity.
meet compliance
requirements and
minimize risk exposure.
6
What happens if we don't have reliable
Metadata?
Without reliable Metadata, an organization does not know what
data it has, what the data represents, where it originates, how it
moves through systems, who has access to it, or what it means
for the data to be of high quality.
Without Metadata, an organization cannot manage its data as
an asset.
Without Metadata, an organization may not be able to manage
its data at all.
7
Like other data, Metadata requires
management. As the capacity of
organizations to collect and store data
increases, the role of Metadata in data
management grows in importance. To
be data-driven, an organization must
be Metadata-driven.
8
9
Data cannot be managed without Metadata. In
addition, Metadata itself must be managed. Reliable,
well-managed Metadata helps:
Increase confidence in data by providing context and enabling the measurement of data quality
Increase the value of strategic information (e.g., Master Data) by enabling multiple uses
Improve operational efficiency by identifying redundant data and processes
Prevent the use of out-of-date or incorrect data
Reduce data-oriented research time
Improve communication between data consumers and IT professionals
Create accurate impact analysis thus reducing the risk of project failure
10
Metadata assists in representing information consistently,
streamlining workflow capabilities, and protecting sensitive
information, particularly when regulatory compliance is required.
Organizations get more value out of their data assets if their data
is of high quality. Quality data depends on governance. Because
it explains the data and processes that enable organizations to
function, Metadata is critical to data governance.
11
Metadata must be well managed. Poorly
managed Metadata leads to:
Redundant data and data management processes
Replicated and redundant dictionaries, repositories, and other Metadata storage
Inconsistent definitions of data elements and risks associated with data misuse
Competing and conflicting sources and versions of Metadata which reduce the confidence of data
consumers
Doubt about the reliability of Metadata and data
12
Well-executed
Metadata
management
enables a consistent
understanding of
data resources and
more efficient cross-
organizational
development.
13
What are the goals of Metadata
management?
To document and manage To collect and integrate
organizational knowledge of Metadata from diverse
data-related business sources to ensure people To ensure Metadata quality,
terminology in order to understand similarities and consistency, currency, and
ensure people understand differences between data security
data content and can use from different parts of the
data consistently organization
To provide standard ways to
To establish or enforce the
make Metadata accessible to
use of technical Metadata
Metadata consumers
standards to enable data
(people, systems, and
exchange
processes)
14
What are the guiding principles we should
follow in the implementation of a successful
Metadata solutions?
Strategy:
Develop a Metadata strategy that Socialization:
Organizational commitment:
accounts for how Metadata will be created, Enterprise perspective: Communicate the necessity of Metadata
Secure organizational commitment (senior
maintained, integrated, and accessed. The Take an enterprise perspective to ensure and the purpose of each type of Metadata;
management support andfunding) to
strategy should drive requirements, which future extensibility, but implement through socialization of the value of Metadata will
Metadata management as part of an
should be defined before evaluating, iterative and incremental delivery to bring encourage business use and, as
overall strategy to manage data as an
purchasing, and installing Metadata value. importantly, the contribution of business
enterprise asset.
management products. The Metadata expertise.
strategy must align with business priorities.
Quality:
Improvement:
Recognize that Metadata is often Audit:
Access: Create a feedback mechanism so that
produced through existing processes (data Set, enforce, and audit standards for
Ensure staff members know how to consumers can inform the Metadata
modeling, SDLC,business process Metadata to simplify integration and
access and use Metadata. Management team of Metadata that is
definition) and hold process owners enable use.
incorrect or out-of-date.
accountable for the quality of Metadata.
15
What are the types of Metadata?
Descriptive Metadata:
Business. describes a resource and
enables identification and
retrieval.
Outside of information Structural Metadata:
Metadata is often technology Metadata is describes relationships
Technical.
categorized into three types: described using a different within and among resources
set of categories: and their component parts.
Administrative Metadata:
Operational. used to manage resources
over their lifecycle.
16
1. Business Metadata
Business Metadata focuses
largely on the content and
condition of the data and
includes details related to data
governance. Business
Metadata includes the non-
technical names and definitions
of concepts, subject areas,
entities, and attributes; attribute
data types and other attribute
properties; range descriptions;
calculations
17
Examples of Business Metadata
include:
Definitions and
descriptions of Schedules by
data sets, Data models which data is Data standards
tables, and updated
columns
Business rules,
Data quality Data
transformation
rules and provenance
rules,
measurement and data
calculations,
results lineage
and derivations
18
2. Technical Metadata
Technical Metadata
provides information
about the technical
details of data, the
systems that store
data, and the
processes that move
it within and between
systems.
19
Examples of Technical Metadata
include:
Documented
Data CRUD
Physical relationships
Database (create,
database table between the
object replace, update
and column data models
properties and delete)
names and the
rules
physical assets
Physical data
models,
Column Access including data
properties permissions table names,
keys, and
indexes
20
Operational Metadata
Operational
Metadata
describes
details of the
processing and
accessing of
data.
21
Examples of Operational
Metadata
Patches and
Version
Logs of job maintenance
Schedule
execution for Error Logs plan and
anomalies
batch programs execution,
current patching
level
Reports and
History of Results of audit, query access
extracts and balance, control patterns,
results measurements frequency, and
execution time
22
ISO’s Metadata Registry Standard, ISO/IEC 11179, provides a framework for defining a Metadata
registry. standard is structured in several parts:
Part 1: Framework for the Generation and Standardization of Data Elements
Part 3: Basic Attributes of Data Elements
Part 4: Rules and Guidelines for the Formulation of Data Definitions
Part 5: Naming and Identification Principles for Data Elements
Part 6: Registration of Data Elements
23
Metadata Repositories
Metadata repository refers to
the physical tables in which
the Metadata is stored. Often
these are built into modeling
tools, BI tools, and other
applications. As an
organization matures, it will
want to integrate Metadata
from repositories in these
applications to enable data
consumers to look across the
breadth of information.
24
Business Glossary
The purpose of a
business glossary is to
document and store an
organization’s
business concepts and
terminology,
definitions, and the
relationships between
those terms.
25
The business glossary application is structured to meet the
functional requirements of the three core audiences:
Data Stewards:
Data Steward use the
business glossary to manage
the lifecycle of terms and
Business users: definitions and to enhance Technical users:
enterprise knowledge by
Data analysts, research Technical users use the
associating data assets with
analysts, management, and business glossary to make
glossary terms; for example,
executive staff use the architecture, systems design,
linking terms to business
business glossary to and development decisions,
metrics, reports, data quality
understand terminology and and to conduct impact
analysis, or technology
data. analysis.
components. Data stewards
raise terminology and usage
issues and help resolve
differences across the
organization.
26
The business glossary should capture business
terms attributes such as:
Conflicting
Term name, Name of the definitions that
Algorithms
definition, acronym person identifying need resolution,
supporting
or abbreviation, the term, and date nature of the
definitions
and any synonyms updated problem, action
timeline
Business unit and Categorization or
or application taxonomy
Common
responsible for association for the
misunderstandings
managing the data term (business
in terms
associated with the functional
terminology association)
27
Metadata lifecycle
Metadata creation
and sourcing
Metadata storage
Metadata control
in one or more
and management
repositories
Metadata
Metadata usage
integration
Metadata delivery
28
Centralized Metadata
Architecture
A centralized architecture
consists of a single Metadata
repository that contains copies of
Metadata from the various
sources. Organizations with
limited IT resources, or those
seeking to automate as much as
possible, may choose to avoid
this architecture option.
Organizations seeking a high
degree of consistency within the
common Metadata repository can
benefit from a centralized
architecture.
29
Advantages of a centralized
repository include:
High availability, since it is independent of the source systems
Quick Metadata retrieval, since the repository and the query reside together
Resolved database structures not affected by the proprietary nature of third party
or commercialsystems
Extracted Metadata may be transformed, customized, or enhanced with additional
Metadata that maynot reside in the source system, improving quality
30
Some limitations of the
centralized approach include:
Complex processes are necessary to ensure that changes in source Metadata are
quickly replicated intothe repository
Maintenance of a centralized repository can be costly
Extraction could require custom modules or middleware
Validation and maintenance of customized code can increase the demands on both
internal IT staff andthe software vendors
31
Figure 85 Centralized Metadata Architecture
32
Distributed Metadata
Architecture
A completely distributed
architecture maintains a single
access point. The Metadata
retrieval engine responds to user
requests by retrieving data from
source systems in real time; there is
no persistent repository. In this
architecture, the Metadata
management environment
maintains the necessary source
system catalogs and lookup
information needed to process user
queries and searches effectively. A
common object request broker or
similar middleware protocol
accesses these source systems.
33
Advantages of distributed
Metadata architecture include:
Metadata requests from
proprietary systems are
limited to query processing
Metadata is always as
rather than requiring a Batch processing is reduced,
current and valid as possible
detailed understanding of with no Metadata replication
because it is retrieved from
proprietary data structures, or synchronization processes
its source
therefore minimizing the
implementation and
maintenance effort required
Development of automated
Queries are distributed,
Metadata query processing
possibly improving response
is likely simpler, requiring
and process time
minimal manual intervention
34
Distributed architectures also
have limitations:
No ability to support user-defined or manually inserted Metadata entries since there
is no repository in which to place these additions
Standardization of presenting Metadata from various systems
Query capabilities are directly affected by the availability of the participating
source systems
The quality of Metadata depends solely on the participating source systems
35
36
Hybrid Metadata Architecture
A hybrid architecture combines characteristics of centralized and distributed architectures.
Metadata still moves directly from the source systems into a centralized repository. However,
the repository design only accounts for the user-added Metadata, the critical standardized
items, and the additions from manual sources.
The architecture benefits from the near-real-time retrieval of Metadata from its source and
enhanced Metadata to meet user needs most effectively, when needed. The hybrid approach
lowers the effort for manual IT intervention and custom-coded access functionality to
proprietary systems. The Metadata is as current and valid as possible at the time of use,
based on user priorities and requirements. Hybrid architecture does not improve system
availability.
Many organizations can benefit from a hybrid architecture, including those that have rapidly-
changing operational Metadata, those that need consistent, uniform Metadata, and those
that experience substantial growth in Metadata and Metadata sources. Organizations with
more static Metadata and smaller Metadata growth profiles may not see the maximum
potential from this architecture alternative
37
Bi-Directional Metadata Architecture
Another advanced
architectural approach is bi-
directional Metadata
architecture, which allows
Metadata to change in any
part of the architecture
(source, data integration,
user interface) and then
feedback is coordinated
from the repository (broker)
into its original source.
38
39
Metadata Strategy
A Metadata strategy describes
how an organization intends to
manage its Metadata and how it
will move from current state to
future state practices. A Metadata
strategy should provide a
framework for development
teams to improve Metadata
management. Developing
Metadata requirements will help
clarify the drivers of the strategy
and identify potential obstacles to
enacting it.
40
The strategy includes defining the organization’s future state
enterprise Metadata architecture and the implementation
phases required to meet strategic objectives. Steps include:
Assess existing
Initiate Metadata Develop a
Metadata sources and phased
strategy information implementation
planning. architecture. plan.
Conduct key Develop future
stakeholder Metadata
interviews. architecture.
41
Understand Metadata
Requirements
Maintenance:
Volatility:
History: Structure: Processes and rules
How frequently
Whether historical How Metadata will for updating
Metadata attributes
versions of Metadata be modeled for Metadata (logging
and sets will be
need to be retained storage and referring for
updated
approval)
Access rights:
Integration:
Synchronization: Who can access
The degree of
Timing of updates in Metadata and how
integration of
relation to source they access, along
Metadata from
changes with specific user
different sources;
interface functionality
rules for integration
for access
42
43
Several general principles of Metadata management
describe the means to manage Metadata for quality:
Accountability:
Recognize that Improvement:
Metadata is often Standards: Create a feedback
produced through mechanism so that
existing processes Set, enforce, and
consumers can inform
(data modelling, audit standards for
the Metadata
SDLC, business Metadata to simplify
Management team of
process definition) and integration and enable
metadata that is
hold process owners use.
incorrect or out-of-
accountable for the date.
quality of Metadata.
44