CDMP
Chapter: Metadata Management
By: Budoor Ahmed
Telegram: @Budoor2654
Email: Budoor2654@[Link]
"This course was crafted during the mysterious hours when even the owls were asking,
'Who's up learning Metadata Management now?' "
Outlines
1. Metadata Management Introduction
1.1 Business Drivers
1.2 Goals and Principles
1.3 Essential Concepts
2. Metadata Management Activities
3. Tools
4. Techniques
5. Implementation Guidelines
6. Metadata Management Governance
Metadata Management
Data Quality - DAMA
Metadata Management
Metadata Management Introduction
• Metadata: More than "data about data", encompassing details on processes, rules, and structures.
• Crucial for understanding data, assessing quality, and managing databases and applications.
• Serves as a data catalog, making data findable and understandable.
• Metadata and Risk Management: Documents organizational knowledge, identifies sensitive data, and
supports compliance.
• Facilitates metadata-driven data exchange and integration.
• As data volumes grow, so does the role of metadata in effective data management.
Metadata Management
Business Drivers
• Metadata Management: Essential for effective data management and enhancing data reliability.
• Benefits of Well-Managed Metadata:
1. Increases confidence in data by providing context.
2. Enables strategic use of data, improving value and operational efficiency.
3. Facilitates regulatory compliance and mitigates project risks.
• Outcomes of Effective Metadata Management:
1. Enhanced data quality and governance.
2. Improved communication and streamlined workflows.
3. Reduced inefficiencies through clear data documentation.
• Consequences of Poor Metadata Management:
1. Leads to data redundancy and inconsistency.
2. Causes reduced confidence among data consumers.
• Positive Impact of Effective Metadata Management:
1. Ensures a consistent understanding of data across the organization.
2. Supports efficient development and utilization of data resources.
Metadata Management
Goals and Principles
Goals of Metadata Management:
• Documentation and Management: Ensure understanding and consistent use of data-related business
terminology.
• Integration of Metadata: Aggregate Metadata from various sources to highlight similarities and differences.
• Quality Assurance: Maintain Metadata's quality, consistency, currency, and security.
• Accessibility: Provide standardized access to Metadata for its consumers.
• Technical Standards: Enforce technical Metadata standards to facilitate data exchange.
Metadata Management
Goals and Principles
Guiding Principles for Metadata Solution Implementation:
• Organizational Commitment: Secure support and funding from senior management as part of managing data as an
asset.
• Strategic Planning: Develop a comprehensive Metadata strategy aligned with business priorities, focusing on
creation, maintenance, integration, and access.
• Enterprise Perspective: Adopt an enterprise view for extensibility, using iterative delivery for immediate value.
• Socialization: Promote understanding and use of Metadata through effective communication and engagement.
• Access and Use: Ensure staff members are knowledgeable about accessing and using Metadata.
• Quality and Accountability: Maintain Metadata quality through existing processes and hold owners accountable.
• Standards and Audit: Implement, enforce, and audit Metadata standards for easier integration and utilization.
• Continuous Improvement: Establish feedback mechanisms for updating and correcting Metadata.
Metadata Management
Essential Concepts
1. Metadata vs. Data
• Metadata is a type of data that should be managed accordingly.
• Organizations face challenges in distinguishing between non-Metadata and Metadata, which relates to the
data's level of abstraction.
• For example, phone numbers and call times were classified as Metadata in reports about the US National
Security Administration's surveillance, suggesting that actual data consisted solely of the conversation
content. Yet, these elements are also considered direct data.
• What is considered Metadata by one might be seen as direct data by another, indicating that the classification
can vary based on perspective and usage.
• Organizations should not focus on the philosophical distinctions of Metadata but rather define their
Metadata requirements based on their specific needs, such as creating, understanding, moving, accessing,
and sharing data.
Metadata Management
Essential Concepts
2. Types of Metadata
• Metadata is classified into three main types: business, technical, and operational, which help to understand
the information scope and the functions producing Metadata.
• These categories may cause confusion regarding the classification of Metadata and its intended users. It is
more useful to consider the origin of Metadata rather than its usage, as usage distinctions are not strict and
cross-functional use is common.
• Beyond information technology, Metadata is categorized differently:
• Descriptive Metadata: Includes title, author, subject, etc., facilitating resource identification and retrieval.
• Structural Metadata: Details the relationships within and among resources and their components, such as page and
chapter counts.
• Administrative Metadata: Contains information like version numbers and archive dates, crucial for resource lifecycle
management.
• Understanding these categories aids in defining precise Metadata requirements.
Metadata Management
Essential Concepts
2. Types of Metadata (1. Business Metadata)
• Business Metadata primarily concerns data content and condition, emphasizing data governance aspects.
• It includes non-technical names and definitions for concepts, subject areas, entities, and attributes, alongside attribute data
types, properties, range descriptions, calculations, algorithms, business rules, and definitions of valid domain values.
• Examples of Business Metadata cover:
• Definitions and descriptions of datasets, tables, and columns.
• Business rules, transformation rules, calculations, and derivations.
• Data models.
• Data quality rules and their measurement results.
• Update schedules for data.
• Data provenance and lineage.
• Data standards.
• System of record designations for data elements.
• Valid value constraints.
• Contact information for stakeholders like data owners and stewards.
• Security and privacy levels of data.
• Documented issues with data.
• Notes on data usage.
Metadata Management
Essential Concepts
2. Types of Metadata (2. Technical Metadata)
• Technical Metadata concerns the technical aspects of data, including storage systems and data movement
processes.
• It encompasses:
• Physical database table and column names.
• Properties of columns and database objects.
• Access permissions.
• Data CRUD (create, replace, update, and delete) rules.
• Physical data models, detailing table names, keys, and indexes.
• Documented relationships between data models and physical assets.
• ETL (Extract, Transform, Load) job specifics.
• File format schema definitions.
• Documentation of source-to-target mappings.
• Data lineage documentation, including information on the impact of changes upstream and downstream.
• Names and descriptions of programs and applications.
• Schedules and dependencies for content update cycle jobs.
• Recovery and backup protocols.
• Data access rights, including groups and roles
Metadata Management
Essential Concepts
2. Types of Metadata (3. Operational Metadata)
• Operational Metadata details the processes involved in data handling and access, highlighting the operational
aspects of data management.
•
Operational Metadata Overview:
• Job execution logs for batch programs.
• Extracts history and results.
• Schedule anomalies.
• Audit, balance, and control results.
• Error logs.
• Access patterns, frequency, and execution time of reports/queries.
• Maintenance of patches and versions, including current levels.
• Backup, retention, and disaster recovery details.
• SLA requirements.
• Data volume and usage patterns.
• Archiving and retention policies.
• Purge criteria.
• Data sharing protocols.
• Technical roles and contact information.
Metadata Management
Essential Concepts
3. ISO / IEC 11179 Metadata Registry Standard
• The ISO/IEC 11179 Metadata Registry Standard offers a structured approach for creating a Metadata registry,
facilitating precise data exchange through clearly defined data elements. Its structure includes:
• Part 1: Framework for the Generation and Standardization of Data Elements.
• Part 3: Basic Attributes of Data Elements.
• Part 4: Rules and Guidelines for the Formulation of Data Definitions.
• Part 5: Naming and Identification Principles for Data Elements.
• Part 6: Registration of Data Elements.
• This standard aids in ensuring that data elements are consistently defined, named, and registered, enhancing
data interoperability and clarity across different systems and organizations.
Metadata Management
Essential Concepts
4. Metadata for Unstructured Date
• Metadata plays a crucial role in managing unstructured data, which is not organized in the structured format
of databases but includes documents, media, and other formats.
• Essential Nature of Metadata for Unstructured Data:
• Descriptive Metadata: Catalog information, thesauri keywords.
• Structural Metadata: Tags, field structures, formats.
• Administrative Metadata: Sources, update schedules, access rights, navigation info.
• Bibliographic Metadata: Library catalog entries.
• Record-Keeping Metadata: Retention policies.
• Preservation Metadata: Storage conditions, archival status, conservation rules.
• Unstructured data, like books and magazines, requires Metadata for organization and retrieval, akin to a
library's card catalog.
• The emergence of data lakes and Big Data platforms (e.g., Hadoop) has underscored the importance of
cataloging ingested unstructured data with Metadata (name, format, source, version, date received, etc.) to
facilitate access and management.
Metadata Management
Essential Concepts
5. Sources of Metadata
Sources of Metadata:
• Application Metadata Repositories: Physical tables storing Metadata, often integrated within modeling and BI tools.
• Business Glossary: Documents business concepts, terminology, and relationships, evolving from simple spreadsheets to robust managed systems.
• Business Intelligence (BI) Tools: Generate Metadata relevant to BI design, like report fields, distribution, and user information.
• Configuration Management Tools: Manage IT asset Metadata, detailing relationships and contractual aspects.
• Data Dictionaries: Define data structure and content for databases or applications, guiding data management practices.
• Data Integration Tools: Document data movement and transformations, supporting lineage tracking and impact analysis.
• Database Management and System Catalogs: Describe database content, offering operational Metadata on network and infrastructure status.
• Data Mapping Management Tools: Facilitate the transformation of requirements into mapping specifications, supporting data integration.
• Data Quality Tools: Assess data quality, enabling exchange of quality scores with Metadata repositories.
• Directories and Catalogs: Provide information about systems, sources, and data locations, aiding in data sourcing and issue research.
• Event Messaging Tools: Manage data movement between systems, generating Metadata on logic and statistics.
• Modeling Tools and Repositories: Produce Metadata for application/system design, including entity relationships and data structures.
• Reference Data Repositories: Document business values and descriptions for enumerated data, supporting business glossary associations.
• Service Registries: Store technical information about services in SOA, including definitions, interfaces, and usage scenarios.
• Other Metadata Stores: Include specialized lists and repositories for event registries, code sets, spatial data, and business rules.
Metadata Management
Essential Concepts
6. Types of Metadata Architecture
Metadata Architecture Lifecycle:
• Creation and Sourcing: The initial stage where Metadata is generated or collected from various sources.
• Storage: Metadata is stored in dedicated repositories for future retrieval and use.
• Integration: Combining Metadata from different sources to ensure consistency and accessibility.
• Delivery: Making Metadata available to end-users and systems that require it.
• Usage: The application of Metadata by end-users to enhance data understanding, discovery, and governance.
• Control and Management: Ongoing tasks to ensure Metadata accuracy, relevance, and security.
Metadata Management
Essential Concepts
6.1 Centralized Metadata Architecture
• Centralized Metadata Architecture: A single, central repository holds all Metadata from various sources.
• Advantages:
• Availability: Always accessible, independent of source systems.
• Speed: Fast Metadata retrieval due to centralized storage.
• Quality: Potential for enhanced Metadata quality through customization.
• Limitations:
• Replication: Needs robust processes for timely updates from sources.
• Cost: High maintenance expenses.
• Integration: Possible need for custom solutions for extraction.
• Function:
• Central repository collects Metadata, which is accessed via a user portal for queries, supporting global search
capabilities.
Metadata Management
Essential Concepts
6.1 Centralized Metadata Architecture
Metadata Management
Essential Concepts
6.2 Distributed Metadata Architecture
• Distributed Metadata Architecture: Real-time Metadata retrieval from sources without a central repository.
• Advantages:
• Always updated directly from the source.
• Efficient and simplified query processing.
• Limitations:
• No support for user-defined Metadata.
• Dependent on source systems' availability and quality.
Metadata Management
Essential Concepts
6.3 Hybrid Metadata Architecture
• Hybrid Metadata Architecture: Merges features of both centralized and distributed architectures for Metadata management.
• Approach: Centralized repository for user-added, critical standardized items, and manual sources, with direct Metadata retrieval
from sources.
• Advantages:
• Combines near-real-time source retrieval with user-specific enhancements.
• Reduces manual IT work and custom access to proprietary systems.
• Ensures Metadata is current and aligned with user needs.
• Limitations:
• Dependent on source systems' availability.
• Requires additional processing to integrate source and augmented Metadata.
• Ideal for Organizations:
• With rapidly-changing operational Metadata.
• Needing consistent, uniform Metadata.
• Experiencing significant growth in Metadata volume and sources.
• Less Beneficial:
• For static Metadata environments with minimal growth.
Metadata Management
Essential Concepts
6.4 Bi-Directional Metadata Architecture
• Bi-Directional Metadata Architecture: Allows Metadata to be updated in any part of the system and synchronizes
these changes across the architecture, including the source.
• Functionality:
• Changes in Metadata at any point (source, integration layer, user interface) are coordinated back to the original source via a
centralized repository.
• The repository ensures it has the latest Metadata version and manages updates to the source.
• Challenges:
• Requires systematic tracking and resolution of changes.
• Needs additional interfaces for linking the repository with Metadata sources for updates.
• Process:
• Users submit queries to a Metadata portal.
• The centralized repository initially tries to fulfill requests with available Metadata.
• For detailed queries, it directly queries specific sources, allowing for global searches across collected Metadata.
• Advantages:
• Ensures Metadata consistency and currency across the system.
• Supports detailed user queries by accessing specific source information when needed.
Metadata Management
Essential Concepts
6.4 Bi-Directional Metadata Architecture
Metadata Management
Activities
1. Define Metadata Strategy
• Metadata Strategy Development: Define short- and long-term goals for Metadata management. Draft a
charter, scope, and objectives aligned with governance efforts. Establish a communications plan involving key
stakeholders.
• Stakeholder Interviews: Gather foundational knowledge through discussions with business and technical
stakeholders.
• Assessment of Current State: Evaluate existing Metadata sources and information architecture.
• Future Metadata Architecture: Refine the vision for future Metadata management. Develop a target
architecture considering organizational structure, data governance, technical architecture, and security.
• Integrate findings from interviews and analyses. Document the Metadata strategy and outline a step-by-step
approach to transition to the future state.
Metadata Management
Activities
2. Understand Metadata Requirements
• Metadata Requirements :
• Determine the scope of Metadata from both business and technical perspectives.
• Functionality Requirements:
• Volatility: Frequency of Metadata updates.
• Synchronization: Timing and process for updates in sync with source data changes.
• History: Need for retaining historical versions of Metadata.
• Access Rights: Definition of access permissions and user interface functionalities.
• Structure: Modeling approach for Metadata storage.
• Integration: How to combine Metadata from various sources and rules for doing so.
• Maintenance: Procedures for updating Metadata, including change logs and approval processes.
• Management: Assigning roles and responsibilities for Metadata management.
• Quality: Establishing standards for Metadata quality.
• Security: Ensuring sensitive Metadata is appropriately protected.
Metadata Management
Activities
3. Define Metadata Architecture
• Metadata Architecture Design Principles:
• Capability:
• Extract and update Metadata from diverse sources.
• Support manual Metadata updates and user interactions like searches and requests.
• User Isolation:
• Shield users from complex Metadata sources with a unified access point.
• Provide transparent access to all related Metadata resources.
• Architecture Types:
• Centralized: A single repository for all Metadata.
• Distributed: Metadata remains in source locations with real-time access.
• Hybrid: Combines elements of both centralized and distributed architectures.
Metadata Management
Activities
3. Define Metadata Architecture
• Key Design Steps:
• Create Meta Model:
• Develop a data model (metamodel) for the Metadata repository.
• Apply Metadata Standards:
• Follow internal and external Metadata standards for consistency and interoperability.
• Manage Metadata Stores:
• Implement control activities for Metadata environment management, including job scheduling, backup, security, and quality
assurance.
• Administrative: Monitoring, performance tuning, security management.
• Quality Control: Assurance, frequency of updates, and reporting on missing or aging Metadata.
• Metadata Management: Loading, versioning, user interface management, linking datasets.
• Training: For users and stewards on Metadata management practices and tools.
Metadata Management
Activities
3. Define Metadata Architecture
Metadata Management
Activities
4. Create and Maintain Metadata
• Principles of Metadata Management:
• Accountability: Process owners are responsible for the Metadata quality produced through their processes.
• Standards: Implement, enforce, and audit Metadata standards for integration and usage.
• Improvement: Establish feedback mechanisms for reporting incorrect or outdated Metadata.
• Metadata Integration Process:
• Integration Challenges: Governance is required to address quality and semantics when integrating diverse data sets.
• Scanning Approaches:
• Proprietary Interface: Direct scan and load into the repository in one step.
• Semi-proprietary Interface: Two-step process involving scanning to a data file, then loading to the repository.
• Scanning Outputs:
• Control, reuse, and log files for each scan and load cycle.
• Temporary and backup files stored in a non-persistent staging area for rollback, recovery, and audit.
• Metadata Distribution and Delivery:
• Mechanisms:
• Intranet sites, reports, glossaries, data warehouses, BI tools, software development tools, messaging, web services, and APIs.
• Links Metadata with BI and CRM solutions for synchronized scope and currency.
• Uses files (flat, XML, JSON) or web services for Metadata exchange with external organizations.
Metadata Management
Activities
4. Create and Maintain Metadata
Metadata Management
Activities
5. Query, Report, and Analyze Metadata
• Utilizing Metadata Purpose: Facilitates Business Intelligence, informs business decisions at all levels, and
clarifies business semantics.
• Metadata Repository Interface:
• Business Users: May require an interface focused on simplicity and direct access to relevant Metadata for decision-
making.
• Technical Users & Developers: Need a more detailed and technical interface for in-depth analysis and development
purposes.
• Key Reporting Features:
• Change Impact Analysis: Helps anticipate the effects of changes within the data environment.
• Data Lineage Reports: Provides insight into the origin, movement, and transformation of data, aiding in
troubleshooting and ensuring data integrity.
Metadata Management
Tools
• Metadata Repository: Serves as the central hub for managing, storing, and accessing Metadata.
• Features:
• Integration layer for consolidating Metadata from various sources.
• User interface for manual Metadata updates and queries.
• Capabilities of Management Tools:
• Centralized Management: Facilitates the organization of Metadata in one location for ease of access and control.
• Data Extraction and Entry: Supports manual entry and automated extraction of Metadata via specialized connectors
from diverse sources.
• Metadata Exchange: Enables sharing and synchronization of Metadata with other systems and repositories,
enhancing interoperability.
Metadata Management
Techniques
Data Lineage and Impact Analysis
• Objective: Understand data transformations across systems.
• Challenges: Limited coverage by Metadata systems; requires linking lineage fragments.
Metadata for Big Data Ingest
• Strategy: Tag data upon ingestion into lakes; profile for domains and quality.
• Benefits: Facilitates data management in lakes; identifies sensitive information.
Metadata Management
Metadata Governance
• Define specific requirements for managing the Metadata lifecycle and establish governance processes.
• Define standards, manage Metadata status changes, and handle promotional activities and training.
• Master Catalog: Includes source and target Metadata, update schedules, retention versions, quality statements,
and system statuses.
• Use industry and sector-specific Metadata standards for data exchange and evaluate management technologies
accordingly.
• Tool Integration: Support for XML, JSON, REST for data exchange; custom development may be needed for
integration into Metadata management.
• Measurement Challenges: Difficult to measure Metadata impact without metrics on the cost of lack of Metadata.
• Suggested Metrics:
• Repository completeness vs. ideal coverage.
• Metadata management maturity assessment.
• Steward representation across the enterprise.
• User engagement with the Metadata repository.
• Business glossary activity and Master Data service data compliance.
• Quality of Metadata documentation and repository availability.
Metadata Management
Congratulations, Metadata Management wizards! You've successfully navigated the data realms, and
now you're certified nocturnal data guardians. Until our next moonlit adventure, stay data-driven
and dream in hexadecimal! #GoodNightData