OS-Climate Data Commons
Overview
OS-Climate & Global Data Commons
OS-Climate applies a community-based open-source approach to solve data & analytics
challenges required for investment to achieve Paris Climate Accord goals
OPEN SOURCE COMMUNITY
• Governance, licensing, and collaboration structures enabling
stakeholders to share cost, intellectual property, and effort.
• Joint projects for new data, modelling, standards, and
supporting technology
GLOBAL DATA COMMONS
• Curated access to library of public and private sources, for
both transition and physical risk/opportunity
• More accurate corporate historical and forward-looking
climate & ESG metrics as a public good
ANALYTICS TOOLS
• Integrate climate-related risk and opportunity into decisions
by investors, financial institutions, regulators, etc
• Scenario analysis and alignment tools for climate change
risk, physical risk and transition risk
2
A Data Mesh provides faster and broader access to climate data
Data Data Lake and
Data mesh
Warehouse Lakehouse
Single point of access
ETL required ELT required Query federation
to all your data
Cost-effective scaling Cost grows with data Separation of storage and Separation of storage
and elasticity retained compute and compute
Easily access new and No data movement
ETL required ELT required
existing data required
Global security and Limited, by region Limited, by region
Global / hybrid / multi-
compliance cloud data access
Specialized skills required Specialized skills required
ANSI SQL interface (Spark, python, etc) (Spark, python, etc)
SQL-based interface
3
OS-Climate Data Commons Architecture Overview
Moving away from centralized data monoliths by adopting a
distributed data mesh approach
Self-service data infrastructure
Standardized self-service infrastructure and
tooling for creating, maintaining and managing
data products
Decentralized data product ownership
Domain data product owners are responsible for all
capabilities within a given domain, including
discoverability, understandability, quality and
security of the data
Federated governance
Common operating standards around data /
metadata / data lineage management, quality
assurance, security and compliance policies
4
Data Commons: Based on Open Data Hub and Operate First
Upstream code enhanced with operational excellence
Open Data Hub
Community driven upstream meta-project demonstrating AI/ML
platform on Red Hat OpenShift comprised of open source
projects
Operate First ([Link]
Incorporate operational experience into Open Data Hub -
operating software and services in the Open for our community
members
OS-Climate Data Commons
Data science platform based on Open Data Hub and delivered as a cloud
service on Red Hat OpenShift on any public or private cloud provider
Open Data Hub: an open source ML architecture blueprint
GOALS
• Provide an end-to-end AI/ML platform
leveraging Open Source components
• One stop easy operator deployment on
Enterprise Kubernetes
• Provide Tools for each stage of the data
science process and for all AI/ML user
personas including:
○ Development tools for Data
Scientists
○ ELT tools used by Data Engineers
○ Monitoring tools for model and
services used by DevOps
• Act as a “glue” for a rich collection of open
source data science projects, which also have
enterprise offering
6
OS-Climate Data Commons Technical Approach
Building a blueprint for a distributed data platform on top of Open Data Hub
Major capabilities being added to support
the Federated Data Governance Layer:
FEDERATED DATA GOVERNANCE ● Open Table Format for big data analytics,
handling partitioning and time-travel /
rollback (Apache Iceberg)
● Automated data versioning and data
lineage (Pachyderm)
● Metadata management, data discovery,
data quality, observability
(OpenMetadata)
● Fine-grained Role Based Access Control
(RBAC) with row-level and column-level
Non-Relational DBs permissions (Trino)
● Data Protection (IBM Fybrik + OPA)
+ + Streaming Data
Operational RDBMS
7
Data Lake Data Warehouse
Architecture: Data Pipelines
Data Science Platform Roadmap
Layer Role What we have now Roadmap
Data Pipeline Data integration, transformation, Build and manage end-to-end data pipelines Data testing, documentation, and profiling
versioning, automation and overall on Jupyter notebooks provided as a service. (great_expectations)
management. Elyra provides visual pipeline editor and batch
management with notebook and python
scripts, as well as version control with Github.
Train Model training, optimization and validation. Training of ML models via any available
training operator in Kubeflow.
Deploy CI/CD pipelines development and Pipeline automation via Elyra / Kubeflow / Improvements in templating / automation of
management, model, experiment and data Airflow. Data-driven pipelines, data versioning metadata ingestion and management.
versioning. (Pachyderm), data lineage (DBT).
Manage Data and metadata catalogs for data sets / Dataset metadata is managed into a data Integration of metadata for security and
pipelines / models. Security and catalogue (OpenMetadata) and refreshed / compliance (Apache Ranger / Fybrik).
compliance management. versioned automatically.
Serve Make trained model available for inference Kubeflow supports KFServing and Seldon
to tool / application. Core by default. It has not been tested /
documented.
Architecture: Data Management
10
Data Access Management
Layer Role What we have now Roadmap
Object Storage Secure access to data source for ingestion Currently based on S3 with secret-based access Automatic secret retrieval by automated ingestion
controls. pipeline for production pipelines. Container storage
Proprietary data sits on standalone bucket. implementation and review of data ingestion so it is
Secrets are only provided for ingestion pipelines not infrastructure-specific (boto3).
developers.
Data Serving Manage data and schema versioning anage data and schema evolution automatically.
automatically with ACID transactions. Time travel enables reproducible queries that use
exactly the same table snapshot, or lets users easily
examine changes. Version rollback allows users to
quickly correct problems by resetting tables to a
good state.
Distributed SQL Query Engine Centralized data access and analytics with query Integration with GitHub SSO via temporary JTW. Support of complex data types such as GeoTIFF.
federation. Authentication and data access Access management by catalog, schema (source /
controls for all data queries. pipeline), table (data set), column (data elements).
Can filter by row and mask data. integration with
data catalogue (OpenMetadata).
Metadata Platform Data schema & metadata management, lineage Data schema and metadata management, data Data compliance management at metadata level.
at the dataset level, data catalogue browse and catalogue.
search.
Data Security Management Enable, monitor and manage data security NA Centralized management and monitoring of access at
across the platform (Admin GUI with query engine level.
authorization management and audit). Data security management at metadata level.
API Gateway Enforce policies which control security aspects NA API Gateway with distributed data management by
such as the authentication, authorization of data owner.
services acquiring data for external applications /
tools.
Demonstration Scope
Enabling real-time energy consumption and carbon emissions reporting through integration of
Kepler with CO2 emissions statistics
CO2 emission
intensity +
Power
Kepler measurement CO2 footprint at Pod, Container, Service level
Data center power
efficiency
Data Center PUE [Link]
12
Kubernetes Efficient Power Level Exporter (Kepler)
Kepler uses uses eBPF to probe energy related system
stats and exports as Prometheus metrics.
● Red Hat, IBM, Intel are major contributors
● Measures K8s node energy usage thru processor
Running Average Power Limit (RAPL) interfaces
● Current support x86_64, being extended to support
ARM64 and S390 platforms
● Estimates pod energy usage from node usage,
including CPU / GPU / RAM, leveraging ML models
for the estimation
● More accurate than existing dashboards such as
CCF / Scaphandre which report energy
consumption based on CPU time
● Project applied for CNCF Sandbox
13
Next Steps: Find Out More
▸ Open Data Hub community page provides
a Get Started guide at
[Link]
▸ OS-Climate Data Commons Architecture:
[Link]
climate/os_c_data_commons
▸ OS-Climate Data Commons article:
[Link]
climate-data-easy-to-find-use-and-share-
5190a0926407
▸ Kepler GitHub:
[Link]
io/kepler
14