NSW Data Governance Toolkit Overview
NSW Data Governance Toolkit Overview
Data Management
Where data governance sets the rules of engagement for how data-related decisions are
made within an organisation through the creation of policies and processes, data
management refers to the planning, execution and operation of these policies and
processes.
As illustrated in the following figure, there are 8 core functions of data management which
contribute to the effective governance of data:
Data
Governance
Data
Integration &
Metadata Interoperability
Data
Warehousing Reference
& Business & Master
Intelligence Data
These functions are adapted from the Data Management Association (DAMA) Data
Management Body of Knowledge (DMBOK). While each data management component is
important, not all the functions must be included in the first phase of a governance
program. For example, some programs will focus more on business definitions (Metadata)
initially, while others may emphasise a single view of the customer (Master Data).
What is it?
An enterprise-wide process to manage and improve the quality of an agency’s data is a key
component of effective data management. Data quality management is a continuous
process which involves managing data across the full data lifecycle, from its initial creation
to its destruction. It involves the implementation of data quality standards and procedures to
address and improve the accuracy, completeness, timeliness, relevance, consistency and
reliability of the data.
Why is it important?
The outcome of decisions depends on the quality of the information used to make that
decision. Poor data quality can result in poor decisions and unintended outcomes. High
data quality can support an agency to achieve desired outcomes by ensuring decision-
making is based on accurate and timely information. Data quality methods and procedures
are essential to ensuring accurate data is available to decision-makers in a timely manner.
• Automated: the quality of data is managed through automated tools that can
automatically detect data quality issues and cleanse and enrich the data.
• Lifecycle management: the quality of data is proactively managed across the data
lifecycle, from collection through to disposal.
• Root-cause remediation: problems with data quality are addressed at their root cause
(e.g. fixing the problem at the source).
• Monitored: data quality requirements are enforced through clear monitoring, reporting
and issues management processes.
Define data quality requirements for your agency that are relevant to your
business needs. This will ensure consistency across the organisation and help you
determine which data to keep, which to get rid of, and which to correct.
Measure data quality levels – a data quality assessment tells you how effectively
data is meeting your business needs and stakeholders’ requirements.
Create and implement a strategy for improving data quality – this strategy should
include:
o Industry standards for available data
o Organisational data standards
o Timeliness for data availability
o Data quality metrics
o Goals for data quality metrics
o Data quality rules for specific fields
Relevant Standards:
• NSW Government Standard for Data Quality Reporting – the purpose of this
document is to establish common principles and protocols for reporting on data
quality, so that agencies can create simply data quality statements and users can
easily evaluate whether shared or published data is suitable for re-use.
• Data Quality Reporting tool – this tool is designed to support the NSW Government
Standard for Data Quality Reporting. It guides you through a reporting questionnaire
to generate a Data Quality Statement. All data should be accompanied by a data
quality statement as it helps a user understand how the data can be used.
• ABS Data Quality Framework – NSW has adopted the Australian Bureau of Statistics
(ABS) Data Quality Framework to describe the dimensions (or characteristics) of data
quality. The framework can assist you with the development of statistical collections
to produce high quality outputs.
• ISO 8000 Data Quality – this is the global standard for Data Quality and Enterprise
Master Data. It describes fundamental concepts of information and data quality and
how these concepts apply to quality management processes and quality
management systems.
What is it?
Metadata management means maintaining information about data to ensure both the
users and systems:
• know why data was created and for what purpose; and
By having high quality information that describes the information in data, as well as its
storage and origin, staff can understand what the information is, what they can learn from it
and how to find it quickly. Depending on the data, metadata may include the lineage,
ownership, format, and any rules to be applied to the data.
Why is it important?
Without metadata, it is very difficult for potential data users to know whether a dataset is
available, where it is stored, what the data means, and how accurate it is. A key reason for
duplicated data collection and re-work across government is the fact that repositories of
what data has been collected are either inadequately maintained or do not exist.
Therefore, implementing robust metadata management practices are required to ensure
that data can be located, understood and used not only across the agency, but also by
other agencies and non-government users.
• Valued: the value of having managed metadata, and its role in improving data
quality, is recognised across the organisation.
• Agreed: changes to metadata are agreed and authorised with due consideration of
impacts to other data management functions and business processes.
Establish or improve metadata policies, rules, practices and roles – this can
be done by implementing a metadata adoption plan and implementation process
across the organisation.
Create feedback mechanisms – to ensure that data users can provide input on
the effectiveness of metadata and incorrect or out-of-date metadata.
Relevant standards:
Useful resources:
• Metadata for records and information – NSW State Archives and Records provides
a range of advice on metadata, including:
What is it?
Data security and privacy management includes the policies, processes and procedures
that are in place to ensure that data is kept safe and secure across all stages of the data
lifecycle. Data security and privacy measures are implemented to protect agencies’ critical,
personal or otherwise sensitive data from unauthorised access and use, and ensure that
data can move securely through the organisation. Adherence to privacy legislation, as well
as customer and community privacy concerns, is paramount when considering data
security and privacy management.
Why is it important?
Data can often contain personal, confidential or otherwise sensitive information that can
have serious implications for both the populations the data is about and the organisations
storing it. Good data governance practices across your organisation will ensure it is
protected against misuse, interference, loss, or unauthorised access, modification or
release. Serious physical, emotional or reputational harm to individuals may occur if data
becomes compromised. Data breaches can also result in reputational damage and loss of
public trust, as well as financial and legal ramifications.
• Clear roles: roles and responsibilities for authorising and overseeing safeguarding
processes are clearly defined and access rights are assigned on a need-to-know
basis.
• Classified: the safe handling requirements of data are known because each data
asset is classified according to the NSW Government Information Classification,
Labelling and Handling Guidelines.
• Privacy-by-design: privacy measures are built into the design and architecture of
information systems, business processes and network infrastructure.
• Minimised: data creation and collection processes are designed to ensure that
minimum personal information is collected.
Define and communicate policies on privacy and security with staff – ensure
alignment with relevant legislation, policies and frameworks. For example, all staff
must comply with the Privacy and Personal Information Protection Act 1998
(NSW).
Assess current data security risk and define controls to manage risk – risk
analysis should include examination of unauthorised access; human factors such
as accidental and intentional errors, omissions, destruction, misuse and
disclosure.
Useful resources:
• NSW Cyber Security Policy – outlines the mandatory requirements for sensitive and
classified information.
• Five Safes Framework – provides guidance on how to develop safe data projects and
manage disclosure risks according to five ‘safe’ components and allows data
custodians to place appropriate controls on not just the data itself, but the manner in
which data can be accessed.
• The IPC Public Interest Test – the Public Interest Test is the practical application of
the Government Information (Public Access) Act 2009 (GIPA Act) and it is designed
to help you decide whether or not your data can and should be made open.
Relevant standards:
What is it?
Why is it important?
Fragmented, inconsistent and outdated data in multiple databases does not enable
informed and strategic decision-making. Data warehousing and BI give business units a
way to consolidate and process vast amounts of information and perform more advanced
analytics. With appropriate data warehousing in place, systems have the right data
available to perform more accurate analysis and get more value from BI and analytics
programs. An agency that acts on knowledge gained from BI and analytics can improve
operational efficiency and find better ways to innovate based on insights from data.
• Business goals: the data warehouse serves agency strategic priorities and
informs the selection of BI solutions.
• Start with the end in mind: the business priority drives the creation of the data
warehouse content.
• Once size does not fit all: use the right warehousing and analytics tools and
products for your specific purpose.
What is it?
Reference and master data are data that provide a consistent, reliable record for all critical
business data across the organisation. Master data can be defined as the “golden record”
of critical information that the organisation relies on (e.g. customers, employees, locations,
and products). Reference data is a type of master data that is more likely to change and
that it less critical to the business. Agencies need to define and manage how master and
reference data will be created, integrated, maintained, and used throughout the
organisation. The challenges of this are determining the most accurate data values from
among potentially conflicting data values and attempting to make that data available
wherever needed.
Why is it important?
Definition and management of data assets used across an agency is necessary to meet
strategic objectives, reduce risks associated with data redundancy, and reduce the cost of
data integration. The management of master and reference data allows agencies to correct
data inconsistencies across business units and systems and apply uniform business rules
to enable sharing of data assets across agencies and government more broadly.
• Single view: Master data is recorded and maintained on an accessible and, where
possible, centralised repository to create a single view of the data.
• Controlled: changes to reference and master data are agreed and authorised with
consideration of impacts to other business processes.
Identify and agree on data definitions – this involves determining the most
accurate data values from among potentially conflicting data values and getting
agreement from different parts of the organisation.
Collect the master data into a central database – this database should link to all
participating applications.
Publish reference and master data – ensure its use in all appropriate business
intelligence and analytics reporting across the organisation, at all levels.
Relevant standards:
• ISO 8000-115 Data Quality – Part 115: Master Data – this is the global standard for
Data Quality and Enterprise Master Data. It describes the features and defines the
requirements for standard exchange of Master Data among stakeholders.
What is it?
Agencies must ensure data storage environments are secure, comply with relevant
legislation, and enable information continuity, sharing and re-use. A number of laws and
policies affect how NSW Government agencies can store their data. For example, NSW
Government agencies must comply with the State Records Act 1998 (NSW), which
requires agencies to ensure appropriate records storage, maintenance, security and
archiving. It is important to note that outsourcing storage does not lessen an agency’s
obligation to ensure information is stored appropriately.
Why is it important?
Due to its rapidly increasing volume, how and where agencies store their data is becoming
increasingly important. Storage environments must be able to manage large volumes of
complex data and to provide consistent levels of security, accessibility and functionality. To
ensure the long-term continuity and accessibility of data assets, agencies need to find
appropriate and secure storage environments that comply with legislative and regulatory
requirements.
• Retention and disposal: storage environments ensure data is kept and disposed
of in accordance with business requirements, protective security requirements,
and legislative requirements under the State Records Act, PPIPA and HRIPA.
• Best practice: database standards and best practices are understood and
applied.
• Re-use: storage environments that promote data re-use and integration are
preferenced.
Identify your agency’s storage needs. All agencies’ storage needs will be
different so it’s important to identify and agree on these requirements.
Ensure the storage infrastructure selected is efficient and flexible. This means
that it is easy to search, query, and store the data.
Useful resources:
• State Archives General Retention and Disposal Authorities – outlines the retention and
disposal requirements for different types of information, as well as the requirements for
storing records outside of NSW.
• NSW Government Cloud Guidance and Policy – provides practical steps to move
services to a cloud. This includes information on preparation, contracting and
managing, as well as considerations to note when moving to cloud.
• NSW Cyber Security Policy – agencies must abide by the Policy when procuring cloud
services. The Policy outlines mandatory requirements to appropriately manage cyber
security risks, including the requirement to identify agency ‘crown jewels’.
• NSW Internet of Things Policy – provides practical guidance to help agencies design,
plan and implement IoT solutions, including guidance on storage options for data
What is it?
Data integration and interoperability is the ability of systems, organisations and people to
exchange data between each other so that they can work together seamlessly, either in
the present or in the future. Integration is the ability to consolidate data into consistent
forms, either physical or virtual, and interoperability is the ability for multiple systems to
communicate. Both are dependent on clear, shared expectations for the context and
meaning of data across systems.
Why is it important?
Data integration and interoperability support the use and reuse of government data by
allowing agencies to get data where it is needed, when it is needed, and in the form in
which it is needed. Having integrated and interoperable data can assist agencies to make
better decisions and to provide consistent, coordinated and more timely services by
ensuring they have access to the right data at the right time. Lack of interoperability
between systems means that government agencies often cannot share information
effectively, which contributes to disjointed services, operational inefficiencies and poor
citizen outcomes.
• User-friendly: interfaces are flexible and generic enough to suit multiple uses.
Build future state vision that defines the requirements for creating new services and
systems. Ensure requirements are defined across business functions to ensure the
architecture supports the overall business strategy
Undertake a gap analysis and quantify gaps between current and future state
Planning and design of solutions to bridge gaps. Avoid boiling the ocean and focus
on bridging gaps that are important for your business. Think quick-wins as well as
long-term planning
Useful resources:
What is it?
Data Architecture defines information flows in an organisation, and how they are controlled.
It relates to incoming data and determines how it is captured, stored and integrated into
other platforms across the organisation. It involves understanding business objectives and
the existing data infrastructure and assets, defining data architecture requirements, and
shaping the enterprise data architecture to provide greater benefits to the organisation. The
primary focus of data architecture is to integrate the existing applications and make them
interoperable so data can be used across the organisation.
Why is it important?
Like many large organisations which have been in existence for a long period of time,
government agencies have many legacy systems which use older technology or bespoke
solutions to hold their data. These systems are often difficult to map out and connect with
and require tremendous effort to support change.
• Scalable: the architecture can be applied to various organisational levels and scopes
(i.e. whole-of-government, cross-agencies, agency, line of business, segments,
capability, etc).
Define future state architecture of the organisation, within the context of the
strategic goals of an agency and its operating model.
Perform a gap analysis between the current state and the future state.
Useful Resources:
Relevant standards: