Data Curation
Alexander Kalinowski
Week 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 1 / 33
Outline
1 Introduction
2 Syllabus and Assignments
3 What is data?
4 The Scale of Data
5 Digital Curation
6 Why do we need curation?
7 References
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 1 / 33
Introduction
Introduction
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 2 / 33
Introduction
Me
B.S. Mathematics from GWU
Two years in financial services (Monte Carlo simulation, SQL, risk modelling)
M.S. Data Science from DePaul
Concurrently worked at start-up in Seattle for 2 years
DoubleDown - unsupervised learning, price prediction, fraud detection
More start-up experience back in Philadelphia for 2 years, evolving interest in NLP
Sidecar - rule-based systems, recommender engines, supervised learning
NLP data science at a large U.S. bank for 5̃ years
Wells Fargo - supervised NLP, chatbots, ontology development
Concurrently begun Ph.D. studies at Drexel (intersection of NLP and Semantic Web)
Data science work at a large pharma company for 2+ years
Zoetis - image recognition, streaming video analysis
After 13 years of industry experience, moved to academia in Drexel’s Information Science
department . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 3 / 33
Introduction
You
Your name
Your current program of study
Your perceived programming proficiency (1-5)
Your familiarity with Python (1-5)
Your career goals
Your reason for taking this course
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 4 / 33
Syllabus and Assignments
Syllabus and Assignments
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 5 / 33
Syllabus and Assignments
Course Objectives (I)
Demonstrate an understanding of the basic concepts, terms, theory, tools, technologies,
and practices of digital curation
Define data, big data, and metadata and identify their roles in organizational,
technological, legal, cultural, and business contexts
Describe the variety of data types generated and used by various scholarly disciplines,
sub-disciplines, and research communities and how it has changed scholarly
communication
Describe the data curation lifecycle and identify the activities associated with each stage
and their social, legal, ethical, and policy implications
Delineate roles, responsibilities, and workflows for data curators
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 6 / 33
Syllabus and Assignments
Course Objectives (II)
Explain the importance of planning, proposal development, and policy to successful data
curation
Identify data curation standards and best practices
Engage challenges surrounding sustainability, auditing, risk management, trust, and
costing
Evaluate digital curation tools
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 7 / 33
Syllabus and Assignments
Assignments
Assignment 1: 1st Jupyter Notebook assignment, due Sunday of week 2 10%
Assignment 2: 2nd Jupyter Notebook assignment, due Sunday of week 3 10%
Assignment 3: 3rd Jupyter Notebook assignment, due Sunday of week 4 10%
Assignment 4: Mid-term exam, due Sunday of week 5 20%
Assignment 5: 4th Jupyter Notebook assignment, due Sunday of week 6 10%
Assignment 6: 5th Jupyter Notebook assignment, due Sunday of week 7 10%
Assignment 7: Personal data curation plan paper, due Sunday of week 10 20%
Class Participation: Ongoing 10%
Total 100%
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 8 / 33
What is data?
What is data?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 9 / 33
What is data?
Defining Data
Definition
Data is defined as “any information in binary digital form”.
What does this mean to you?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 10 / 33
What is data?
Digital Objects
Definition
Digital objects can be simple or complex.
A simple digital object is a discrete digital item: text file, sound file, image, associated
metadata.
A complex digital object is a discrete digital object made by combining a number of other
digital objects, for example a Web site.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 11 / 33
What is data?
Databases
Definition
A database is a structured collection of records or data scoted in a computer system.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 12 / 33
What is data?
Is data inherently digital?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 13 / 33
What is data?
Reading: Avoiding a Digital Dark Age
Analog versus digital data
Data encoding, data loss
Physical data versus BITS
How can we preserve data as we move to a digital world?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 14 / 33
What is data?
Cassette
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 15 / 33
What is data?
Magnetic Tape
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 16 / 33
What is data?
Reading: Data’s Shameful Neglect
“Scandalous shortfall” in research data sharing
“Data management should be woven into every course in science, as one of the
foundations of knowledge”
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 17 / 33
The Scale of Data
The Scale of Data
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 18 / 33
The Scale of Data
Internet Minute
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 19 / 33
The Scale of Data
Large Hardon Collider
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 20 / 33
The Scale of Data
Cyberscholarship
Estimates suggest that the LHC experiments collect over 1 petabyte (1 million gigabytes)
of data per day
An estimated 98% of that data is “thrown out” → not enough storage or processing to
capture it all!
Even so, CERN passed the 200 petabyte milestone back in 2017!
Some of the data (5PB) is available as open assess ([Link]
highlighting the importance of data sharing in scientific communities
We will touch on many examples of ‘cyberscholarship’ in this course
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 21 / 33
Digital Curation
Digital Curation
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 22 / 33
Digital Curation
Curation
Definition
Curation is the act of selecting and organizing objects (such as works of art) for presentation,
distribution, or publication.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 23 / 33
Digital Curation
Digital Curation
Definition
Digital curation involves maintaining, preserving and adding value to digital research data
throughout its lifecycle (DCC).
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 24 / 33
Digital Curation
What ISN’T Digital Curation?
Not digital archiving: backup and maintenance of data
Not digital preservation: actions required to maintain access to digital materials
Digital curation is more broad: includes adding value, such as metadata or annotations,
for continued re-use of data
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 25 / 33
Why do we need curation?
Why do we need curation?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 26 / 33
Why do we need curation?
Example: Zoetis VetScan
Work project: use data streaming from Veterinary clinics
Context: cats often have kidney issues, need medication
Question: can we use this data to identify cats for clinical trials?
Data: diagnostic test for kidney health levels
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 27 / 33
Why do we need curation?
VetScan
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 28 / 33
Why do we need curation?
Data Lakes
Definition
A data lake provides a scalable and secure platform that allows enterprises to: ingest any data
from any system at any speed—even if the data comes from on-premises, cloud, or
edge-computing systems; store any type or volume of data in full fidelity; process data in real
time or batch mode; and analyze data using SQL, Python, R, or any other language,
third-party data, or analytics application.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 29 / 33
Why do we need curation?
Data Warehouse
Definition
A data warehouse is a type of data management system that is designed to enable and support
business intelligence (BI) activities, especially analytics. Data warehouses are solely intended
to perform queries and analysis and often contain large amounts of historical data. The data
within a data warehouse is usually derived from a wide range of sources such as application log
files and transaction applications.
A data warehouse centralizes and consolidates large amounts of data from multiple sources.
Its analytical capabilities allow organizations to derive valuable business insights from their
data to improve decision-making. Over time, it builds a historical record that can be invaluable
to data scientists and business analysts. Because of these capabilities, a data warehouse can
be considered an organization’s “single source of truth.”
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 30 / 33
Why do we need curation?
Data Lake Issues
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 31 / 33
Why do we need curation?
VetScan Data Lake
VetScan data lake was a new collection of data with little metadata or data definitions
Stored in Microsoft Azure cloud as Parquet files (big data storage format)
Initial reads of the data: hard to find the right keys to join together different data
Quality: found records that seemed to be from the future (bad timestamps!)
Found inconsistent values for kidney tests (depending on how the Vet set up the device)
Found many other inconsistencies → BUG IN THE VETSCAN CODE!
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 32 / 33
Why do we need curation?
Why Digital Curation
Adds value
Increases trust and quality
Tracks data over the entire lifecycle
Creates fit-for-purpose datasets (such as AI/ML applications)
Allows for sharing and re-use
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 33 / 33
Next Time
Next Time
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 34 / 33
Next Time
Next Lecture
Cyberinfrastructure
Cyberscholarship
Curators
Description and Representation Information
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Alexander Kalinowski INFO 202 Week 1 35 / 33