Testing Individual Data Types
Laura K. Wiley, PhD
Welcome to the first coding example of Identify Patient Populations, the third course in the Coursera Clinical Data
Science Specialization created by the University of Colorado Anschutz Medical Campus and supported by our industry
partner Google Cloud.
As was mentioned in Week 1, as we move through the course we will be demonstrating each of the tools and techniques
you can apply for computational phenotyping with the example phenotype of type II diabetes. The first half of this reading
contains important background information on what type II diabetes is and how it is diagnosed and treated. Don’t worry if
you don’t fully understand all of the medical information. As a real clinical data scientist you will find yourself working in
areas that you aren’t as familiar with - this is why it is essential to have clinical expert collaborators or consultants as part
of clinical data science teams.
Background on Diabetes
Type II diabetes is a type of diabetes that is caused by the body no longer recognizing and appropriately responding to
insulin.
Diagnostic Criteria Treatments Laboratory Tests
Type II diabetes is diagnosed when:
A fasting plasma glucose level of 126 mg/dL (7.0 mmol/L) or higher -OR-
A 2-hour plasma glucose level of 200 mg/dL (11.1 mmol/L) or higher during a 75g oral glucose tolerance test
(OGTT) -OR-
A random plasma glucose of 200 mg/dL (11.1 mmol/L) or higher in a patient with symptoms of hyperglycemia -
OR-
A Hemoglobin A1C (HbA1C) of 6.5% or higher.
Programming Examples
Let set up our environment. We need four packages:
tidyverse - group of packages for data wrangling and visualization
magrittr - package for piping data analysis chains
bigrquery - package for connecting to bigquery database
caret - package for statistical analysis
We also set up our connection to the Google BigQuery project to be able to access the MIMIC-III demo data. If you aren’t
familiar with these packages I highly recommend working through the R programming section of “Introduction to Clinical
Data Science”, the first course in the Coursera Clinical Data Science Specialization. We have included a few of the
readings from that course for your reference.
library(tidyverse)
library(magrittr)
library(bigrquery)
library(caret)
con <- DBI::dbConnect(drv = bigquery(),
project = "learnclinicaldatascience")
This next section of the reading is split into two parts:
1. Techniques for calculating algorithm performance with the results of manual record review.
2. Querying different data types and seeing how each different data type performs individually.
Calculating Algorithm Performance Querying and Assessing Individual Data Types
Manual Record Review Results
Although we are limited in sharing the text notes for the patient in the demo dataset, we have access to those notes and
have manually reviewed the records to determine if the patients have a history of type II diabetes. A description of the
manual review protocol used is available in the Introduction to Course Example
([Link]
reading.
The results of the manual record review are stored in the course3_data.diabetes_goldstandard table in the Google
BigQuery learnclinicaldatascience project.
diabetes <- tbl(con, "course3_data.diabetes_goldstandard")
diabetes
SUBJECT_ID DIABETES
<int> <int>
10011 0
10013 0
10026 0
SUBJECT_ID DIABETES
<int> <int>
10036 0
10038 0
10040 0
10044 0
10045 0
10046 0
10056 0
1-10 of 99 rows Previous 1 2 3 4 5 6 ... 10 Next
In this table the DIABETES column is a 1 if the patient has a record of type II diabetes and a 0 if they did not have the
condition.
Of the 100 patients in the demo data set, 99 had notes that could be reviewed. Of those 99 records reviewed, 34 had type
II diabetes.
Creating Training and Testing Populations
Notice that we are using the Gold-Standard approach to phenotyping algorithm development. If we use the entire data set
in our algorithm development then we are likely overfitting our sample and our algorithm won’t perform well on a larger
dataset. To avoid this overfitting, we can separate our gold standard population into training and testing populations. We
use the training population to develop our algorithm, then we check the final performance of that algorithm in the testing
population. Let’s do a roughly 80/20 split. We will put 80 records in the training population and the remaining 19 records
will be our testing population.
training <- diabetes %>%
collect() %>%
sample_n(80)
You may have noticed that I added an extract function in the pipe - collect() . This function tells R to download the
data from the Google BigQuery database into R for further processing. The sample_n() function in dplyr randomly
selects the number of rows you require - in this case 80.
To create our testing population we can just invert this list.
testing <- diabetes %>%
filter(!SUBJECT_ID %in% training_population$SUBJECT_ID)
Because the sample function will produce difference populations everytime you call this function, I have stored these
populations in Google BigQuery in the course3_data.diabetes_training and course3_data.diabetes_testing
tables. You should only use these tables when doing graded class exercises as they will be used to calculate the correct
answers.
Since we are in the algorithm development process, we will load the training population data to use for the rest of this
training.
training <- tbl(con, "course3_data.diabetes_training")
Function to Calculate Performance
As discussed in Data Types for Compuational Phenotyping ([Link]
phenotyping/lecture/kyDmV/data-types-for-computational-phenotyping). There are four primary metrics you should use
to assess algorithm performance:
Sensitivity
Specificity
Positive Predictive Value
Negative Predictive Value
It is also helpful to have a copy of the 2x2 table used to calculate these metrics because it can reveal common
programming errors (like the number of records across all the boxes isn’t the same as the number of records reviewed).
While you can create these 2x2 tables and calculate these performance metrics by hand, it’s a lot easier to use a function
that will return all of these in a single step. The caret package in R has a function called confusionMatrix() that outputs all
of this information. I have written a wrapper function to make it easier to use.
## getStats(df, predicted, reference)
getStats <- function(df, ...){
df %>%
select_(.dots = lazyeval::lazy_dots(...)) %>%
mutate_all(funs(factor(., levels = c(1,0)))) %>%
table() %>%
confusionMatrix()
}
This function accepts a data frame that has at least two columns - one column that has the algorithm you built
(“predicted”) and one column that has the results from the manual record review (“reference”). We’ll see this function in
action in the “Querying and Assessing Individual Data Types” tabs.