BMI‐260 Daniel Rubin 3/31/2019

Outline for Today

• About the course
• Expectations
Biomedical Informatics 260 • Logistics
• Resources
Computational Methods for Biomedical • Interactive teaching
Image Analysis and Interpretation • Lecture 1: Imaging modalities and need for
computerized help in image interpretation
Spring 2019

Objectives
• Understand medical imaging modalities
• Learn about how to get computers to
“understand” images
About the course
• See applications for computerized image
analysis
• Get hands on experience!

About the
course

Daniel Rubin, MD, MS David Paik, PhD

Audience
• Graduate students
• Medical Students
• Medical Residents / Fellows ● Professor of Biomedical Data ● Adjunct Lecturer in Radiology
• Undergraduates Science, Radiology, and Medicine
● Research: Imaging AI (esp weak
● Research: Visualization, image
analysis, computer aided
• Auditors welcome learning), image mining, machine
learning, decision support with
detection, image quantitation,
cancer modeling
images

About the About the

course course

1
BMI‐260 Daniel Rubin 3/31/2019

Teaching Assistants
Guhan Venkataraman Kevin Thomas

Expectations

About the
course

Pre-requisites Readings
• Articles
• What you absolutely need to know – Assigned with each lecture
– Programming ability (CS 106A)
– Links posted on Canvas
– Basic statistics helpful
• Books
– Not required
• What would be really nice to know – Supplement required readings
– Familiarity with Python 3 (initial TA sessions will
– (see Canvas, [Link] redirects here)
do Python review, or look for online tutorials)
– iPython/Jupyter Notebooks

Expectations Expectations

Coursework Final Project

• A substantive programming project that covers
• Assignments (N=3)
element(s) of BOTH image quantitation and image
– Involves programming (Python)
semantics
– OUT on Mondays, DUE on Fridays
– Up to groups of 2
• Can be done in groups, up to 4 students
• Midterm (during class, Mon, May 6) – Project proposal (due Fri 4/26)
• Final project presentations (during Final – Milestone writeup (due Fri 5/17)
Exam slot, Mon, Jun 10, 3:30‐6:30pm in Gates B03) • Final Write‐up (Due Monday 6/10 by 11:59 PM)
• Talking with others acceptable, all work individual • Project Final Presentation (Monday, June 10 from
• Submissions on Canvas (one submission per group) 3:30 PM ‐ 6:30 PM, Gates B03)
Expectations Expectations

2
BMI‐260 Daniel Rubin 3/31/2019

Schedule

• Lectures
– Mon / Wed 1:30‐2:50pm, Gates B03
Logistics
• Section/Office hours
– Fridays 10:30‐11:30am, 3rd floor Conf Rm,
MSOB X‐393, 1265 Welch Rd
– First one this Friday (focusing on setting up
notebooks—please come!)
Logistics

Syllabus
Course Outline
• Overview of imaging modalities Image
• Visualization : From Machine to Screen Processing,
• Image Processing 101: (Filtering, Smoothing., image analysis
Segmentation, Registration, Normalization)
• Feature Extraction (Quantitative and Image Image Feature
Semantic) Semantics Extraction/
• Machine Learning (and deep learning) for Representation
Images
• Decision Support Applications
• Clinical Applications
Rubin: Semantic Features
Paik: Image Processing, Quantitative features
Logistics TAs: Machine Learning / CNN / Q/A Logistics
Guests: Applications

Grading Late Submission Policy

• Grade Breakdown
– 3 Assignments 45% total (15% each) • Problem sets are due at or before 11:59 PM on
– Midterm exam 15% the due date
– Participation (piazza) 10%
• You have 4 free late days total
– Final project 30%
– Total 100% – Not valid for final project assignments
• Class participation: There are many different ways • After that, 10% off your grade per day late (late
to participate, including but not limited to: time within a day used is counted as a full late
– Attending Class day‐that is, it rounds up)
– Attending TA Sections
– Asking/answering questions on Piazza (discussion
forum)
Logistics Logistics

3
BMI‐260 Daniel Rubin 3/31/2019

Resources
• Course Support
– Canvas for files, assignments
– [Link]
Resources – Above link redirects to Canvas site,
[Link]
– Piazza for questions
– [Link]

Resources

[Link]
[Link]

Resources Resources

Interactive Teaching
• Audience participation
– Bring your laptop or smartphone
– Real‐time polls and quizzes
Interactive Teaching – In class demos
• TA sessions
– Assignment Help
– Review of Material

Interactive
Teaching

4
BMI‐260 Daniel Rubin 3/31/2019

Responding to Polls / Quizzes Multiple Choice Questions

• Open‐ended or multiple choice questions
– You send in the CODE indicating your response (with text /imaging
for open‐ended questions)
• Make sure you are connected on WiFi (not cellular)
• Via phone
– IM the CODE to a number provided
– e.g., 37607 (NB, your texting app must support shortcodes)
• Via laptop
– Submit the CODE at [Link]
• SCPD and watching taped lectures
– Answers provided in class, so you can play at home

Interactive Interactive
Teaching Teaching

Let’s try…

Interactive
Teaching

Our goal is to go from images to

understanding…

Biological
Model

5
BMI‐260 Daniel Rubin 3/31/2019

From Images to Understanding

Image
Processing
± Raw Data
Biomedical Informatics 260
Image Image
Semantics Analysis Model
Computational Methods for Image Analysis
Image Feature Introduction to Imaging Modalities
Extraction/
Representation Applications Lecture 1

Outline of Lecture
• What are the medical imaging modalities?
• Why do we need computer help working with
images? What are the medical imaging modalities?
• What are techniques for computer
understanding of image data?
• What are example applications?

Medical imaging includes many A taxonomy of biomedical imaging

Physical Imaging Specific Procedure

modalities… Basis for

Imaging
Modality

• Radiology

• Pathology

• Ophthalmology

• Dermatology

• Microscopy

Imaging Imaging
Modalities Modalities Slide courtesy David Paik

6
BMI‐260 Daniel Rubin 3/31/2019

Image contrast Contrast resolution & spatial

resolution in images
• Medical imaging is the process of converting
tissue characteristics into a visual image • Contrast resolution
• Image contrast: Difference in the image pixel – The ability of the imaging modality to distinguish
values between closely adjacent regions on the between differences in image intensity
image (seen by human or computer) – Differs according to the physical principles
governing image generation
• Spatial resolution
– The ability of the imaging modality to visualize
small objects
– Differs according to the amount of signal
generated by the modality (e.g., photon flux)
Imaging Imaging
Modalities Modalities

Basis for contrast resolution depends Basis for spatial resolution depends on
on the imaging modality the imaging modality
• Radiography, CT • High signal flux  high resolution
– X‐ray attenuation, absorption – Radiography
• Nuclear medicine/molecular imaging – CT
– Uptake of targeted agents; attenuation/absorption – MRI
• Ultrasound • Low signal flux  low resolution
– Sound transmission, reflection – Nuclear medicine/molecular imaging
• MRI – Ultrasound
– Proton relaxation (generates RF signal)
Imaging Imaging
Modalities Modalities

Endogenous and exogenous factors in Radiography

image contrast
• Endogenous differences in tissue
properties
– Xray, CT: Atomic number, density of tissue
– MRI: Differences in T1/T2 relaxation
– PET/NM: none without contrast agent
• Exogenous (pharmaceutical agents)
– Intravenous or oral
– Xray: Iodine, Barium
– CT: Iodine • High energy electromagnetic radiation
– MRI: Gadolinium • Produced in a “cathode‐ray tube”
– PET/NM: Various radionuclide tagged • Pass through tissues or are absorbed
compounds based on the tissue composition
Imaging Imaging
Modalities Modalities

7
BMI‐260 Daniel Rubin 3/31/2019

Contrast resolution in X‐ray imaging A Pelvic “X‐ray”

Bone
• Contrast resolution based on differences in X‐ray
attenuation Gas
– Atomic number (calcium, iodine, lead)
– Tissue density/thickness Muscle
(water)

• Bright on image: Bone, contrast agent, markers,

foreign bodies, very dense tissue
• Dark on image: Air/gas Fat

Imaging Imaging
Modalities Modalities

Computed Tomography Hounsfield units

• Each pixel in CT images is assigned a number
• Row of x‐ray detectors (Hounsfield unit) that is related to the linear
• X‐ray cross‐sections attenuation coefficient (µ) of tissue within
• Tube‐detector apparatus rotates 360º each voxel
• Computer reconstruction of xray attenuation in
patient • Water defined HU = 0
µ µ
• CT Number = 1000 ∙ µ
• Large range of HU in CT images:

Imaging
Modalities Imaging
Modalities 50

Limitation in viewing all the CT numbers Contrast resolution in CT imaging

• Human eye cannot
• As with radiography, contrast resolution based on
appreciate full range of CT differences in X‐ray attenuation
numbers – Atomic number (calcium, iodine, lead)
• Visualized by setting window – Tissue density
width and level • Bright on image: Bone, contrast agent, markers,
foreign bodies
• Dark on image: Air/gas

“Soft tissue window” “Lung window” “Bone window” Imaging

Imaging Modalities
Modalities 51

8
BMI‐260 Daniel Rubin 3/31/2019

Superiority in contrast resolution on CT

Contrast resolution much higher in CT

Cavitary tuberculosis in
Chest X‐Ray CT upper lobe of left lung

Why need
Imaging
Imaging computer
Modalities
Modalities help?

CT studies have lots of slices Ultrasound

Uterus Ovaries

• High frequency (>1 MHz) sound

• Echoes of tissue interfaces
• Image in real time/motion
• Blood flow imaging without
Bladder
contrast agents
• No ionizing radiation
• Any scan plane
Cervix Vagina Rectum

Imaging
Imaging Modalities
Modalities

Contrast resolution in ultrasound

imaging Ultrasound image
• Contrast resolution based on differences in Grayscale reflects Axial image

sound impendence among tissues amplitude of echoes

– Homogeneous tissue transmits sound generated at tissue
– Heterogeneous tissue or gas reflects sound interfaces
Fluid filled structures: Bladder
Bladder
Bladder
• Bright on image: Tissue interfaces, black
stones/calcification, gas, heterogeneous tissue Solid tissue: gray R ovary

Fat: white
• Dark on image: Fluid filled structures,
Calcifications, air: white
homogenous solid organs with “shadow”
Uterus

L ovary

Imaging Imaging
Modalities Modalities

9
BMI‐260 Daniel Rubin 3/31/2019

Magnetic Resonance Imaging (MRI) Contrast resolution in MRI

• Contrast resolution based on differences in T1, T2
• Basis for imaging: radio waves emitted from relaxation and proton density among tissues, as
patient well as the pulse sequence used for imaging
– “T1 weighted” emphasizes T1 contrast
• Superb and varied soft tissue contrast – “T2 weighted” emphasizes T2 contrast
• Any scan plane • Bright on T1 image: Iron deposits, contrast
agents, fat
• Superb depiction of musculoskeletal system and
• Dark on T1 image: Fluid filled structures, iron
bone marrow deposits
• Used to characterize pathology seen on • Bright on T2 image: Fluid filled structures,
ultrasound or CT hemorrhage
• Dark on T2 image: Iron deposits, bone, stones,
flow void
Imaging Imaging
Modalities Modalities

Brain MRI: Multi‐modal imaging important

Magnetic Resonance Imaging (MRI) for diagnosis

T1W T2W

Spine: T1 weighted Spine: T2 weighted T1W +contrast

Imaging Imaging
Modalities Modalities

Nuclear medicine (molecular) imaging

• Injection of radioactive probes that are
metabolically‐active
– Image wide range of molecular events
– Varying disease sensitivity/specificity
– 18FDG glucose: a radio‐labeled probe for glucose uptake
(increased uptake in tumors and inflammation) Pre-treatment
• Imaging with scintillation camera (planar), detector
array (SPECT/PET), optical, fluorescence Post-treatment

Imaging used to monitor response to new drug (Gleevac)

Imaging
64
Modalities Copyright © Daniel L. Rubin 2006

10
BMI‐260 Daniel Rubin 3/31/2019

Medical images show us different

kinds of information about disease Example case
Spatial resolution (anatomic information)

Radiography

• 55 y.o. runner with hip

PET‐CT pain; rule‐out fracture
CT MRI • Plain film:
normal; no fracture
US PET • Can we see anything else
to explain the pain?
Planar NM

Functional information (biological processes) Imaging

Imaging
Modalities Modalities

The abnormality is seen better on MRI… Digital pathology

• Scanner converts whole
slide (or tissue
• MRI: more sensitive to microarray) into digital
marrow changes images
–T1‐weighted image • Huge size
–Dark area=edema • Opportunities for
computational analysis,
disease classification, etc.

Imaging Imaging
Modalities Modalities

Contrast resolution in digital pathology Pathology images: H&E stain

• Contrast resolution based on differences staining
of tissues
• Different kinds of stains
– H&E: Most common; nuclei purple, blood red, cellular
elements pink
– Special stains for specific kinds of proteins
• Multi‐resolution images (40x – 1x) is unique
aspect
• White on the image: non tissue areas
• Color on images: based on tissue stain (nuclei are Low power: Low power: High power:
purple) Tissue regions Image segmentation Cell/nuclear regions
showing different
Imaging tissue regions
Imaging
Modalities
Modalities

11
BMI‐260 Daniel Rubin 3/31/2019

Ophthalmology: Fundus photography Retinal fundus photography

• Photograph of retina Healthy Retina Diabetic Retinopathy
• Full view of retina
• Color images
• Lower resolution than
optical coherence
tomography

Imaging Imaging
Modalities Modalities

Optical Coherence Tomography (OCT)

• A non‐invasive 3D imaging technique
• Standard of care since 1991
• Emit lights into the eye; measure
reflectivity of tissues within a target cube
• Visual rendering the retinal structures
Why do we need computer help working
x z
with images?
y y
x

z
OCT volume

x
z
Imaging
OCT slice
Modalities

Challenge 1: Too many images to

Why do we need computer help? look at (data explosion)…
1. Too many images to look at
2. Variation in image interpretation
3. Image data are unstructured

Why need
computer
2001 Why need
computer 2009
help? help?

12
BMI‐260 Daniel Rubin 3/31/2019

Challenge 2: Variation in Image Radiologist performance is variable

Interpretation
• Radiologists are not perfect
• Missed observations in images
• Misinterpretation of observations in images
• Both cause incorrect diagnoses
• Variation in practice produces suboptimal
quality of care

Barlow-WE, et al, JNCI 96(24):1840-50, 2004

Why need Why need

computer computer
help? help?

Image interpretation is variable Radiology: Errors in human perception…

Why need Why need

computer computer
help? help?

Human ability to discriminate image 50 shades of gray

features

Can you distinguish among all pairs?

Why need
Ans = 30
Why need
computer computer
help? help?

13
BMI‐260 Daniel Rubin 3/31/2019

Errors in interpretation: What’s the Challenge 3: Image data are

abnormality? unstructured
• Structured data
– Data whose meaning is explicit
– Machine‐accessible and interpretable
– Format: attribute‐value pairs, controlled terms
– Clinical data, biological data
• Unstructured data
– Lack explicit meaning; no inherent structure
– Limited machine‐accessibility
Normal CT Abnormal CT – Format: binary objects, narrative text, numbers
– Images and tests (the majority of biomedical data!)
Why need
Agenesis (absence) of right kidney computer
help?

Examples of structured data Two key types of unstructured data

related to images
• Biological data
– A, C, T, G, U (DNA, mRNA)
– Ala, Asx, Lys, Pro, Gly,… (proteins) • Images themselves (sets of pixels)
– (gene expression)
– ATOM PRO 21.2 21.5 17.4 (protein structure) • Free text (e.g., radiology and pathology
• Medical data reports that describe qualitative features and
– K=2.4, Cl=112, Hgb=13.2,… (lab values) diagnoses)
– ICD 112.3, CPT 11707 (diagnostic codes)
– BP 119/80, HR 77, RR 12 (vital signs)
Why need Why need
computer computer
help? help?

Image contents are unstructured Texts related to images are unstructured

Things radiologists say about images: Anatomy,
• Images are just pixels; lack knowledge about
image observations, regions of interest, etc.
their contents
• Computer vision methods access/process
image pixels “There is a hypodense
mass measuring 4.5 x 3.5
cm in the right lobe of the
liver, likely a metastasis.”
Organ = liver
Location = right lobe
Diagnosis = metastasis
Probability = likely
Radiology Image
Why need Why need Radiology Report
computer computer
help? help?

14
BMI‐260 Daniel Rubin 3/31/2019

In Summary

Exploding data…

Challenging features…
What are techniques for computer
understanding of image data?
Variable interpretation…

Unstructured data…
What to do?
Why need
computer
help?

We will learn methods for deriving

structure from unstructured data
Imaging informatics:

• Image (and text) annotation by humans

(“semantic annotation”) to structure human‐
“The application of computer observed image features
science methods to the challenges
of medical imaging” • Image processing (and natural language
(This includes “AI”) processing) techniques to extract features
from images/texts
Tackling the Tackling the
challenges challenges

Pre‐defined image features

Extracting quantitative image features
• Two approaches for deriving structure from Shape:
images:
– Extract pre‐defined (hand‐crafted) features
– Unsupervised feature learning (data‐driven)
Edge:
• N.B.: Image classification using deep learning
is an example of deriving structure from Texture features: scale
images (characterize lesion
interior)
Tackling the directions
challenges Tackling the
challenges

15
BMI‐260 Daniel Rubin 3/31/2019

Machine learning with pre‐defined

image features Unsupervised feature learning
Radiomics: High‐throughput extraction of quantitative
image features with the intent of creating mineable • Raw image pixel data input into a model
databases from radiological images – Image patch analysis
Image data
malignant, nonresponder – Deep learning
malignant, responder
benign
– Word embeddings Text data
undefined
quant. feature #2

pre-malignant

Supervised
learning
Feature extraction
and qualification
quant. feature #1
Radiomics: The process and the challenges, Magnetic Resonance Imaging, 30(9):1234-48, 2012.
Tackling the Tackling the
challenges challenges

Unsupervised feature learning: Deep learning Word embeddings

• High‐level abstractions of image features (hierarchical, non‐linear
transformations)
• Inspired by hierarchical visual processing by the brain
• Higher‐level features (layers) are defined from lower‐level ones,
and represent higher levels of abstraction
Unsupervised
learned quantitative
features
f( )=y
Low level
representation

High level representation

Input Edges, shapes, Recognizable Word embedding provides vector-based
image object parts objects Output representation of text (learned using unsupervised
label methods);
“Lung
mass” e.g., to permit learning a classifier for document x
being classified to label y
Modified after slide by Jeff Dean, Google
Tackling the Tackling the
challenges challenges

Key clinical uses of unsupervised

feature learning
• Disease detection
• Lesion segmentation Current applications
• Diagnosis
What are example applications?
• Treatment selection
You will build such applications too! • Response assessment
• Clinical prediction (of Active research area
treatment response or
future disease)
Applications

16
BMI‐260 Daniel Rubin 3/31/2019

Automated detection of bone Classification of Malignant vs. Benign Tumors with

Semantic and Computational Features
tumors

Yi, 2016
Yu, Tuo, 2013
Applications Applications

A skin lesion diagnostic tool Automated segmentation and classification

Applications Applications

Imaging Modalities

Filtering Visualization

Registration Interpretation

Putting this in the larger context… Segmentation Reporting

Geometric Features Texture Features Semantic Features

Quantitative Features

Machine Learning

Applications

17
BMI‐260 Daniel Rubin 3/31/2019

• Medical imaging is broad

To Summarize… • Many different imaging modalities
• The basis for image contrast varies with
modality

Medical
Imaging?

For computers to understand images:

• We need computer help because: 1. They must recognize and extract image
– Too many images to look at features
– Variation in image interpretation 2. These image features are both
– Image data are unstructured quantitative and qualitative
3. Extracting these features is challenging
and requires specialized techniques
(which you will learn in this class…)
Why need
Tackling the
computer
challenges
help?

From Images to Understanding

Image
Many cool imaging applications can be created Processing
to meet important clinical needs Image
Image Model
Semantics Analysis

Image Feature
Extraction/
Representation Applications

Applications

18
BMI‐260 Daniel Rubin 3/31/2019

Thank you!

Next time:
Image visualization

19