This copy is for personal use only. To order printed copies, contact reprints@rsna.

org
1427

INFORMATICS
Deep Learning: An Update for
Radiologists

Phillip M. Cheng, MD, MS

Emmanuel Montagnon, PhD Deep learning is a class of machine learning methods that has been
Rikiya Yamashita, MD, PhD successful in computer vision. Unlike traditional machine learn-
Ian Pan, MD1 ing methods that require hand-engineered feature extraction from
Alexandre Cadrin-Chênevert, input images, deep learning methods learn the image features by
[Link], MD which to classify data. Convolutional neural networks (CNNs), the
Francisco Perdigón Romero, MSc core of deep learning methods for imaging, are multilayered artifi-
Gabriel Chartrand, PhD cial neural networks with weighted connections between neurons
Samuel Kadoury, PhD that are iteratively adjusted through repeated exposure to training
An Tang, MD, MSc data. These networks have numerous applications in radiology,
particularly in image classification, object detection, semantic
Abbreviations: CNN = convolutional neural
network, GAN = generative adversarial network,
segmentation, and instance segmentation. The authors provide an
R-CNN = regions with CNN features, ROC = update on a recent primer on deep learning for radiologists, and
receiver operating characteristic they review terminology, data requirements, and recent trends in
RadioGraphics 2021; 41:1427–1445 the design of CNNs; illustrate building blocks and architectures
[Link] adapted to computer vision tasks, including generative architec-
tures; and discuss training and validation, performance metrics,
Content Codes:
visualization, and future directions. Familiarity with the key con-
From the Department of Radiology, Keck School
of Medicine of the University of Southern Cali-
cepts described will help radiologists understand advances of deep
fornia, Los Angeles, Calif (P.M.C.); Research learning in medical imaging and facilitate clinical adoption of these
Center (E.M., F.P.R., S.K., A.T.) and Depart- techniques.
ment of Radiology (A.T.), Centre Hospitalier de
l’Université de Montréal, 1058-2117 rue Saint-
Denis, Montréal, QC, Canada H2X 3J4; De-
Online supplemental material is available for this article.
partment of Biomedical Data Science, Stanford Published under a CC BY 4.0 license.
University School of Medicine, Stanford, Calif
(R.Y.); Warren Alpert Medical School, Brown
University, Providence, RI (I.P.); Department of
Medical Imaging, CISSS Lanaudière, Université
Laval, Joliette, Québec, Canada (A.C.C., S.K.);
École Polytechnique, Montréal, Québec, Canada SA-CME LEARNING OBJECTIVES
(F.P.R.); and AFX Medical, Montréal, Québec,
Canada (G.C.). Presented as an education ex- After completing this journal-based SA-CME activity, participants will be able to:
hibit at the 2020 RSNA Annual Meeting. Re-
„ Differentiate among four computer vision tasks using deep learning techniques on radio-
ceived October 25, 2020; revision requested April
14, 2021, and received May 2; accepted May 7.
logic images: classification, detection, semantic segmentation, and instance segmentation.
For this journal-based SA-CME activity, the „ Identify building blocks that constitute components of more complex neural net-
authors I.P. and A. T. have provided disclosures work architectures.
(see end of article); all other authors, the editor,
and the reviewers have disclosed no relevant re- „ Discuss neural network architectures adapted to different computer vision tasks.
lationships. Address correspondence to A.T. See [Link]/learning-center-rg.
(e-mail: [Link]@[Link]).
1Current address: Department of Radiology, Brigham and Women’s Hospital,

Boston, Mass.
1428 September-October 2021 [Link]

requirements for training deep learning mod-

TEACHING POINTS els, architectural building blocks that compose
„ Four key computer vision tasks for which deep learning mod-
modern neural network architectures, the valida-
els have been applied to medical images are classification,
object detection, semantic segmentation, and instance seg-
tion process for testing deep learning systems for
mentation. radiology applications, and future directions in
„ Medical images need labels to be used for supervised learn- the field.
ing, the most common form of machine learning, in which
the goal is to predict labels for new inputs. Depending on the Definitions
task, labels for classification may arise from radiology reports, Four key computer vision tasks for which deep
expert reviews, or clinical or pathologic data.
learning models have been applied to medical im-
„ Classification networks are the simplest deep learning archi-
ages are classification, object detection, semantic
tectures, as their goal is simply to predict a category for an im-
age. However, refinements of these architectures have trans-
segmentation, and instance segmentation (Fig 1).
lated into improvements in other applications as well, as the
basic structures of these networks are often used as building Image Classification
blocks of more complex architectures. Image classification is the task of predicting the
„ Detection architectures build on the architectural innovations class or label of an entire image and can be bi-
of CNNs, often incorporating the backbone of a trained clas- nary (two classes) or multiclass (more than two).
sification network. However, detection architectures must not
An example is the binary classification of normal
only classify objects in an image but also predict the coordi-
nates of bounding boxes that localize the detected objects. versus diseased chest radiographs.
„ Architectures for segmentation tasks such as semantic seg-
mentation and instance segmentation must label every pixel Object Detection
in an image. Object detection refers to the identification and
localization of individual examples of a specific
entity of interest on an image or volume, such as
the detection and localization of liver metastases
Introduction on a CT image. An object detection algorithm
Deep learning is a subfield of artificial intelligence typically specifies the location and spatial extent
that has achieved recent success and popularity for of detected objects with a rectangular box sur-
many complex problems (1,2). The breakthrough rounding the object (bounding box).
performance gains of deep learning systems in
automated image analysis tasks have a variety of Semantic Segmentation
direct applications and implications for radiol- Semantic segmentation assigns each pixel in an
ogy (3). In a previous article, Chartrand et al (4) image to a specific class. For example, each pixel
reviewed the basic concepts underlying deep learn- in the liver could be assigned to parenchyma,
ing. We recommend referring to that article as an tumor, or blood vessel. The output of this task
accessible introduction to the basic concepts. This would be a binary (black and white) image mask
article expands on the topics described in the prior for each class, in which a pixel is “on” if it be-
article, with a deeper discussion of more recent and longs to that class.
advanced topics.
Briefly, deep learning systems for imaging use Instance Segmentation
multilayer neural networks to transform input im- Instance segmentation is the pixel-level detec-
ages into useful outputs. A deep learning system tion and delineation of multiple objects within
learns not only the mappings of image features the same class, such as lung nodules individually
to the outputs but also the image features them- distinguished on a chest radiograph. In contrast
selves. Example outputs include image categories to semantic segmentation, instance segmentation
(for image classification), object locations (for requires an object detection step to separate the
detection), and pixel labels (for segmentation). different objects (instances) of the same class.
For image analysis, the fundamental architecture
of deep learning systems is the convolutional Data
neural network (CNN). A CNN designed for im- Training an effective CNN is dependent on
ages contains convolutional layers that compare labeled data. In classification, the data are images
overlapping rectangular patches of the input to with category labels. In detection, the data are
small learnable weight matrices (termed kernels images and rectangular bounding box coordi-
or filters) that encode features. nates delimiting features of interest. In segmenta-
Neural network architectures have rapidly tion, the data are images and image masks that
evolved in size, complexity, and applications since provide labels for each pixel or voxel.
the breakthrough performances of early CNNs in Preparing medical image data for machine
image classification. In this article, we review data learning tasks is a complex process that has been
RG • Volume 41 Number 5 Cheng et al 1429

Figure 1. Computer vision tasks as depicted on axial contrast-enhanced CT images. (a) Classification aims
to assign a label from a list to a given image (eg, liver metastases). (b) Object detection aims to locate lesions,
structures, or organs (eg, liver metastases are in red squares, the aorta is in a green square, the stomach is in a
blue square, and the spleen is in a yellow square). (c) Semantic segmentation assigns an object category label
to each pixel in the image (eg, all liver metastases are in yellow). (d) Instance segmentation assigns individual
labels to each pixel in the image (eg, individual liver metastases are segmented in red, blue, purple, and yellow).

reviewed in detail (5,6). For deep learning, it is Medical images need labels to be used for
critical to have training images that are repre- supervised learning, the most common form of
sentative of the task to be solved. Images from a machine learning, in which the goal is to predict
single medical center may be insufficient to train labels for new inputs. Depending on the task,
a model for a given task or may be biased because labels for classification may arise from radiology
of the sampled population. Multicenter datasets reports, expert reviews, or clinical or pathologic
help to address these problems but introduce data. Labels for detection and segmentation tasks
challenges related to privacy as well as standard- are more complicated and time-consuming to
ization of image acquisition and labels. create compared with classification datasets. Dis-
With limited data, it is easy for a model to be tributing the labeling task among more human
trained to the point of predicting labels perfectly labelers reduces the labeling burden on individu-
on the training data but poorly on new data; such als but increases overall labeling work and raises
a model is said to overfit the training set (7) or to consistency issues that may require averaged or
exhibit poor generalization. One common way to consensus labels among several labelers. Recent
expand the training dataset to prevent overfitting experiments have found value in crowdsourced
is image augmentation (Fig 2). Simple methods of segmentation labels by nonexpert reviewers
increasing the number of training images include (10,11). For tasks with abundant imaging data,
random translations, rotations, flips, scalings, crops, low-quality labels may be sufficient to train a
and brightness and contrast adjustments. There has network. Weak supervision describes training
also been interest in generative adversarial net- on such low-quality or noisy labels, as may arise
works (GANs) (discussed further in this article) to from natural language processing of radiology
produce fake images that resemble real images (9). reports (12).
1430 September-October 2021 [Link]

Figure 2. Diagrams demonstrate data augmentation. (a) Classic data augmentation consists of applying various transformations
(random translations, rotations, flips, scalings, crops, and brightness and contrast adjustments) to initial CT images and using these
new CT images for training. (b) Synthetic data augmentation uses a generative adversarial network (GAN) to produce additional syn-
thetic images that have a statistical distribution similar to that of the initial dataset. In this example, a CycleGAN is trained to convert
contrast-enhanced CT images to noncontrast images. The trained generator is then used to augment the initial dataset for training
on a task segmenting noncontrast images, as proposed in reference 8.

Since the labeling process is expensive, semisu- ity of a network to learn complex features (Fig 5).
pervised learning methods use unlabeled images However, deeper networks can be more difficult to
to augment the dataset, allowing the network train, and the addition of layers has been observed
to learn more about the underlying structure of to lead to performance degradation and higher
unseen data. The simplest semisupervised method training error (23,24). Further architectural refine-
is pseudo-labeling, whereby a partially trained ments were required to improve model training
model predicts labels (termed pseudo-labels) for the and performance, as detailed further in this article.
unlabeled data, and these pseudo-labeled images
are then incorporated into further training (13). Skip Connections
Innovations in image augmentation and label- Skip connections are shortcut connections from
ing cannot fully replace the need for labeled real one layer to a deeper layer, skipping one or more
image datasets with sufficient variations in subject layers (Fig 6). A skip connection typically adds
or lesion appearance. Despite barriers in sharing or concatenates the output of a shallower layer
medical image data, there have been increasing with the output of a deeper layer. These connec-
examples of public medical image datasets. Some tions were empirically found to improve training
prominent datasets are listed by the Data Science of very deep neural networks, starting with the
Institute at the American College of Radiology residual neural network (ResNet) (23). The in-
(14) and the Cancer Imaging Archive (15). formal intuition behind these connections is that
they allow the skipped layers to fit a residual or
Convolutional Neural Networks error mapping, which may be easier than training
those layers to fit a more complex full mapping.
Toward Deeper Networks Further analysis has shown that skip connections
One of the defining features of deep CNNs is the facilitate training by eliminating large irregulari-
number of hidden layers within the networks (Fig ties in the shape of the loss function, which mea-
3). Shortly after the groundbreaking performance sures the output error of the model (25).
of AlexNet (17) in the 2012 ImageNet Challenge,
many networks have been designed to improve Bottleneck Blocks
its performance, with a trend toward larger and Bottlenecks in neural networks improve com-
deeper neural networks (Fig 4). The increase in putational efficiency by reducing the number of
layers has been postulated to increase the capac- feature maps (Fig 7). A feature map or channel
RG • Volume 41 Number 5 Cheng et al 1431

Figure 3. Fully convolutional networks typically consist of a stack of layers performing successive convolutions on an input (eg, an
image), as depicted in this diagram. The first input layer on the left corresponds to the original image, with individual signal intensities
for each pixel. The successive convolutional layers color coded in orange allow the extraction of features to compute intermediate
representations. The changes in box size indicate the evolution of the dimension of the feature maps after successive convolutions
and pooling operations. The prediction layer color coded in green predicts the class of each pixel. In this example, the presented
architecture allows further segmentation and detection of lesions and organs (16).

in a CNN is the output of a convolution kernel to efficiently increase the representational power
applied to either an input image or to the set of of the network, with prominent examples again
feature maps produced by the previous neural seen in Inception modules (Fig E1) (27) and the
network layer. The number of output feature ResNeXt architecture (23).
maps from a layer is therefore the number of
convolution kernels in the layer. Bottlenecks are Wider Networks
implemented by a set of 1 3 1 convolution ker- Owing to diminishing returns in neural network
nels, which preserve the spatial dimensions of the performance with increasing depth, there has also
previous layer but can change the number of fea- been work on scaling the width of the networks,
ture maps (dependent on the number of convo- referring to the number of feature maps or
lution kernels). Reducing the number of feature channels per convolutional layer. Wide residual
maps reduces the computational complexity of networks (Wide ResNets) in some cases can
subsequent convolution operations and effectively be trained more easily and perform better than
compresses the input feature maps into a more deeper conventional ResNets (28) although at
compact representation. The number of feature the expense of increased number of parameters
maps can be subsequently augmented by a 1 3 and memory requirements. More recently, the
1 convolution layer with more output channels EfficientNet family of models scales depth, width,
than input channels. This architecture was used and resolution of networks in a balanced manner
effectively in Inception modules (27), as well as to provide an effective trade-off between size and
in the building blocks of ResNets (23). accuracy (29).

Multibranch Convolutions Ensembles of Networks

Multibranch convolutional architectures use Combining the results of an ensemble of inde-
convolutional operations in parallel in place of pendently trained neural networks can improve
a single convolution (Fig 8). Each branch, for performance (Fig 9). Ensembles have produced
instance, can process information at a different winning results in ImageNet image classification
spatial scale; the outputs of the branches are then competitions (30), as well as in radiology tasks
aggregated by concatenation or summation. Such such as pediatric bone age prediction and pneu-
multibranch architectures (Fig E1) are postulated monia detection (31,32). Recent experimental
1432 September-October 2021 [Link]

Figure 4. Evolution of deep neural networks toward deeper architectures. The increase in layers may increase the capacity of a
network to learn complex features. Representative models are shown here: AlexNet (a) (17), VGG16 (b) (18), and VGG19 (c) (18).
The numbers below the convolution layers color coded in orange indicate the two-dimensional (2D) kernel size and the number
of channels. The maximum (Max) pooling operations color coded in blue consist in extracting the maximum value in a kernel to
preserve information while reducing computation requirements. The changes in box size indicate the evolution of dimensions of the
feature maps after successive convolutions and pooling operations. The fully connected layers color coded in pink allow reasoning
about the entire image.

work with neural network ensembles suggests that encoded in the channel dimension. The final lay-
independently trained networks effectively sample ers distill the encoded semantic information into
from different local optima in the solution space a limited number of task-specific classes.
and improve accuracy through functional diver-
sity (33). Detection Architectures
Detection architectures build on the architectural
Architectures Adapted to Tasks innovations of CNNs, often incorporating the
backbone of a trained classification network. How-
Classification Architectures ever, detection architectures must not only classify
Classification networks are the simplest deep learn- objects in an image but also predict the coordi-
ing architectures, as their goal is simply to predict nates of bounding boxes that localize the detected
a category for an image. However, refinements of objects. The most common detection architectures
these architectures have translated into improve- can be organized into two categories on the basis
ments in other applications as well, as the basic of the number of stages in the detector (Fig 11).
structures of these networks are often used as build-
ing blocks of more complex architectures (Fig 10). Two-Stage Detection.—In two-stage detectors,
The basic CNN building blocks described pre- the first stage is used to propose a sparse set of
viously are combined to create the architecture candidate regions for objects in the image, and
of a backbone encoding CNN network. This base the second stage classifies the proposals. Regions
network progressively downsamples the input with CNN features (R-CNN) (37), Fast R-CNN
image in the spatial dimensions while translating (38), and Faster R-CNN (35) were a pioneer-
the spatial information into semantic information ing series of detectors that used this two-stage
RG • Volume 41 Number 5 Cheng et al 1433

Figure 5. Graphs show the evolution of performance for three

computer vision tasks (object detection, classification, and seg-
mentation). The highest performance per model and year are
shown in red. (a) Graph shows the average precision (AP) for ob-
ject detection task on the Common Objects in Context (COCO)
dataset (19). (b) Graph shows the top-1 accuracy (accuracy of
predicted class with the highest probability) for classification
task on the ImageNet dataset (20). (c) Graph shows the mean
intersection over union (mIOU) for segmentation task on the
PASCAL VOC 2012 dataset (21). The data were extracted from
reference 22. The mIOU depicts the mean overlap of the pre-
dicted segmentations with ground truths. CFNet = cascade and
fused network, CSP = cross stage partial network, D-RFCN = de-
formable region-based fully convolutional networks, DetectoRS =
detecting objects with recursive feature pyramid and switchable
atrous convolution, FCN = fully convolutional network, FPN =
feature pyramid networks, HiRes = high resolution, NAS = neural
architecture search, PSPNet = pyramid scene parsing network,
SNIP = scale normalization for image pyramids, TDM = top-
down modulation, VGG = Visual Geometry Group.

design. Successive architectures within the series feature hierarchy of a CNN could be leveraged to
were characterized by progressive optimizations, predict objects at different scales (40).
including sharing of computations between the A limiting factor for accuracy in early single-
first and second stages. stage architectures was the large imbalance
between true and false positives among the large
Single-Stage Detection.—Single-stage detec- number of candidate object locations. To ad-
tors directly provide classifications and bound- dress this problem, the RetinaNet architecture
ing boxes in a single CNN. These networks introduced a new focal loss function that helped
have the advantage of high efficiency but until focus training on difficult misclassified train-
recently have been less accurate than two-stage ing examples (41). RetinaNets were the basis of
approaches. A series of networks called You Only several top-ranking solutions in the Radiological
Look Once introduced the approach of predict- Society of North America pneumonia detection
ing a fixed number of bounding boxes regularly challenge (32).
distributed over an image and classifying the
presence of objects within the boxes (36,39). The Feature Pyramid Networks.—Feature pyramid
Single-Shot Detector network was one of the first networks (FPNs), proposed by Lin et al (42),
to demonstrate that the pyramidal shape of the are a cornerstone of modern object detection
1434 September-October 2021 [Link]

Figure 6. Skip connections are shortcut

connections from one layer to a deeper
layer, skipping one or more layers. (a) Di-
agram shows the standard neural network
architecture, with successive connections
between layers. (b) Diagram shows the
skip connection by element-wise addition
as used in the ResNet architecture, color
coded in red. (c) Diagram shows the skip
connection by channel-wise concatena-
tion as used in DenseNet architecture,
color coded in red. (d) Artistic rendering
of the loss function in the case of direct
connections, as presented in a. (e) Artis-
tic rendering of the loss function in the
case of skip connections, as shown in b
and c. Skip connections tend to induce
smoother loss landscapes compared with
direct connections, thus facilitating con-
vergence (ie, iteratively converging to-
ward the minimum of the loss function)
during training (25).

Figure 7. Bottleneck blocks.

(a) Diagram shows a convolution
layer color-coded in orange that
indicates a 3 3 3 kernel size with
256 channels. (b) Diagram shows
a bottleneck block that takes ad-
vantage of convolution layers
color coded in green to indicate a
1 3 1 kernel size to decrease the
dimension of channels to 64 and
reduce the computation burden.
(c) Graph shows that the number
of channels (color coded in blue)
is preserved without a bottleneck
block and reduced with bottle-
neck blocks (color coded in red).
(d) Example table shows the cal-
culations for the two scenarios in
a and b (26).
RG • Volume 41 Number 5 Cheng et al 1435

Figure 8. Multibranch convolutions diagrams. (a) The input is processed in parallel by several layers,
and the output consists in the concatenation of these layers. (b) The Inception module diagram exhibits
four parallel branches (27).

Figure 9. Ensemble networks diagram. A set of trained models, with identical or different architectures, is used to generate multiple
predictions, which are then processed by fully connected layers to provide a prediction. This ensemble architecture can be used for
various computer vision tasks (eg, detection, classification, and segmentation) (30).

Figure 10. Architecture for classification diagram. A deep convolutional network for image classification can combine several con-
volution, maximum pooling, and average pooling layers. A convolution layer comprises image filters that detect features relevant to
the task at hand. Maximum pooling operations (MaxPool) downsample by sliding a small window (eg, 2 3 2 pixels) across the image
and taking the maximum value within the window. Average pooling (AvgPool) is analogous to maximum pooling, except the aver-
age value is used. Dropout randomly turns off neurons (ie, weights) within a layer to prevent overfitting. Fully connected layers are
typically used at the end of the CNN to map the feature vector to the predicted classes. Throughout the architecture, feature maps
from different branches are joined together (concatenation), and outputs from different layers are summed (residual connections).
This increases the representational power of the CNN and stabilizes the training process (34).
1436 September-October 2021 [Link]

Figure 11. Architectures for object detection diagrams. (a) Two-stage detection involves a region proposal network (RPN), which
generates a set of region of interest (ROI) proposals based on possible objectness (step 1) before further classification (step 2) (35).
(b) Single-stage detection combines a class probability mask with regions extracted from the initial image (36).

approaches in both single-stage and two-stage de- must label every pixel in an image. Using CNNs
tectors (Fig 12). CNN architectures progressively efficiently for segmentation requires solving an
increase the number of feature maps throughout upsampling problem, in which low-resolution se-
the depth of the network; for computational ef- mantically rich maps produced by convolutional
ficiency, the input image must be downsampled and pooling layers must be converted to high-
to accommodate more feature maps. Thus, there resolution segmentation masks.
is a trade-off between the spatial resolution of
the feature maps and the semantic richness of Upsampling Techniques.—Two important opera-
their contents. This is especially relevant in object tions that perform upsampling are unpooling
detection, as the decreased spatial resolution of and transpose convolution (Fig 13). Unpool-
deeper feature maps leads to difficulty in identify- ing involves recording the locations of maxima
ing small objects. FPNs were developed as a solu- in each pooling operation and later using these
tion to this problem and comprise two parts: a locations to convert a low-resolution feature map
bottom-up pathway, which is simply the conven- into a sparse higher-resolution representation.
tional CNN backbone, and a top-down pathway, Transpose convolution is an upsampling opera-
which progressively upsamples the deeper seman- tion using kernels with learnable weights. In
tically rich feature maps to a higher spatial reso- contrast to conventional convolution, which sums
lution (43). Importantly, shallower feature maps the products of kernel elements with input pixel
are processed by a 1 3 1 convolutional layer and values, transpose convolution uses pixel values of
added to the output of each level of the top-down the input as weights for copies of the kernel to be
pathway to provide valuable spatial information added to the higher-resolution output.
for object detection. These more powerful feature
maps are then provided as input to the final clas- Encoder-Decoder.—The fully convolutional net-
sification and bounding box heads. work (16) pioneered an encoder-decoder design
for segmentation. An encoder network uses a
Segmentation Architectures series of downsampling convolutional layers from
Architectures for segmentation tasks such as se- a classification model to output a low-resolution
mantic segmentation and instance segmentation spatial map instead of classification scores. A
RG • Volume 41 Number 5 Cheng et al 1437

Figure 12. Feature pyramid network diagram. Bottom-up pathway (left) consists of consecutive convolutions produc-
ing a pyramidal hierarchy of feature maps at several scales. The coarsest feature map, which encodes the semantically
strongest features, is then upsampled along the top-down pathway (right). Lateral connections (horizontal arrows)
merge localization-rich information from bottom-up feature maps with semantic-rich information from the top-down
feature maps. (Adapted and reprinted, under a CC BY 4.0 license, from reference 33.)

Figure 13. Diagrams of upsampling techniques. (a) Unpooling saves locations of maxima in pooling operations, using them later
to upsample low-resolution feature maps. (b) An example of transposed convolution to upsample initial dimensions of a 2 3 2 input
image to a 3 3 3 output image. First, each element of the input separately multiplies the kernel. Products are then summed, taking
into account initial locations in the input, leading to a final 3 3 3 output (44).
1438 September-October 2021 [Link]

Figure 14. Diagrams show segmentation architectures: U-Net for semantic segmentation and Mask R-CNN for instance segmenta-
tion. (a) In the U-Net architecture, an input image first follows a succession of convolution and pooling operations to downsample
the image and produce feature maps that represent an abstract representation. Then, features maps are upsampled and concat-
enated along the expanding path through skip connections to provide a segmentation map with the same spatial dimensions as the
input image. The U-Net architecture, dedicated for segmentation, may be as deep as required by the data (46). (b) The Mask R-CNN
architecture contains two parts: the first part includes a region proposal network applied to feature maps (red and yellow map beside
convolution layer) to identify multiple regions of interest, followed by a second part to generate three outputs: bounding box for
detection, classification for the category of object (eg, metastasis, hemangioma, or cyst), and instance segmentation of individual
objects (47).

subsequent decoder network upsamples these An alternative method for evaluating features
maps by using transpose convolutions to produce at several scales is to use dilated (also known
per-pixel labeled outputs. SegNet uses both trans- as atrous) convolutions (Fig 15) (49). Dilated
pose convolution and unpooling in the decoder to convolutions are convolution operations with
upsample low-resolution encoder maps (45). expanded kernels that contain spaces between
adjacent kernel elements. These kernels allow
U-Net for Semantic Segmentation.—The U-Net modeling of larger scale dependencies among
is a popular architecture originally developed for pixels without losing resolution. Dilated convolu-
segmenting microscopy images (46) but which tions are a central component of the DeepLab
continues to be widely used both within and out- family of segmentation architectures (50).
side the medical domain. U-Net has a symmetric
U-shaped architecture in which a descending Mask R-CNN for Instance Segmentation.—Com-
encoder portion downsamples the image and pared with semantic segmentation, there has
produces increasingly abstract representations been less work on instance segmentation owing
and a subsequent ascending decoder portion to fewer use cases and increased complexity of
uses transpose convolutions to upsample these the instance segmentation problem. Instance
representations to the original dimensions of the segmentation can be considered as a problem
image (Fig 14). A key component of the U-Net of simultaneous object detection and semantic
design is the use of horizontal skip connections segmentation. The Mask R-CNN architecture
that facilitate the upsampling process by copying is a prototypical two-stage network for instance
features from encoder stages directly to resolu- segmentation that extends previous work on two-
tion-matched decoder stages. stage detection models (47). As in the detection
RG • Volume 41 Number 5 Cheng et al 1439

Figure 15. Dilated convolutions as depicted on axial contrast-enhanced CT images. (a) One-dilated convolution is equivalent to a
standard convolution; that is the receptive field (yellow tiles) has the same size as the kernel (red dots). (b, c) Two-dilated (b) and
four-dilated (c) convolutions increase the receptive field and thus introduce more contextual information in the process while main-
taining a constant output shape (48).

models, the first stage proposes candidate regions Training and Validation
of interest, while the second stage predicts
bounding boxes and object classifications. Mask Training
R-CNN adds a branch to the second stage that Training a neural network involves fitting the
predicts a binary mask for the region of inter- model weights to a training dataset to achieve
est for each object category by using a convolu- good performance on a given task, such as clas-
tional architecture based on a fully convolutional sification or detection. Several factors affect the
network. The detection and mask components of training performance, speed of convergence
the model are trained jointly to produce instance (finding a solution), and whether the model will
segmentation masks (Fig 14). perform well on new data.

Generative Architectures Model Selection.—Given the wide variety of neural

GANs (8) have rapidly evolved with a broad network designs, selecting a suitable architecture
range of computer vision applications. Typi- for a task may be an iterative process. Design
cally, a GAN consists of two distinct networks: choices are often based on one’s intuition about
a generator, which aims to learn to create “fake” the task. Experimenting with different loss func-
images that appear to fit into the distribution of tions and probing intermediate results within the
training samples, and a discriminator, dedicated network can also provide useful information about
to distinguishing samples from the training set how to use specific layers or network configura-
(real) or from the generator (fake). In the origi- tions. Model exploration may start with a simple
nal vanilla GAN (Fig 16), random noise is trans- architecture known to work on a similar task. Once
formed by the generator and submitted to the convergence is observed on the first training itera-
discriminator. Training alternates between the tions, the capacity of the network, represented by
discriminator and generator, with both networks the number of trainable parameters, is increased
improving to a point where ideally the generator until overfitting is observed on a validation dataset.
produces realistic images. Conditional GANs,
or cGANs (51), use extra input information (eg, Hyperparameters.—While millions of parameters
labels or data from other modalities) to force are automatically optimized during the training
structure on how the generator produces images. phase, some of them called hyperparameters are
CycleGANs (52) use images as input to trans- set manually, such as the number of layers and
late them from one domain to another, such as the learning rate. Using systematic grid search
T2-weighted MR images into T1-weighted MR or random search for hyperparameters may be a
images (53). good strategy to find suitable values (59).
In medical imaging, GANs have been applied Among the hyperparameters for training, the
to classic tasks such as detection, classification learning rate most directly affects the training
(54), and segmentation (55), as well as to image convergence speed by specifying how much the
reconstruction (56), synthesis (57), and registra- model weights are updated with respect to the loss
tion (58). GANs have also been used for data gradient at each training step. A small learning rate
augmentation to increase the training dataset can result in slow training and possibly overfitting
size (8). (60), while a large learning rate causes the training
1440 September-October 2021 [Link]

Figure 16. Standard GAN architecture diagram. A first network, known as the generator (G), aims to transform a random input into
a realistic image to fool a second network, known as the discriminator (D). During training, the generator learns from the response
of the discriminator (8).

process to diverge. A cyclical learning rate which Transfer Learning.—Owing to barriers in shar-
rises and falls during training may reduce training ing medical image data, sufficient labeled training
time and improve accuracy (61), likely by overcom- images are often not readily available for a given
ing local minima in the loss landscape. task. Transfer learning is a process by which models
pretrained on larger generic image datasets such
Regularization.—Regularization refers to strate- as ImageNet (20) can be fine-tuned for tasks on
gies designed to prevent overfitting during train- smaller datasets. Transfer learning can mitigate
ing, sometimes at the expense of increased train- data requirements for model convergence and has
ing error. Regularization can take many forms. become routinely used in medical imaging research.
Early stopping involves monitoring validation
error during training and stopping training when Validation
validation error starts increasing (2). Weight de- The performance of a deep learning model on
cay penalizes extreme values of network weights, training data does not predict its ability to general-
considered a symptom of overfitting. Dropout ize to unseen data. A standard strategy to improve
directly affects the capacity of the model by and predict the generalizability of a model is to
randomly removing a subset of neurons from the split a dataset randomly into three subsets: training,
network at each training step, forcing the network validation, and test sets. The split should be disjoint,
to employ different computation pathways to such that the same patient is not represented in
reach the same output, and making the network more than one set. It may also be helpful to balance
as a whole more robust (62). the sets by stratifying the split according to variables
such as sex, age, or label prevalence.
Data Sampling.—Data sampling can also affect The model’s weights are optimized by using
convergence speed and performance. A common the training set data. The validation set is used
challenge is imbalance of classes in a dataset, for in- to tune model hyperparameters, with periodic
stance when healthy cases significantly outnumber evaluation of model performance on validation
diseased cases. Randomly sampling such a dataset set data guiding the training process. Progres-
is likely to push the model toward classifying most sively worsening performance of the model on
cases as healthy, reaching high specificity but poor the validation set data is a sign of overfitting.
sensitivity for disease. Common solutions to miti- The test dataset, unseen by the model during
gate this imbalance include strongly weighting the training, is used to assess a fully trained model’s
minority class in the loss function (63) or oversam- ability to generalize to new data. Ideally the test
pling the minority class to expose the model equally dataset is evaluated only once. Multiple evaluations
to all classes (64). Focusing on difficult recurrently of the test dataset among several training cycles
misclassified cases may also improve training ef- may lead to overfitting on the test set, invalidating
ficiency (65). In addition, an example of normaliza- its utility for predicting real-world performance.
tion, the process of shifting and scaling variables so Real-world performance of a model can be
that they have comparable statistical distributions, further assessed by separate datasets that are
can be found in Figure E2 (66). completely external from the original data col-
RG • Volume 41 Number 5 Cheng et al 1441

Figure 17. Common metrics used to assess model performances on various tasks: calibration plot for prediction probabilities (a), re-
ceiver operating characteristic curve (ROC) for classification of binary classes or dichotomized ordinal classes (b), confusion matrix for
classification of multiple classes of objects (c), intersection over union (IOU) for segmentation and detection (d), and Dice score (e).
Both IOU and Dice score quantify the degree of spatial overlap between ground truth and predicted masks.

lection. Such an assessment may include tempo- probability can be applied. As deep learning
ral validation with newly recruited patients, or models typically output a probability that an
geographic validation with data from a different image belongs to a particular class, a decision
site (67). Geographic validation may be especially threshold probability can be adjusted to trade off
helpful to evaluate how a model works on data sensitivity and specificity in the model. Cali-
acquired with different equipment or technical bration plots assess the accuracy of a model’s
parameters or with different patient populations. output probability by plotting the fraction of true
Once a model is deployed, performance positives in a test set as a function of the output
should be monitored to detect any bias or loss probability. A diagonal line on the calibration
of accuracy. Modes of continuous learning have plot represents perfect calibration, meaning that
been proposed to keep models current with the output probability of the model accurately
changing data and equipment configurations measures the model’s uncertainty.
(68). For instance, feedback from radiologist us- A receiver operating characteristic (ROC)
ers who may accept or reject the findings of the curve depicts the trade-off of sensitivity and spec-
system could theoretically be used as new train- ificity by plotting the true-positive rate (sensitiv-
ing data to improve performance. ity) versus the false-positive rate (1-specificity) as
the decision threshold is varied. The area under
Performance Metrics the ROC curve (AUC) provides a measure of
Metrics are the quantitative assessment of model model performance across all decision thresholds,
performance during training, validation, and with a perfect model having an AUC of 1 and a
monitoring steps. Appropriate selection of met- random model having an AUC of 0.5. However,
rics is dependent on the task (Fig 17). from a practical standpoint it is useful to report
For simple binary classification (eg, assess- sensitivity and specificity at a decision threshold
ing whether some disease process is present), optimized for the intended use case of the model.
the common biostatistical metrics of sensitivity, For multiclass classifications (eg, for mul-
specificity, and positive and negative predictive tiple lesion types), statistics for each class can be
1442 September-October 2021 [Link]

Figure 18. Common visualization techniques for a classification network as depicted on axial contrast-enhanced CT images.
(a) Features are learned by the CNN from training samples. (b) The saliency map technique computes the gradient of the output class
with respect to the input image, highlighting which pixels were involved in the final classification result (71). (c) Gradient-weighted
class activation mapping (grad-CAM) uses class probability and backpropagation from the last convolution layer to provide an atten-
tion map (72).

reported. These statistics can be averaged across want to visualize the areas in an image that ex-
classes, optionally weighting each class accord- plain a particular classification. Such visualization
ing to its prevalence in the test set. Confusion can increase a user’s confidence in the system or
matrices are contingency tables that tabulate the help reveal confounding factors that influence the
predicted classes for the instances of each actual system’s classification, such as external markers
class in a test set and are useful to help evaluate on a chest radiograph (70).
whether particular classes tend to be confused. The most common methods for visualization
Object detection and segmentation require calculate gradients on the basis of a forward and
metrics that describe how well a predicted area backward pass through the network for an image
matches the ground truth area, which is typically (Fig 18). Saliency maps (71) compute gradients
delineated by a radiologist. The intersection is of the class score with respect to image pixels;
the overlap between the predicted area and the these gradients indicate which pixels need to be
ground truth area, while the union is the total changed the least to affect the class score the
area encompassed by the prediction and ground most. Class activation maps (CAMs) are exem-
truth. Intersection over union and Dice score plified by gradient-weighted CAM (Grad-CAM)
(69) combine these measures in slightly different (72), which computes gradients of the class score
ways, but both equal 1 in the case of a perfect with respect to channels in the last convolutional
match between prediction and ground truth. layer in the model rather than to the input im-
For object detection, a particular detection can age. These gradient values are used to produce
be considered correct by comparing intersection a weighted sum of the channels in this layer,
over union with a cutoff value (eg, 0.5). Preci- resulting in a heat-map of important features in
sion is the number of true-positive detections as a the input image. Although this map is coarser
fraction of all detections and measures the model’s than a saliency map, the features tend to be more
positive predictive value. Recall is the number of specific to the predicted class.
true-positive detections as a fraction of all ground Visualization techniques are less relevant in
truth objects and measures the model’s sensitiv- object detection and segmentation tasks, where
ity. Thresholding the confidence values assigned the model output already provides relevant local-
to bounding boxes by an object detection model ization information.
is used to trade-off precision and recall, resulting
in a precision-recall curve (analogous to an ROC Future Directions
curve). Average precision (AP) summarizes model A proposed solution to privacy concerns re-
precision across the entire range of recall values garding multisite sharing of clinical image data
and is calculated by a modified area under the is federated learning (6). Federated learning
precision-recall curve, analogous to the AUC. The allows data to stay with the originating hospital,
mean AP is the average of the AP calculated for all with neural network training instead distributed
the detected types of objects. among the different institutions. However, there
remain formidable obstacles to such a strategy,
Visualization including accounting for the heterogeneity of
While deep neural networks can perform state-of- patient populations across institutions without
the-art image classification, clinical users typically centralized access to all the data.
RG • Volume 41 Number 5 Cheng et al 1443

Most deep learning models in radiology pro- References

cess two-dimensional (2D) images even when the 1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature
image datasets are three-dimensional (3D). In- 2015;521(7553):436–444.
2. Goodfellow I, Bengio Y, Courville A. Deep learning. Cam-
creased availability of medical 3D image data­sets bridge, Mass: MIT Press, 2016.
will likely result in evolution and optimization of 3. Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep
3D CNN architectures. Evolving alternatives to learning in medical image analysis. Med Image Anal
2017;42:60–88.
3D CNNs include combinations of 2D CNNs 4. Chartrand G, Cheng PM, Vorontsov E, et al. Deep
with neural networks specialized for sequence Learning: A Primer for Radiologists. RadioGraphics
data to process sequential 2D images of a 3D 2017;37(7):2113–2131.
5. Montagnon E, Cerny M, Cadrin-Chênevert A, et al. Deep
volume. learning workflow in radiology: a primer. Insights Imaging
As deep learning models transition into clini- 2020;11(1):22.
cal applications, we must consider the ethical 6. Willemink MJ, Koszek WA, Hardell C, et al. Preparing
Medical Imaging Data for Machine Learning. Radiology
ramifications of their use (73). Bias in machine 2020;295(1):4–15.
learning remains under-researched. For instance, 7. Salman S, Liu X. Overfitting Mechanism and Avoidance
deep learning researchers usually report ag- in Deep Neural Networks. CoRR. 2019;abs/1901.06566.
[Link] Published January 19,
gregate metrics over the entire dataset without 2019. Accessed May 1, 2021.
consideration of subgroups, especially under- 8. Sandfort V, Yan K, Pickhardt PJ, Summers RM. Data
served populations that are underrepresented in augmentation using generative adversarial networks (Cycle-
GAN) to improve generalizability in CT segmentation tasks.
the training data. Sci Rep 2019;9(1):16884.
Since deep learning models are likely to 9. Goodfellow IJ, Pouget-Abadie J, Mirza M, et al. Generative
serve as adjunctive tools for radiologists, work Adversarial Nets. In: Ghahramani Z, Welling M, Cortes C,
Lawrence ND, Weinberger KQ, eds. Advances in Neural
on model interpretability is crucial to adoption Information Processing Systems 27: NIPS 2014, Montreal,
and usage of these typically black-box models. Canada, December 8–13, 2014; 2672–2680.
For example, in contrast to a CNN that learns 10. Heim E, Roß T, Seitel A, et al. Large-scale medical image
annotation with crowd-powered algorithms. J Med Imaging
deterministic weights, a Bayesian neural network (Bellingham) 2018;5(3):034002.
learns parameters of random variables used to 11. Mehta P, Sandfort V, Gheysens D, Braeckevelt GJ, Berte J,
sample weights. These parameters can be used Summers RM. Segmenting The Kidney On CT Scans Via
Crowdsourcing. In: 2019 IEEE 16th International Sym-
to express the uncertainty of the model’s predic- posium on Biomedical Imaging (ISBI 2019), Venice, Italy,
tions and thereby help radiologists understand a April 8–11, 2019. Piscataway, NJ: IEEE, 2019; 829–832.
model’s limitations. 12. Zech J, Pain M, Titano J, et al. Natural Language-based
Machine Learning Models for the Annotation of Clinical
Good model performance does not guarantee Radiology Reports. Radiology 2018;287(2):570–580.
improved patient outcomes. The value of a model 13. Lee DH. Pseudo-Label: The Simple and Efficient Semi-
is dependent on its impact on clinical decisions Supervised Learning Method for Deep Neural Networks.
ICML Workshop on Challenges in Representation Learning,
and the nature and prevalence of the clinical Atlanta, GA, 2013.
problem. As a result, analogous to other technol- 14. DSI Use Cases. American College of Radiology Data Science
ogy assessments, controlled studies measuring Institute. [Link] Accessed July 2, 2020.
15. The Cancer Imaging Archive (TCIA). [Link]
practical clinical endpoints are necessary for [Link]/. Accessed July 2, 2020.
understanding the clinical value of deep learning. 16. Long J, Shelhamer E, Darrell T. Fully Convolutional Net-
works for Semantic Segmentation. Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
Conclusion Piscataway, NJ: IEEE, 2015; 3431–3440.
Deep learning is an artificial intelligence tech- 17. Krizhevsky A, Sutskever I, Hinton GE. ImageNet Clas-
nique that has been successful in computer vi- sification with Deep Convolutional Neural Networks. In:
Pereira F, Burges C, Bottou L, Weinberger K, editors.
sion. Familiarity with the key concepts described Advances in Neural Information Processing Systems 25.
in this article will help radiologists stay informed Curran Associates; 2012:1097–1105.
on the advances in deep learning and facilitate 18. Simonyan K, Zisserman A. Very Deep Convolutional Net-
works for Large-Scale Image Recognition. Presented at the
clinical adoption of these techniques. International Conference on Learning Representations, San
Diego, CA, May 7–9, 2015.
Disclosures of Conflicts of Interest.—I.P. Activities related to 19. Lin TY, Maire M, Belongie S, et al. Microsoft COCO:
the present article: disclosed no relevant relationships. Activi- Common Objects in Context. arXiv:14050312 [prepint]
ties not related to the present article: consultant for [Link]. Other [Link] Posted May 1, 2014. Ac-
activities: disclosed no relevant relationships. A.T. Activities cessed March 3, 2021.
related to the present article: disclosed no relevant relationships. 20. Deng J, Dong W, Socher R, Li LJ, Li K, Li F. ImageNet:
Activities not related to the present article: research scholarships A Large-Scale Hierarchical Image Database. 2009 IEEE
from Fonds de recherche du Québec en Santé (FRQ-S) and Conference on Computer Vision and Pattern Recognition,
Fondation de l’association des radiologistes du Québec; ac- Miami, FL, June 20–25, 2009. Piscataway, NJ: IEEE,
tive grants from Institut devalorisation d données (IVADO), 2009; 248–255.
Onco-Tech Project Grant (consortium composed of Onco- 21. Everingham M, Gool L, Williams CK, Winn J, Zisserman
pole, Medteq, Institut TransMedTech, Société de recherche A. The Pascal Visual Object Classes (VOC) Challenge. Int
sur le cancer), and the Canadian Institutes of Health Research J Comput Vis 2010;88(2):303–338.
(CIHR #389385); and speakers honoraria from Siemens 22. Papers with Code. Browse the State-of-the-Art in Machine
Healthineers and Eli Lilly. Other activities: disclosed no rel- Learning. [Link] Accessed
evant relationships. October 21, 2020.
1444 September-October 2021 [Link]

23. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for 42. Lin TY, Dollár P, Girshick R, He K, Hariharan B,
Image Recognition. In: Proceedings of the IEEE Conference Belongie S. Feature Pyramid Networks for Object De-
on Computer Vision and Pattern Recognition, Las Vegas, tection. ArXiv:161203144 [preprint] [Link]
NV, June 27–30, 2016; 770–778. abs/1612.03144. Posted December 9, 2016. Accessed
24. Sun S, Chen W, Wang L, Liu X, Liu TY. On the Depth of September 14, 2020.
Deep Neural Networks: A Theoretical View. In: Proceedings 43. Hui J. Understanding Feature Pyramid Networks for
of the Thirtieth AAAI Conference on Artificial Intelligence, object detection (FPN). Medium. [Link]
Phoenix, AZ, February 12–17, 2016: AAAI Press, 2016; jonathan_hui/understanding-feature-pyramid-networks-
2066–2072. for-object-detection-fpn-45b227b9106c. Published 2020.
25. Li H, Xu Z, Taylor G, Studer C, Goldstein T. Visualizing Accessed September 14, 2020.
the Loss Landscape of Neural Nets. In: Bengio S, Wallach 44. Mishra D. Transposed Convolution Demystified. Towards
H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, Data Science. [Link]
eds. Advances in Neural Information Processing Systems convolution-demystified-84ca81b4baba. Published March
31. Red Hook, NY: Curran Associates, 2018; 6389–6399. 10, 2020. Accessed October 21, 2020.
26. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen 45. Badrinarayanan V, Kendall A, Cipolla R. SegNet: A Deep
LC. MobileNetV2: Inverted Residuals and Linear Convolutional Encoder-Decoder Architecture for Image
Bottlenecks. ArXiv:180104381 [preprint] [Link] Segmentation. IEEE Trans Pattern Anal Mach Intell
org/abs/1801.04381. Posted January 13, 2018. Accessed 2017;39(12):2481–2495.
October 21, 2020. 46. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional
27. Szegedy C, Liu W, Jia Y, et al. Going deeper with convolu- Networks for Biomedical Image Segmentation. In: Navab
tions. In: 2015 IEEE Conference on Computer Vision and N, Hornegger J, Wells WM, Frangi AF, eds. Medical Image
Pattern Recognition. Piscataway, NJ: IEEE, 2015; 1–9. Computing and Computer-Assisted Intervention: MICCAI
28. Zagoruyko S, Komodakis N. Wide Residual Networks. ArXiv: 2015. Cham, Switzerland: Springer, 2015; 234–241.
1605.07146 [preprint] [Link] 47. He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN.
Posted May 23, 2016. Accessed May 1, 2021. 2017 IEEE International Conference on Computer Vision
29. Tan M, Le QV. EfficientNet: Rethinking Model Scaling (ICCV), Venice, Italy, October 22–29, 2017. Piscataway,
for Convolutional Neural Networks. In: Chaudhuri K, NJ: IEEE, 2017; 2980–2988.
Salakhutdinov R, eds. Proceedings of the 36th international 48. Dumoulin V, Visin F. A guide to convolution arithmetic
conference on machine learning. Cambridge, MA: PMLR, for deep learning. ArXiv:160307285 [preprint] [Link]
2019;97:6105–6114. org/abs/1603.07285. Posted March 23, 2016. Accessed
30. Ju C, Bibaut A, van der Laan M. The relative performance of October 21, 2020.
ensemble methods with deep convolutional neural networks 49. Yu F, Koltun V. Multi-Scale Context Aggregation by Dilated
for image classification. J Appl Stat 2018;45(15):2800–2818. Convolutions. Presented at the 4th International Confer-
31. Halabi SS, Prevedello LM, Kalpathy-Cramer J, et al. The ence on Learning Representations, ICLR 2016, San Juan,
RSNA Pediatric Bone Age Machine Learning Challenge. PR, May 2–4, 2016.
Radiology 2019;290(2):498–503. 50. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille
32. Pan I, Cadrin-Chênevert A, Cheng PM. Tackling the Ra- AL. DeepLab: Semantic Image Segmentation with Deep
diological Society of North America Pneumonia Detection Convolutional Nets, Atrous Convolution, and Fully
Challenge. AJR Am J Roentgenol 2019;213(3):568–574. Connected CRFs. IEEE Trans Pattern Anal Mach Intell
33. Fort S, Hu H, Lakshminarayanan B. Deep Ensembles: A 2018;40(4):834–848.
Loss Landscape Perspective. ArXiv:191202757 [preprint] 51. Mirza M, Osindero S. Conditional Generative Adver-
[Link] Posted June 25, 2020. sarial Nets. ArXiv:14111784. [preprint] [Link]
Accessed August 5, 2020. abs/1411.1784. Published November 6, 2016. Accessed
34. Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, August 11, 2020.
Inception-ResNet and the Impact of Residual Connections 52. Zhu JY, Park T, Isola P, Efros AA. Unpaired Image-to-Image
on Learning. ArXiv:160207261. [preprint] [Link] Translation Using Cycle-Consistent Adversarial Networks.
org/abs/1602.07261. Posted February 26, 2016. Accessed 2017 IEEE International Conference on Computer Vision
May 18, 2018. (ICCV), Venice, Italy, October 22–29, 2017. Piscataway,
35. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards NJ: IEEE, 2017; 2242–2251.
Real-Time Object Detection with Region Proposal Net- 53. Welander P, Karlsson S, Eklund A. Generative Adversarial
works. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Networks for Image-to-Image Translation on Multi-Contrast
Garnett R, eds. Advances in Neural Information Processing MR Images: A Comparison of CycleGAN and UNIT. ArXiv
Systems 28. Red Hook, NY: Curran Associates, 2015; 91–99. 1806.0777 [preprint] [Link]
36. Redmon J, Divvala S, Girshick R, Farhadi A. You Only Posted June 20, 2018. Accessed May 1, 2021.
Look Once: Unified, Real-Time Object Detection. In: 54. Yi X, Walia E, Babyn PS. Unsupervised and semi-supervised
Proceedings of the IEEE Conference on Computer Vision learning with Categorical Generative Adversarial Networks
and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, assisted by Wasserstein distance for dermoscopy image
2016; 779–788. Classification. ArXiv 1804.03700 [preprint] [Link]
37. Girshick R, Donahue J, Darrell T, Malik J. Rich Feature org/abs/1804.03700. Posted April 10, 2018. Accessed
Hierarchies for Accurate Object Detection and Semantic May 1, 2021.
Segmentation. In: Proceedings of the IEEE Conference 55. Rezaei M, Yang H, Harmuth K, Meinel C. Conditional
on Computer Vision and Pattern Recognition, 2014; Generative Adversarial Refinement Networks for Unbal-
580–587. anced Medical Image Semantic Segmentation. 2019 IEEE
38. Girshick R. Fast R-CNN. In: Proceedings of the IEEE Inter- Winter Conference on Applications of Computer Vision
national Conference on Computer Vision, 2015; 1440–1448. (WACV), Waikoloa, HI, January 7–11, 2019. Piscataway,
39. Redmon J, Farhadi A. YOLO9000: Better, Faster, Stronger. NJ: IEEE, 2019; 1836–1845.
In: 2017 IEEE Conference on Computer Vision and Pattern 56. Li Z, Zhang T, Wan P, Zhang D. SEGAN: Structure-
Recognition (CVPR), 2017; 6517–6525. Enhanced Generative Adversarial Network for Compressed
40. Liu W, Anguelov D, Erhan D, et al. SSD: Single Shot Sensing MRI Reconstruction. Proceedings of the AAAI
MultiBox Detector. In: Leibe B, Matas J, Sebe N, Welling Conference on Artificial Intelligence 33(1):1012–1019.
M, eds. Computer Vision: ECCV 2016—Lecture Notes in 57. Jin D, Xu Z, Tang Y, Harrison AP, Mollura DJ. CT-Realistic
Computer Science, vol 9905. Cham, Switzerland: Springer, Lung Nodule Simulation from 3D Conditional Generative
2016; 21–37. Adversarial Networks for Robust Lung Segmentation. In:
41. Lin TY, Goyal P, Girshick R, He K, Dollar P. Focal Loss Frangi A, Schnabel J, Davatzikos C, Alberola-López C,
for Dense Object Detection. In: Proceedings of the IEEE Fichtinger G, eds. Medical Image Computing and Computer
International Conference on Computer Vision. Piscataway, Assisted Intervention: MICCAI 2018. Cham, Switzerland:
NJ: IEEE, 2017; 2980–2988. Springer, 2018; 732–740.
RG • Volume 41 Number 5 Cheng et al 1445

58. Yan P, Xu S, Rastinehad AR, Wood BJ. Adversarial Image September 8–14, 2018, Proceedings, Part XIII. Springer;
Registration with Application for MR and TRUS Image 2018:3–19.
Fusion. In: Shi Y, Suk HI, Liu M, eds. Machine Learning 67. Park SH, Han K. Methodologic Guide for Evaluating
in Medical Imaging: MLMI 2018. Cham, Switzerland: Clinical Performance and Effect of Artificial Intelligence
Springer, 2018;197–204 Technology for Medical Diagnosis and Prediction. Radiol-
59. Bergstra J, Bengio Y. Random Search for Hyper-Parameter ogy 2018;286(3):800–809.
Optimization. J Mach Learn Res 2012;13(10):281–305. 68. Pianykh OS, Langs G, Dewey M, et al. Continuous Learn-
60. Smith LN. A disciplined approach to neural network hyper- ing AI in Radiology: Implementation Principles and Early
parameters: Part 1 – learning rate, batch size, momentum, Applications. Radiology 2020;297(1):6–14.
and weight decay. ArXiv:180309820 [preprint] [Link] 69. Bertels J, Eelbode T, Berman M, et al. Optimizing the Dice
org/abs/1803.09820. Posted March 26, 2018. Accessed Score and Jaccard Index for Medical Image Segmentation:
August 8, 2020. Theory and Practice. In: Shen D, Liu T, Peters TM, et al,
61. Smith LN. Cyclical Learning Rates for Training Neural eds. Medical Image Computing and Computer Assisted
Networks. 2017 IEEE Winter Conference on Applications of Intervention: MICCAI 2019. Cham, Switzerland: Springer,
Computer Vision (WACV), Santa Rosa, CA, March 24–31, 2019; 92–100.
2017. Piscataway, NJ: IEEE, 2017; 464–472. 70. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ,
62. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Oermann EK. Confounding variables can degrade gener-
Salakhutdinov R. Dropout: A Simple Way to Prevent alization performance of radiological deep learning models.
Neural Networks from Overfitting. J Mach Learn Res CoRR. 2018;abs/1807.00431 [preprint] [Link]
2014;15(56):1929–1958. abs/1807.00431. Posted July 2, 2018. Accessed May 1, 2021.
63. Huang W, Song G, Li M, Hu W, Xie K. Adaptive Weight 71. Simonyan K, Vedaldi A, Zisserman A. Deep Inside Convo-
Optimization for Classification of Imbalanced Data. In: Sun lutional Networks: Visualising Image Classification Models
C, Fang F, Zhou ZH, Yang W, Liu ZY, eds. Intelligence and Saliency Maps. ArXiv:13126034. [preprint] [Link]
Science and Big Data Engineering: IScIDE 2013. Berlin, org/abs/1312.6034. Posted December 20, 2013. Accessed
Germany: Springer, 2013; 546–553. July 23, 2017.
64. Buda M, Maki A, Mazurowski MA. A systematic study of the 72. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh
class imbalance problem in convolutional neural networks. D, Batra D. Grad-CAM: Visual Explanations from Deep
Neural Netw 2018;106:249–259. Networks via Gradient-Based Localization. 2017 IEEE
65. Shrivastava A, Gupta A, Girshick R. Training Region- International Conference on Computer Vision (ICCV),
based Object Detectors with Online Hard Example Venice, Italy, October 22–29, 2017. Piscataway, NJ: IEEE,
Mining. ArXiv:160403540. [preprint] [Link] 2017; 618–626.
abs/1604.03540. Posted April 12, 2016. Accessed Sep- 73. Geis JR, Brady AP, Wu CC, et al. Ethics of Artificial Intel-
tember 17, 2020. ligence in Radiology: Summary of the Joint European and
66. Wu Y, He K. Group Normalization. In: Ferrari V, Hebert North American Multisociety Statement. J Am Coll Radiol
M, Sminchisescu C, Weiss Y, eds. Computer Vision: ECCV 2019;16(11):1516–1521.
2018—15th European Conference, Munich, Germany,

TM
This journal-based SA-CME activity has been approved for AMA PRA Category 1 Credit . See [Link]/learning-center-rg.