Deep Learning and Imaging for the Orthopaedic Surgeon: How

Machines "Read" Radiographs

The technology of deep learning is beginning to enter the clinical setting, and it is anticipated that
orthopaedic surgeons will soon be exposed to machines that automatically "read" medical imaging
studies like radiographs, CT scans, and MRI scans. Understanding this technology, specifically how a
deep learning method called Convolutional Neural Networks (CNNs) works, is becoming important for
surgeons.

Machine Learning vs. Traditional Programming

Machine learning involves using algorithms to identify patterns in data, build mathematical models
based on those patterns, and then use those models to make determinations or predictions.

* Traditional Programming: Requires a human to enter explicit, line-by-line instructions in the form of
computer code.

* Machine Learning (and Deep Learning): Does not require step-by-step guidance. Humans only need to
supply data and a learning system, and the computer learns patterns on its own.

How a CNN Model Works

A CNN is a type of Artificial Neural Network (ANN). Since computers "think" in numbers, an image's
information must be converted into a numerical form.

Artificial Neural Networks (ANNs)

* ANNs are a network of interconnected mathematical functions inspired by biologic neurons.

* Artificial neurons exchange the results of their mathematical functions.

* They are organized in layers of interconnected neurons.

* Each neuron takes weighted inputs, sums them to produce an output or "activation," and this output
is passed to the next layer.

* ANNs consist of three layer categories: input, hidden, and output layers.

* The "deep" in deep learning refers to the mathematical depth, which is the number of hidden layers.
These hidden layers, shielded from view between the input and output layers, are where mathematical
functions add depth.

Image Processing and Features

* For computers, images are broken down into pixels—small, illuminated dots.
* Each pixel is represented numerically, with a number assigned to its brightness (e.g., white = 1, black =
0).

* For grayscale images like radiographs, the image is represented as a 2-dimensional matrix (a table of
numbers in columns and rows).

* CNNs are chains of interconnected artificial neurons designed to detect different shapes or visual
features within an image, much like the visual cortex.

| Low-level | Elementary shapes | Lines and edges |

| Mid-level | More complex shapes | Cortical border, joint line |

| High-level | Entire objects | Femoral head |

Key CNN Layers and Functions

The types of layers commonly used for "reading" a radiograph include: convolutional layers, pooling
layers, and fully connected layers.

* Convolutional Layers

* The first layer detects low-level features like lines or edges using a "drawing" called a filter.

* The filter is a small matrix characterizing the pixel brightness of the shape it's looking for.

* The filter is scanned across the image, and the degree of similarity is measured at each step.

* This mathematical process is called convolving the image with the filter, giving the CNN its name.

* The output of this neuron is a feature map.

* Feature maps are passed to the next layer, where neurons detect more complex shapes by looking at
several prior feature maps.

* Pooling Layers

* Pooling layers are included to avoid overly specific filters that can result from large filter sizes.

* They summarize the visual features in each area, pooling together the most relevant information and
abstracting away less helpful details (e.g., the black background of the radiograph).

* The result is a lower-resolution version of the feature maps.

* Fully Connected Layers

* The final layers are dedicated to answering the question the model was trained for (e.g., "is a
fracture present?").

* Each neuron in these layers considers all feature measurements from the prior layer.

* They look for a specific combination of higher-level features and output a confidence measure.

* For simple fracture detection, the final layer might be a single neuron outputting a number (0 to 1)
indicating the certainty of a fracture.

* For fracture classification, the number of neurons would equal the types of fractures the system can
identify, plus one for "no fracture".

️How a CNN Is Created: Supervised Learning

Creating a CNN model to "see" fractures requires three things: a rough draft model, data, and a teacher.
The "teacher" is actually a proctor/examiner that tests the model.

* Initial Setup: The model starts with random values for all filters and neuron functions, so its initial
output is random.

* Testing and Adjustment: The programmer provides the proctor with training data in the form of tests
and answer keys.

* The Learning Cycle:

* The model initially scores poorly.

* The proctor suggests what changes in the mathematical functions (like filter shapes and neuron input
weights) would have given a better score.

* The CNN is updated with these adjustments and given a new version of the test.

* This cycle of repeatedly testing and adjusting the mathematical functions is called supervised
learning.

* This process is repeated until the CNN's accuracy levels off.

* Successful Training: A CNN is successfully trained when it can find the most important patterns in the
training data, apply those patterns to new prospective data, and maintain an acceptable degree of
accuracy.

Data and Limits to Accuracy

 * Machine learning creates programs through examples, not logic, meaning a model is only as good as
the data it is trained on.

* Overfitting: Too little data can cause the model to "memorize" the training data and predict poorly on
new data.

* Data Quality: Even substantial data may not have enough difficult radiographs to train the model for
challenging cases.

* Human Limits: Accuracy is often limited by the task itself. If experts disagree on difficult radiographs, a
model trained to match expert opinion will reflect these limits of human judgment.

* Programmer Role: While the specific filters are learned from data, the programmer still creates the
model architecture (the kind and count of neurons in each layer), which determines the limits of what
the CNN can "see".

Conclusion

A CNN does not perceive an image as humans do; it reads pixels as numerical values of brightness and
proximity. Its "neural" structure, inspired by biology, allows it to extract visual features (shapes) from an
image to make useful predictions. For fracture identification, a machine "reads" a radiograph by
applying a CNN trained for fracture detection to predict the likelihood of a fracture.