Self-supervised

Learning for
Inverse Problems
EI, ICCV 21 Oral
REI, CVPR 22 Oral
ZS-SSL, ICLR 22 Poster

2022.09.14
Inverse Problems
Forward process: generate from
noise
𝑥

𝐴 ⊕ 𝑦
sensing 𝑦 = 𝐴(𝑥)+𝝐

unknow

Inverse problem: recover from

 Example: Black Hole Imaging
𝑥
𝐴

Black Hole image

Event Horizon Telescope
(EHT)
First M87 event horizon telescope results. IV. Imaging the central supermassive black hole.
The Astrophysical Journal Letters 875.1 (2019): L4.
?
Example: Black Hole Imaging
𝑦
𝐴 𝑥

sensing recove
r

Event Horizon Telescope Spatial-frequency Black Hole image

(EHT)
 Example: Magnetic Resonance Imaging
(MRI)

𝐴 𝑦 𝑥

recove
? sensing r

MRI scanner k-space data Brain image

 Example: Magnetic Resonance Imaging
(MRI)

𝐴 𝑦 𝑥

recove
? sensing r

Cryo-EM 2D projections of Covid-19 virus’

Protein particles structure
 𝑥 𝐴 𝑦 reconstruction

MRI
sensin
𝑓
: Fourier matrices
? g recover
Brain image

Black Hole imaging sensin 𝑓

？ g recover Black Hole
: Spatial-frequency
e.g. Event Horizon Telescope
image
(EHT)

Cryogenic electron 𝑓
microscopy
(Cryo-EM)
? sensin
g
recover
Covid-19 virus’
: 2D projections of structure
Protein particles

And image SR, CT, PET, PR

 Learning with groundtruth
Idea: learn a neural network mapping between and groundtruth

{𝑦 } {𝑥 }
2
arg min 𝔼 𝑥 , 𝑦 ||𝑥 − 𝑓 ( 𝑦 )||
𝑓

 SOTA reconstruction on e.g. super-resolution, etc…

 Supervised learning based on pairs (could be unpaired)
 could be e.g. Unet, RestNet, Transformers, unrolled models,
etc…
 Learning with groundtruth
Challenging: If obtaining training targets is expensive or impossible

{𝑦 } ?

 Biomedical imaging (e.g. CT, MRI)

 Astronomical imaging (e.g. black hole)
Self-supervised learning
• Image domain ssl: noise2noise
• Measurement domain ssl: e.g., Fourier domain, wss.
• Other…
 Equivariant Imaging (EI)
- Learning Beyond the Range
Space
Dongdong Chen, Julián Tachella, Mike E. Davies.

The University of Edinburgh

In ICCV 2021 (oral)

Code: [Link]
Background
Imaging systems capture noisy measurements of a signal
through a linear operator :

ℝ𝑛
𝑥
Inverse Problems 𝐴 ?
 Target: recover the signal set from only measurements .
𝑦 ℝ𝑚
 Challenge: generally the solution is not unique due to ill-posed , e.g. .
 Applications: compressive MRI, sparse-view CT, super-resolution, inpainting, etc.
Model-based solutions 😅
Maximum a posterior (MAP): → .
 Requires strong prior knowledge of x, e.g. sparsity [Daubechies et al. 2004; Ng et al. 2010; Boyd et al.
2011]

 Generally with no-closed-form solutions + long inferring time …

Learning-based solutions😅
Learn a reconstruction function using sampled data
 Supervised: need paired or unpaired x and y [ Jin [Link] 2007].
 Unsupervised: need multiple operators [ Bora et al. 2018; Lehtinen et al. 2019; Liu et al. 2020; Hendriksen et al.
2020]

What if neither ground-truth nor multiple operators are available for learning?
Target:
The EI model aim to learn the reconstruction where
 No groundtruth data for training (most inverse problem don't have) ground-truth!
 Only a single forward operator is available;
 has a non-trivial nullspace (i.e. ).
nullspace()
𝑥

Challenges:
 We have NO information about outside the range().
 It is IMPOSSIBLE to learn the signal set using alone.
range()
┼
𝐴 𝑦
Motivation-Geometric Priors
The natural signals enjoy simple but fundamental geometric priors:
1) Low dimensionality: authors assume the signal set has a low-dimensional structure.
2) Symmetry: authors assume the signal set is invariant to a certain group of certain of
transformations which are orthogonal matrices such that for , :

i.e., the sets and are the same.

 Natural images are shift invariant.

 In CT/MRI, images are rotation invariant
- organs can be imaged at different angles.
Key observations:
 Invariance provides access to implicit operators with potentially different
range spaces:

Where , .

 For , , the composition is equivariant to transformation :

 nullspace()
𝑥
MC constraint is not
Equivariant Imaging: enough to learn :
It can’t capture ?
For , , the funcion should enjoy: information about
outside the range of
😅 range()
1) Measurement consistency
┼
𝐴 𝑦

2) Invariant set consistency

3) Invariant distribution consistency

Equivariant Imaging:
Define: and is a neural net ( → )

Training loss:
 : MSE; equivariance strength
 Test time: reconstruct signals with learned , i.e.
Adversarial EI:
One can further enforce the invariant distribution consistency .

Training loss:
Analysis:
 A necessary condition for recovering the signal set from is:

 How big the group : at least .

 e.g. if is invariant to single reflections (), we need at least .

 The range( ) cannot be invariant to :

range() range
Experiments:
 Sparse-view CT
 CT 100 clinical dataset
 100 real CT images
 128*128 pixels
 : radon transformation
 50 views
 : iradon
 random rotation
Experiments:
 Inpainting
 Urban 100 dataset
 100 building images
 256*256 pixels
 : radon drop 30% of pixels
 ,;=
 is a binary mask (a fixed set of deleted pixels)
 random shift
Take away:
 New unsupervised learning framework for inverse problems.
 Self-supervised; End-to-end; physics-based
 Leverages simple but fundamental priors about natural signals
 Symmetry and low-dimensionality
 Framework has theoretical identifiability guarantees.
Robust Equivariant Imaging
(REI)
A fully unsupervised framework for learning to image from noisy and partial
measurements

Dongdong Chen, Julián Tachella, Mike E. Davies.

The University of Edinburgh

In CVPR 2022 (oral)

Code: [Link]
 Forward process: generate from
noise
𝑥

𝐴 ⊕ 𝑦
sensing 𝑦 = 𝐴(𝑥)+𝝐

unknow

Inverse problem: recover from

Equivariant Imaging:
Define: and is a neural net ( → )

Training loss:
 : MSE; equivariance strength
 Test time: reconstruct signals with learned , i.e.
However, EI degrades with measurements noise!
 Robust Equivariant Imaging (REI)
REI: Learning from incomplete and noisy alone!
Idea: learning image model via sys. Equivariance + handling noise via Stein’s Unbiased Risk Estimator
(SURE)

⊕noise
⊕noise

REI:
 can be ANY neural nets!
 is specific to noise model:
 Poisson
 Mixed Poisson –Gaussian
 Gaussian:
 EI REI
y: undersampled data for y in loader: for y in loader:
x1: mc-recon t = select(T) t = select(T)
x2: EI-recon x1 = net([Link](y)) x1 = net([Link](y + noise))
x2 = t(x1) x2 = t(x1)
[Link]: measure process (x->y) x3 = x3 = net([Link]([Link](x2)) +
noise)
[Link]: reconstruct process (y->x)net([Link]([Link](x2)))
loss = MSE([Link](x1), y) loss = SURE([Link](x1), y)
+ alpha * MSE(x2, + alpha * MSE(x2, x3)
x3)
[Link]()
[Link]() update([Link]())
update([Link]())

SURE= [Link]((y1 - y0).pow(2)) / (K * m) - sigma2 + (2 * sigma2 / (tau *m * K)) * (b * (y2

- y1)).sum()
Summary
• EI
• REI
Zero-Shot Self-Supervised Learning for MRI
Reconstruction
Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Mehmet Akcakaya

Department of Electrical&Computer Engineering, University of Minnesota

In ICLR 2022 (Poster)
No code
Zero-shot learning: Deep Image
Prior (DIP)
Zero-shot learning: Deep Image
Prior (DIP)
Zero-shot learning: Deep Image
Prior (DIP)
ZS-SSL
𝐴 𝑦 𝑥

recove
? sensing r

MRI scanner k-space data Brain image

ZS-SSL
 Λ

Ω¿

Ω
Θ
Γ
ZS-SSL
锐评
Here is how the discussion went:
(6)- Reviewer pt6r is supportive of acceptance, but notes a few potential irregularities, such as the method pre-trained on
brain and tested on knees performing better than the method pre-trained on knees and tested on knees, and not providing a
comparison of the computational cost. The authors added a table to the appendix revealing that the computational costs
are very high, much higher than for DIP even. The reviewer was content with the response and raised the score.

(5) - Reviewer mBMk argues that the contribution is too incremental compared to prior work, in particular relative to the
results of [Yaman et al., 2020], and also argues that the idea of partitioning the measurements is not new. The authors
argue in response that their approach of partitioning the measurements is new, and the reviewer was inclined to raise the
score slightly, but still thinks that the novelty on the technical ML side remains limited, and doesn't want to back the
submission too much, and did not raise the score at the end in the system.

(5) - Reviewer 19v3 has the concern that the all elements used (transfer learning, plug-and-play, etc) are well known
techniques and have been applied before to MRI, and therefore thinks that the paper does not clear the bar for
acceptance. The paper points out that while those ideas might be applied for the first time to MRI, they have been used
before in other image reconstruction problems, in particular denoising.

I've read the paper in detail too, and am somewhat on the fence: I think it's very valuable to see that a clever application of
self-supervised learning works so well for MRI. I agree with the reviewers that the technical novelty is relatively small, but
on the other hand this is the first time that I see self-supervised learning being applied that successfully to MRI. I
don't share the concern about novelty --- yes, the paper's approach builds on prior work, but it's not clear from the literature
how well such a well tuned self-supervised learning approach would work.
What I would have liked to see in addition to the experimental results is a proper evaluation on the FastMRI dataset: An
advantage of the FastMRI dataset is that it provides a benchmark and if researchers evaluate on that benchmark (on the
testset/validation set) we can compare different methods well. The paper under review doesn't do that, it only evaluates on
30 test slices, and thus it's hard to benchmark the method. Also, the paper would benefit from more ablation studies.

In conclusion, I would be happy to discuss this paper at the conference, and think that other researchers in the
intersection of deep learning and inverse problems would be too, and therefore recommend acceptance.