Module 5: Image Sensors and Spectral Acquisition in Machine
Vision
1 Annotation
This module introduces the physical and electronic principles of image sensors in machine
vision systems. It explains how incident radiation is converted into an electrical signal using
photodiodes and how this signal is processed within CMOS sensor architectures. The
module further analyses acquisition modes, shutter mechanisms, and quantum efficiency,
and introduces spectral imaging principles including multispectral and hyperspectral
acquisition.
2 Learning Outcomes
After completing this module, students will be able to:
• Explain the physical principle of photoelectric conversion in photodiodes.
• Analyse CMOS pixel architecture and signal readout.
• Compare different image acquisition modes.
• Evaluate rolling and global shutter mechanisms.
• Interpret quantum efficiency and its impact on image quality.
• Design and compare spectral imaging approaches.
3 Structured Exposition
3.1 From Radiation to Digital Signal
Image sensors convert incident radiation into an electrical signal through a sequence of
physical and electronic processes, including photon absorption, charge generation, charge
accumulation, and signal readout.
This transformation represents a fundamental transition from a continuous radiometric
field to a discrete digital representation.
3.1.1 Physical Principle of Signal Formation
When electromagnetic radiation impinges on a photosensitive element (pixel), photons
with sufficient energy generate electron–hole pairs in the semiconductor material via the
photoelectric effect.
The number of generated charge carriers depends on:
• the number of incident photons,
• the wavelength of the radiation,
• the material properties of the sensor,
• the exposure duration.
3.1.2 Charge Generation Model
The total generated charge 𝑄 can be expressed as:
𝑄 = 𝜂(𝜆) ⋅ 𝑁photon
where:
• 𝑄– number of collected electrons,
• 𝜂(𝜆)– quantum efficiency,
• 𝜆 - wavelength-dependent
• 𝑁photon – number of incident photons.
This relation establishes a direct link between optical input (radiation) and electrical
output (charge).
3.1.3 Temporal Integration
Each pixel does not measure instantaneous intensity, but integrates incoming radiation
over a finite exposure time 𝑇:
𝑇
𝑄 = ∫ 𝜂(𝜆) Φ(𝑡) 𝑑𝑡
0
where Φ(𝑡) represents the photon flux.
Thus a pixel acts as a temporal integrator of radiant energy.
3.1.4 Spatial Sampling
In addition to temporal integration, the sensor performs spatial sampling:
• each pixel corresponds to a discrete spatial location,
• the continuous image formed by the optical system (Module 4) is sampled into a
grid.
Therefore, image formation consists of:
• spatial sampling (pixel grid),
• temporal integration (exposure),
• spectral weighting (via 𝜂(𝜆)).
3.1.5 Signal Formation Chain
The complete signal formation process can be summarised as:
Radiation → Photon flux → Electron generation → Charge accumulation
→ Voltage conversion → Digital signal
Each stage introduces constraints and potential sources of degradation.
3.1.6 Engineering Implications
The final pixel value depends on:
• illumination intensity (Module 3),
• spectral properties (Module 2),
• optical projection and blur (Module 4),
• sensor quantum efficiency,
• exposure time and readout strategy.
Thus, the image sensor does not merely “capture an image”, but performs a physically
constrained transformation of radiometric information into a digital signal.
3.1.7 System-Level Insight
Understanding this transformation is essential because it directly affects:
• signal-to-noise ratio (SNR),
• dynamic range,
• sensitivity,
• motion artefacts (via exposure time),
• spectral response.
This section therefore forms the conceptual foundation for:
• sensor architecture (Section 3.2),
• shutter mechanisms (Section 3.5),
• quantum efficiency (Section 3.6),
• spectral imaging (Section 3.8+).
3.2 Photodiode Principle
A photodiode is the fundamental sensing element of an image sensor. It converts incident
radiation into electrical charge through the photoelectric effect in a semiconductor
structure.
3.2.1 Physical Structure
A photodiode is typically formed by a p–n junction, consisting of:
• a p-type semiconductor layer (positive charge carriers),
• an n-type semiconductor layer (negative charge carriers),
• a depletion region at the interface.
The depletion region plays a critical role, as it provides an internal electric field that
separates generated charge carriers.
3.2.2 Photoelectric Conversion Mechanism
When a photon with sufficient energy is absorbed in the semiconductor, it generates an
electron–hole pair.
If this generation occurs within or near the depletion region:
• the electron is driven toward the n-region,
• the hole is driven toward the p-region.
This separation prevents recombination and enables charge collection.
3.2.3 Charge Accumulation
The generated electrons are collected and stored in a potential well, forming an electrical
charge that is proportional to the incident radiation.
The total accumulated charge depends on:
• the number of absorbed photons,
• the quantum efficiency 𝜂(𝜆),
• the exposure time 𝑇,
• the effective sensitive area of the photodiode.
This directly extends the model introduced in Section 3.1.
3.2.4 Spectral Sensitivity
The photodiode response is inherently wavelength-dependent.
This dependence is determined by:
• the semiconductor material (e.g., Si, InGaAs),
• the thickness of the active region,
• absorption depth of photons in the material.
Key implication:
• shorter wavelengths are absorbed near the surface,
• longer wavelengths penetrate deeper into the material.
Thus, the photodiode acts as a spectral filter defined by material physics.
3.2.5 Capacity and Saturation
Each photodiode has a finite charge capacity (full-well capacity), determined by:
• physical dimensions of the photodiode,
• design of the potential well.
Once this capacity is reached, additional photons do not increase the signal → saturation
occurs.
3.2.6 Dark Charge (Dark Signal)
Even in the absence of illumination, photodiodes generate charge due to thermal effects:
• thermal generation of electron–hole pairs,
• leakage currents in the semiconductor.
This unwanted signal is known as dark charge (or dark current)
It introduces noise and limits sensitivity, especially at long exposure times.
3.2.7 Engineering Implications
The behaviour of the photodiode directly determines:
• sensor sensitivity,
• noise characteristics,
• dynamic range,
• spectral response.
Design parameters such as:
• material selection,
• doping profile,
• photodiode geometry
must therefore be optimised according to the application.
3.2.8 System-Level Insight
The photodiode is not merely a passive detector - it defines how physical radiation is
transformed into a measurable electrical quantity.
Its properties must be co-designed with:
• illumination (Module 3),
• spectral filtering (Module 2),
• optical image formation (Module 4).
3.3 CMOS Pixel Architecture
Modern image sensors are predominantly based on CMOS (Complementary Metal-Oxide-
Semiconductor) technology. The fundamental sensing element—a photodiode—is integrated
with electronic circuitry at the pixel level to enable signal conversion, storage, and readout.
The most widely used architecture in machine vision is the 4-transistor (4T) pixel with a pinned
photodiode (PPD).
3.3.1 Pinned Photodiode (PPD)
A pinned photodiode is a modified photodiode structure designed to reduce noise and
improve charge transfer efficiency.
It consists of:
• a lightly doped n-type region (charge collection region),
• a p-type substrate,
• a heavily doped p+ surface layer.
The p+ layer “pins” the surface potential, which:
• suppresses surface-generated dark charge,
• stabilises the electrostatic potential,
• improves noise performance.
3.3.2 4-Transistor Pixel Structure
The 4T pixel architecture includes four key transistors:
• TG (Transfer Gate) – controls transfer of charge from the photodiode to the
readout node,
• FD (Floating Diffusion) – temporary charge-to-voltage conversion node,
• RST (Reset transistor) – resets the floating diffusion to a reference voltage,
• SF (Source Follower) – amplifies the signal,
• SEL (Select transistor) – connects the pixel to the column readout line.
This architecture enables efficient and low-noise signal processing directly within each
pixel.
3.3.3 Signal Readout Sequence
The signal acquisition process in a 4T CMOS pixel follows a defined sequence:
1. Reset phase
The floating diffusion node is reset to a known reference voltage.
2. Charge integration
The photodiode accumulates charge during exposure (as described in Section 3.1).
3. Charge transfer
The transfer gate is activated, moving accumulated charge from the photodiode to
the floating diffusion node.
4. Readout
The voltage change at the floating diffusion node is buffered by the source follower
and read via the column bus.
3.3.4 Charge-to-Voltage Conversion
The accumulated charge 𝑄 is converted into a voltage signal:
𝑄
𝑉=
𝐶
where:
• 𝑉– output voltage,
• 𝑄– accumulated charge,
• 𝐶– capacitance of the floating diffusion node.
This conversion defines the conversion gain of the pixel.
3.3.5 Advantages of CMOS Architecture
CMOS sensors offer several advantages:
• direct pixel-level readout,
• random access to pixels,
• high frame rates,
• low power consumption,
• integration of additional circuitry (e.g., ADC, amplification).
3.3.6 Region of Interest (ROI)
CMOS addressing schemes allow selective readout of a portion of the sensor:
• only selected rows and columns are read,
• enables faster acquisition,
• reduces data volume and processing requirements.
3.3.7 Noise Considerations
The CMOS pixel architecture influences several noise sources:
• readout noise – introduced during signal amplification,
• dark noise – from thermally generated charge (Section 3.2),
• reset noise – associated with resetting the floating diffusion node.
Pinned photodiodes and correlated double sampling (CDS) are used to minimise these
effects.
3.3.8 Engineering Implications
The pixel architecture determines:
• signal-to-noise ratio (SNR),
• dynamic range,
• sensitivity,
• achievable frame rate.
Design trade-offs include:
• pixel size vs resolution,
• capacitance vs conversion gain,
• noise vs readout speed.
3.3.9 System-Level Insight
The CMOS pixel is not merely a sensing element but a complete signal processing unit at
the microscale.
It integrates:
• physical detection (Section 3.2),
• temporal integration (Section 3.1),
• electrical conversion,
• digital interfacing.
Thus, the architecture of the pixel fundamentally determines the quality and usability of the
acquired image data.
3.4 Acquisition Modes
Image acquisition modes define how spatial information from the projected image
(formed by the optical system, Module 4) is sampled by the sensor. The acquisition
strategy is determined by the arrangement of photodiodes and the readout mechanism.
From a system perspective, acquisition modes represent different ways of transforming a
continuous image into discrete data, with implications for:
• spatial resolution,
• temporal resolution,
• sensitivity to motion,
• system complexity.
3.4.1 Point-Based Acquisition
In the simplest configuration, a single photodiode is used to measure the intensity of
radiation at one spatial location.
To acquire a full image:
• the scene must be scanned point-by-point,
• either the object or the sensing system must move.
Applications
• laser profilometry,
• LIDAR systems,
• point spectroscopy.
Characteristics
• very high radiometric precision,
• flexible spatial sampling,
• inherently slow acquisition for 2D scenes,
• strong dependence on mechanical scanning.
3.4.2 Line-Scan Acquisition
In line-scan systems, photodiodes are arranged in a one-dimensional array.
• one spatial dimension is captured directly,
• the second dimension is obtained through relative motion between the object and
the sensor.
Image Formation Principle
2D image = 1D sensor + controlled motion
Applications
• conveyor-based inspection systems,
• web inspection (paper, textiles, foil),
• high-resolution industrial measurement.
Characteristics
• very high spatial resolution in the scan direction,
• high sensitivity (larger pixel area per line),
• strict requirement for constant motion speed,
• sensitivity to vibration and timing errors.
3.4.3 Area-Scan Acquisition
Area-scan sensors consist of a two-dimensional matrix of photodiodes, enabling full
image capture in a single exposure.
Image Formation Principle
2D image = 2D sensor (single exposure)
Applications
• general machine vision systems,
• object detection and classification,
• robotics and automation.
Characteristics
• simultaneous capture of the entire scene,
• simple system integration,
• limited frame rate compared to line-scan systems,
• trade-off between spatial resolution and sensor size.
3.4.4 Region of Interest (ROI) Acquisition
Modern CMOS sensors allow selective readout of a region of interest (ROI):
• only a subset of pixels is read,
• reduces data throughput,
• increases effective frame rate.
Engineering Benefits
• faster acquisition,
• reduced computational load,
• lower power consumption.
Limitations
• loss of global spatial context,
• potential need for precise object positioning.
3.4.5 Temporal Aspects of Acquisition
Acquisition modes are inherently linked to temporal sampling:
• point and line-scan systems rely on sequential acquisition,
• area-scan systems rely on simultaneous acquisition (per frame).
This leads to different sensitivities to:
• object motion,
• illumination stability,
• synchronisation errors.
3.4.6 Engineering Trade-Offs
Mode Strengths Limitations
Point High precision, flexibility Very slow for imaging
Line-scan High resolution, high sensitivity Requires controlled motion
Area-scan Simple, robust, fast integration Limited resolution vs data rate
3.4.7 System-Level Insight
The choice of acquisition mode must be aligned with:
• object dynamics (static vs moving),
• required spatial resolution,
• system throughput requirements,
• mechanical constraints.
Acquisition mode defines how spatial and temporal information are coupled in the imaging
system.
This concept becomes essential in the following sections, particularly in:
• shutter mechanisms (Section 3.5),
• spectral imaging (Section 3.8+),
• motion artefact analysis.
3.5 Shutter Mechanisms
The shutter mechanism defines the temporal window during which each pixel collects
charge. It determines how the exposure is initiated and terminated across the sensor array.
From a system perspective, the shutter introduces a temporal sampling structure, which
directly influences:
• motion artefacts,
• geometric fidelity,
• radiometric consistency,
• synchronisation with illumination.
3.5.1 Exposure as Temporal Integration
As introduced in Section 3.1, each pixel integrates incident radiation over a finite time
interval 𝑇:
𝑇
𝑄 = ∫ 𝜂(𝜆) Φ(𝑡) 𝑑𝑡
0
The shutter mechanism defines:
• when the integration starts,
• when it ends,
• whether all pixels integrate simultaneously or sequentially.
3.5.2 Rolling Shutter
In rolling shutter operation, exposure is performed sequentially across the sensor,
typically row by row.
Principle
• each row starts and ends exposure at a slightly different time,
• there is a temporal offset between rows.
Consequences
For static scenes:
• no visible distortion.
For dynamic scenes:
• geometric distortion occurs.
Typical effects include:
• skew (tilted objects),
• stretching or compression,
• temporal misalignment across the image.
Mathematical Insight
If an object moves with velocity 𝑣, and rows are exposed with time delay Δ𝑡, the spatial
displacement between rows is:
Δ𝑥 = 𝑣 ⋅ Δ𝑡
This leads to row-dependent geometric deformation.
Advantages
• simpler sensor design,
• lower cost,
• lower power consumption.
Limitations
• unsuitable for fast motion,
• sensitive to vibration,
• problematic for precise measurement tasks.
3.5.3 Global Shutter
In global shutter operation, all pixels start and end exposure simultaneously.
Principle
• charge is accumulated in all pixels during the same time interval,
• readout is performed after exposure is complete.
Consequences
• no motion-induced geometric distortion,
• consistent temporal sampling across the image.
Advantages
• accurate imaging of moving objects,
• suitability for metrology and high-speed inspection,
• compatibility with pulsed illumination.
Limitations
• more complex pixel architecture,
• reduced fill factor (due to additional circuitry),
• higher cost.
3.5.4 Global Reset (Hybrid Mode)
Some CMOS sensors implement a global reset shutter:
• all pixels start exposure simultaneously,
• readout is still performed sequentially.
Implication
• partial mitigation of rolling shutter effects,
• still susceptible to illumination inconsistencies during readout.
3.5.5 Interaction with Illumination
Shutter mechanisms must be co-designed with illumination:
• continuous illumination → compatible with both shutter types,
• pulsed (strobe) illumination → often required for rolling shutter to avoid
artefacts.
For rolling shutter systems, synchronised short light pulses can effectively “freeze” motion.
3.5.6 Relation to Acquisition Modes
Shutter behaviour interacts with acquisition modes:
• area-scan + rolling shutter → distortion in moving scenes,
• area-scan + global shutter → accurate snapshot imaging,
• line-scan systems → inherently sequential, similar to rolling acquisition but
controlled by motion.
3.5.7 Engineering Trade-Offs
Shutter Type Advantages Limitations
Rolling Low cost, simple design Motion distortion
Global Accurate motion capture Higher cost, complexity
Global reset Compromise solution Partial artefacts remain
3.5.8 System-Level Insight
The shutter mechanism defines how time is mapped into the image.
Crucially: the captured image is not only a spatial representation, but also a temporally
structured measurement.
Thus, shutter selection must consider:
• object velocity,
• required geometric accuracy,
• illumination strategy,
• synchronisation constraints.
3.6 Quantum Efficiency
Quantum efficiency (QE) is a fundamental parameter describing the ability of an image
sensor to convert incident photons into measurable electrical charge.
It establishes the direct link between optical input and electrical signal generation.
3.6.1 Definition
Quantum efficiency is defined as:
𝑁𝑒 −
𝜂(𝜆) =
𝑁𝑝ℎ𝑜𝑡𝑜𝑛
where:
• 𝜂(𝜆) quantum efficiency (wavelength-dependent),
• 𝑁𝑒 − number of generated and collected electrons,
• 𝑁𝑝ℎ𝑜𝑡𝑜𝑛 number of incident photons.
3.6.2 Physical Interpretation
Quantum efficiency expresses the probability that an incident photon:
1. is absorbed in the semiconductor,
2. generates an electron–hole pair,
3. contributes to the measured signal.
Thus:
𝜂(𝜆) ≤ 1
In practice, 𝜂(𝜆) < 1 due to reflection, transmission, and recombination losses.
3.6.3 Wavelength Dependence
Quantum efficiency is strongly dependent on wavelength due to material absorption
properties.
Key behaviour:
• Short wavelengths (UV)
o absorbed near the surface,
o high recombination losses → reduced QE.
• Visible range
o optimal absorption in silicon,
o highest QE for standard CMOS sensors.
• Long wavelengths (NIR)
o deeper penetration into the material,
o reduced absorption efficiency → decreasing QE.
3.6.4 Influence of Sensor Design
Quantum efficiency is not fixed; it depends strongly on sensor architecture.
Microlenses
• concentrate incoming radiation onto the photodiode,
• increase effective fill factor,
• improve QE, especially for small pixels.
Front-Illuminated Sensors
• light passes through metal interconnects before reaching the photodiode,
• partial blockage and reflection reduce QE.
Back-Illuminated Sensors
• sensor is inverted so that light directly reaches the photodiode,
• eliminates obstruction by circuitry.
Advantages of back-illumination:
• higher QE,
• better low-light performance,
• improved spectral response (especially in NIR).
Trade-offs of back-illumination:
• higher production cost,
• increased dark noise.
3.6.5 Relation to Signal Formation
From Section 3.1:
𝑄 = 𝜂(𝜆) ⋅ 𝑁photon
Thus, QE directly scales the generated signal.
Implication:
• higher QE → stronger signal → better signal-to-noise ratio (SNR).
3.6.6 QE and Noise
QE influences the dominant noise mechanisms:
• Shot noise (photon noise):
𝜎 ∝ √𝑁photon
Higher QE increases signal faster than noise → improves SNR.
However, increased sensitivity may also amplify unwanted signals (e.g., background
radiation).
3.6.7 Spectral System Perspective
QE must be considered together with:
• illumination spectrum 𝐿(𝜆)(Module 3),
• object reflectance 𝑅(𝜆)(Module 2),
• optical transmission 𝑇(𝜆)(Module 4).
Thus, the effective detected signal is:
𝑄 = ∫ 𝐿(𝜆) 𝑅(𝜆) 𝑇(𝜆) 𝜂(𝜆) 𝑑𝜆
3.6.8 Engineering Trade-Offs
Optimising QE involves balancing:
• sensitivity vs noise,
• spectral range vs efficiency,
• pixel size vs fill factor,
• cost vs performance.
3.6.9 Practical Implications
High QE is critical in:
• low-light imaging,
• high-speed imaging (short exposure),
• spectral imaging applications,
• precision measurement systems.
3.6.10 System-Level Insight
Quantum efficiency is not merely a sensor parameter. It determines how effectively the
entire imaging system converts optical information into usable data.
Therefore, QE must be co-optimised with:
• illumination design (Module 3),
• spectral filtering (Module 2),
• optical transmission (Module 4),
• acquisition strategy (Section 3.4–3.5).
3.7 Spectral Imaging
Spectral imaging extends conventional intensity-based imaging by capturing information
across multiple wavelength bands. Instead of measuring only total radiant intensity, the
system records spectrally resolved information, enabling analysis of material properties.
3.7.1 Conceptual Principle
As established in previous modules, the detected signal depends on wavelength:
𝑄 = ∫ 𝐿(𝜆) 𝑅(𝜆) 𝑇(𝜆) 𝜂(𝜆) 𝑑𝜆
Conventional imaging integrates this expression over the entire spectral range, producing a
single intensity value per pixel.
Spectral imaging, in contrast, decomposes the signal into multiple wavelength-dependent
components.
3.7.2 Types of Spectral Imaging
Spectral imaging systems are classified according to the number of spectral bands:
RGB Imaging
• three spectral bands (red, green, blue),
• approximates human colour perception,
• limited spectral discrimination.
Multispectral Imaging
• typically 4–15 spectral bands,
• selected wavelength ranges based on application,
• enables material classification and enhanced contrast.
Hyperspectral Imaging
• tens to hundreds of narrow spectral bands,
• continuous spectral representation,
• allows detailed identification of chemical composition.
3.7.3 Spectral Signatures
Each material exhibits a characteristic spectral response:
𝑅 = 𝑅(𝜆)
This spectral signature enables:
• material identification,
• defect detection,
• classification tasks.
Thus, spectral imaging transforms the imaging problem from geometric analysis to spectral
analysis.
3.7.4 Spectral Resolution vs Spatial Resolution
Spectral imaging introduces a fundamental trade-off:
• increasing spectral resolution → more wavelength bands,
• decreasing spatial resolution (due to sensor or acquisition constraints).
This trade-off is particularly evident in:
• snapshot spectral systems (pixel-level filtering),
• hyperspectral imaging.
3.7.5 Data Representation
Spectral images are represented as a data cube:
𝐼(𝑥, 𝑦, 𝜆)
with:
• two spatial dimensions 𝑥, 𝑦,
• one spectral dimension 𝜆.
This representation enables advanced analysis techniques such as:
• spectral clustering,
• dimensionality reduction,
• classification algorithms.
3.7.6 Acquisition Strategies
Spectral information can be obtained using several methods:
[Link] Sequential Acquisition (Time-Multiplexed)
• spectral bands captured one after another,
• implemented using:
o filter wheels,
o tunable filters,
o switched illumination.
Advantages:
• high spectral flexibility,
• full spatial resolution.
Limitations:
• sensitive to motion,
• longer acquisition time.
[Link] Parallel Acquisition (Spatially Separated)
• multiple sensors or optical paths capture different spectral bands simultaneously,
• implemented using:
o beam splitters,
o dichroic prisms,
o multi-camera systems.
Advantages:
• simultaneous capture,
• high radiometric stability.
Limitations:
• increased system complexity,
• need for geometric registration.
[Link] Snapshot Spectral Imaging (Pixel-Level Multiplexing)
• spectral filters integrated at pixel level,
• each pixel measures a different spectral band.
Advantages:
• single-shot acquisition,
• robust to motion.
Limitations:
• reduced spatial resolution per band,
• reduced sensitivity.
[Link] Line-Scan Spectral Imaging
• combines spectral decomposition with line scanning,
• uses motion to build the second spatial dimension.
Advantages:
• high spectral and spatial resolution,
• high sensitivity.
Limitations:
• strong dependence on motion stability,
• complex calibration requirements.
3.7.7 Engineering Trade-Offs
Spectral imaging requires balancing:
• spectral resolution vs spatial resolution,
• acquisition speed vs data volume,
• sensitivity vs noise,
• system complexity vs robustness.
3.7.8 Practical Applications
Spectral imaging is used in:
• food quality inspection,
• pharmaceutical analysis,
• material sorting and recycling,
• agricultural monitoring,
• biomedical imaging.
3.7.9 System-Level Insight
Spectral imaging extends the classical imaging pipeline:
• Module 2 → spectral interaction (material properties),
• Module 3 → illumination spectrum,
• Module 4 → optical transmission,
• Module 5 → sensor response.
Thus, spectral imaging represents a fully integrated interaction between physics, optics, and
sensing.