Problem Formulation and Notation: Let the input video sequence be denoted as
CNN Architecture and Layer-wise Operations: The proposed CNN adopts a lightweight encoder– Temporal Adaptation Mechanism: To enable long-term adaptability, the background model is
decoder structure with depth wise separable convolutions to minimize computational complexity. Let updated using an exponential moving average:
F(l) denote the feature map at layer l. The transformation between layers is defined as:
Computational Complexity Analysis: Let NNN be the total number of CNN parameters and PPP the
number of multiply-accumulate operations. The lightweight CNN ensures:
Background Estimation Function: The final decoder layer produces the estimated background:
Role in the Hybrid Framework: The CNN-based background model provides a high-quality estimate
Bt=F(L) that captures semantic scene information, while deferring pixel-level subtraction and refinement to the
where L denotes the total number of CNN layers. VLSI accelerator. This separation allows the system to preserve adaptability without incurring the high
This background representation captures static and slowly varying components of the scene while energy cost of full CNN-based segmentation.
suppressing transient foreground objects. 3.3 CNN-Based Adaptive Background Modeling
Loss Function and Optimization: During training, the CNN parameters θ are optimized to minimize Network Design: The CNN is designed to be lightweight and edge-friendly, incorporating:
the reconstruction error between the predicted background Bt and the ground truth background BtGT
• Depthwise separable convolutions
• Reduced channel width
• No fully connected layers
• Online background adaptation
Input: Current frame It ; Output: Estimated background frame Bt
CNN Mathematical Formulation:For each convolutional layer l:
Background Learning on CNN (Software Domain): Task: Adaptive background estimation
Execution Platform: Edge CPU / NPU
3.4 VLSI-Based Background Subtraction Engine The background learning task requires nonlinear feature extraction, spatial context awareness, and
temporal adaptability. These operations are inherently data-dependent and benefit from programmable
Absolute Difference Computation: For each pixel (x,y) execution. Therefore, they are implemented using a lightweight CNN running on an embedded CPU or
NPU.
Dt(x,y)=∣It(x,y)−Bt(x,y)
Let:
This operation is implemented using parallel subtractor units in hardware.
• CCNN be the number of multiply–accumulate (MAC) operations required for one CNN
Adaptive Thresholding: Foreground decision:
inference,
• K be the background update interval (in frames).
The average per-frame computational cost of background learning is:
Morphological Refinement: To remove noise and fill holes, the VLSI module applies:
By updating the background model intermittently rather than every frame, the CNN computation load
is significantly reduced while maintaining adaptability.
Pixel Subtraction on VLSI Accelerator (Hardware Domain)
Task: Absolute pixel difference computation
Execution Platform: Dedicated VLSI accelerator
Pixel subtraction involves computing:
Dt(x,y)=∣It(x,y)−Bt(x,y)
3.5. Hardware–Software Co-Design Strategy for all H×W pixels. This operation is repetitive, deterministic, and highly parallelizable. Implementing
it in hardware enables the use of:
The proposed Hybrid CNN–VLSI framework adopts a hardware–software co-design strategy to
efficiently map different computational tasks to the most appropriate execution platforms. This strategy • Parallel subtractor arrays
is motivated by the heterogeneity of background subtraction workloads, which include both learning-
• Fixed-point arithmetic
intensive operations requiring flexibility and pixel-level arithmetic operations demanding high
throughput and energy efficiency. By carefully partitioning these tasks, the system achieves substantial • Deep pipelining
reductions in computation load, memory bandwidth usage, and overall energy consumption.
Let:
Workload Characterization and Task Classification: Let the total background subtraction workload
• N=H×WN = H \times WN=H×W be the total number of pixels,
per frame be defined as:
• Fclk be the accelerator clock frequency.
The throughput of the VLSI accelerator is:
3.6 Algorithm : Hybrid CNN–VLSI Background Subtraction
resulting in substantially lower latency and energy per operation.
Thresholding and Morphological Operations in Hardware
Tasks: Foreground classification and refinement
Execution Platform: VLSI accelerator
Thresholding and morphological operations such as erosion and dilation are formulated as local
neighborhood operations:
where S is a structuring element.
These operations require simple comparisons and logical operations, making them ideal candidates for
hardware acceleration. Implementing them in VLSI avoids expensive memory accesses and branching
overheads typically incurred in software implementations.
Memory Bandwidth Reduction Analysis: Let:
• MCNN be memory accesses required for full CNN-based segmentation,
• Mhybrid be memory accesses in the proposed hybrid system.
Since only background estimation is handled by the CNN and pixel processing is performed locally in
hardware:
Mhybrid≪MCNN
Shared memory buffers further eliminate redundant data transfers between software and hardware
modules, reducing off-chip memory traffic—a major contributor to power consumption.
Energy Consumption Model: The total energy consumption per frame is given by:
Etotal=ECNN+EVLSI
where:
This results in substantial energy savings, enabling continuous operation on battery-powered edge
devices.