1
Understanding AI, Part 2: Inference
By Sylvia He
(Written in 2021 for Mouser)
Machine Inferencing
Inferencing is the second phase of machine learning, following the training phase. During the
training phase, the algorithm generates a new model or repurposes a pre-trained model for a
specific application and helps the model learn its parameters. During the inferencing phase,
predictions and decisions on new data are made based on the learned parameters.
Learning requires a significant amount of time, computation power, and electricity. In contrast,
the inferencing phase requires less processing and electric power. However, as mentioned in
"Part 1: AI Training", the traditional way of computing in the central cloud may be too resource-
intense for IoT devices as each IoT device collects large datasets, which will make the data
transfer back and forth between the edge (where the IoT device is) and the central cloud
expensive and slow.
Computing at the edge will reduce the cost and time lag of data transfer, but this model also
has downsides. For example, the need for IoT devices for power efficiency runs contrary to its
need for high processing power during learning and inferencing. Therefore, there is a need to
develop accelerators for AI edge computing.
AI Accelerators
An AI accelerator is a hardware or software that expedites machine learning.
Hardware acceleration can target training, inferencing, or both. In some instances, the
hardware can reduce the power requirement; in other cases, the hardware can improve the
processing capacity. For hardware acceleration, there are several main types of chips (or
processing units), including central processing units, graphics processing units, field-
programmable gate arrays, systems on a chip, application-specific integrated circuits, vision
processing units, and neuromorphic chips. In addition to hardware accelerations, software
such as machine learning frameworks also help improve AI software development and
optimize system performance.
1|Page
2
CPUs and GPUs
Central processing units (CPUs)s are what AI traditionally uses. While CPUs are designed to
be all-purpose, they are often inadequate in supporting the massive calculations in model
generation and inferencing. In response to the need, some companies have started making AI-
specific chips, such as ARM (DynamIQ chips) and Samsung (Exynos 9 series).
While ARM and Samsung stay with AI-specific CPUs, others are shifting towards GPUs.
Originated from the video gaming industry and built for processing massive datasets, Graphics
Processing Units (GPU) are a good match for machine learning. Because GPUs have more
processing units per chip, higher throughput, and greater parallel processing capability than
CPUs, they will significantly cut the computation time. In addition, a GPU's single processing
unit weight is less than the CPU's, making it perfect for IoT devices, which require small and
nimble components. The companies that are making AI-specific GPUs include AMD (Radeon
Instinct chips), NEC (SX-Aurora), NVIDIA (DGX), and Qualcomm (Adreno).
FPGAs
While CPUs and GPUs have considerable processing power and are effective for accelerating
learning and inferencing, they spend a lot of time and energy moving data between memory to
processing. In addition, since CPUs and GPUs are densely packed with circuits, they can often
overheat and cause system failures. For remotely located IoT devices, the combination of high
energy consumption and system failures is not ideal. Therefore, it will make sense to find a
way to offload some tasks to another hardware that is more energy-efficient.
Field-programmable gate arrays are a type of integrated circuit that can be reconfigured by
customers or designers in the field (hence "field-programmable") after manufacturing
production. While generally not as powerful as CPUs or GPUs, FPGAs offer fast processing for
some calculations such as multiplication, addition, integration, differentiation, and exponentials
by computing inside the chip instead of transferring data.
Although FPGA offers more flexibility, it has an ecosystem that is less mature than that of
CPUs or GPUs. Also, FPGAs tend to be quite bulky; therefore, miniaturization for IoT devices
will be a challenge for this type of chips.
The major companies that offer AI-targeted inference chips include NVIDIA (TensorRTTM) and
Xilinx. Microsoft is also using FPGA chips to accelerate inference. Intel is trying to expand in
FPGA through its acquisition of Altera.
System on a Chip
A System on a Chip (SoC) is a silicon chip whose design contains any combination of
electronic components of a system: microprocessors, microcontrollers, digital signal
processors, on-chip memory, hardware accelerator, as well as other accessories. Due to the
2|Page
3
integration of the components, SoC is more powerful than a microcontroller chip. For example,
the SoC may have integrated video, audio, and image processing capabilities in a smartphone.
ARM has developed a Machine Learning Processor and the ARM Object Detection Processor,
which will be incorporated into SoCs. HiSilicon, a Huawei-backed company, has licensed the
IP from ARM to make SoCs (K3V2, K3V2E, and the Kirin series) that are seeing preliminary
use in phones and tablets made by Huawei. Also, HiSilicon is making the Ascend chips for
Huawei.
Another big player in the SoC space is Arteris, which is developing a network-on-chip
interconnect fabric technology (FlexNoc) that many mobile and wireless companies are using.
Because of Arteris' dominant position in IP, it has a bird's-eye view of the space. According to
an interview with Arteris, other companies making a play SoC market include Intel (via its
subsidiary Altera and Movidius), NXP, Renesas, Toshiba, Texas Instruments,
STMicroelectronics, Samsung, Xiaomi, Achronix, Schneider Electronics, Kyocera, ZTE, Baidu,
LG, Hisense, Mega Chips, VeriSilicon, CASIO, GUC, NTTElectronics, NationalChip, National
Technologies.
ASICs, VPUs, and Neuromorphic Chips
ASIC. A subtype of SoC, application-specific integrated circuits (ASIC), are specifically built for
accelerating the training of deep learning algorithms. Examples of ASIC include Google's Edge
TPU and Intel's Nervana.
VPU. A vision processing unit is specifically designed to accelerate machine vision tasks and
run machine vision algorithms such as convolutional neural networks. VPU is thus different
from GPU's video processing capability. Examples of vision processing units include Intel's
Movidius™ Myriad™ chips, Google's Pixel Visual Core, Microsoft's HoloLens, Inuitive's NU
series, NVIDIA's Programmable Vision Accelerator, and Mobileye's EyeQ.
Neuromorphic Chips. Digital chips and analog chips have their respective deficiencies:
Digital circuitry is precise but is high on energy consumption, while analog circuitry is low on
energy consumption and latency but lacks precision. Therefore, researchers are looking for
ways to combine the technical advantages of both digital and analog chips while sidestepping
the weaknesses.
Inspired by the human analog brain, neuromorphic chips are designed to follow the digital
architecture but use analog circuitry for mixed-signal processing. For example, IBM's
TrueNorth is a neuromorphic processor targeting sensor data pattern recognition and
intelligence tasks. Also, Columbia University, Stanford University's Brains in Silicon project, the
DARPA-backed University of Michigan IC Lab, and the U.S. Office of Naval Research are
working on various aspects of neuromorphic systems.
3|Page
4
Machine Learning Frameworks
AI accelerators also include software. For example, Machine Learning Frameworks, which can
be interfaces, libraries or tools, help reduce the complexity of machine learning so that
developers can build models and optimize performance more quickly and easily.
ML frameworks are built into specific languages, such as Python and Java. Some of the most popular
open-source ML frameworks include offerings from Amazon (AWS), Apache, Caffe2, Keras, Theano,
Microsoft (Azure), Scikit-Learn, and Google (TensorFlow). Also, some companies offer their in-
house platform. For example, Intel's Open Vino tool kit is a software and hardware accelerator
that optimizes inference with Convolution Neural Network models. In addition, Qualcomm's
Snapdragon is a mobile platform and a software accelerator, IBM has its Watson Machine
Learning Accelerator platform, and Huawei has recently launched its MindSpore AI framework.
Looking to the Future
Meanwhile, since NVIDIA remains dominant in the industrial application space, most
newcomers are in the IoT AI space.
GreenWave and Reduced Energy Microsystems are in the space of low-power chips, while
Mythic and Syntiant are developing battery-powered processors. Similarly, Wiliot is making a
Bluetooth chip that can be powered by ambient radio frequencies. In the massive parallel data
processing space, there are Vathys, Graphcore, Cerebras, and Wave Computing. Meanwhile,
Hailo Technologies and Horizon Robotics are working on specialized chips for autonomous
vehicles. In the deep learning space, BrainChip has made the first Spiking Neural Processor,
ThinCI has rolled out the Streaming Graph Processor, and Gyrfalcon is developing a deep
learning processor with the AI Processing in Memory (APiM) technology. Lastly, at Groq, the
ex-Googlers who designed Google's TPU are developing a chip with ultra-low latency.
As the field of machine learning makes astonishing progress, many technical challenges
remain for IoT edge computing, and hardware and software developers will continue to work
towards a balance between processing performance and energy efficiency.
4|Page