In the previous lesson, we said that the host uses your plugin in a
prepareToPlay() - processBlock() - releaseResources() cycle. We noted
that the host calls your plugin processor's processBlock() method, the audio
callback, many times per second to prompt you to process audio data. We also
emphasized that your plugin should complete processing a block of samples in a time
significantly shorter than the block itself represents. In this lesson, we will explain why
it's the case and how to meet this deadline.
Let’s start by considering where the audio callback comes from. How does it happen
that the host calls your processBlock() method at all?
During audio playback, a hardware device transports the audio signal to the
speakers. This hardware device is typically a sound card or an audio interface
containing a DAC. This hardware device requests a fresh portion of audio samples at
regular time intervals. Each such request is termed an audio callback, identically as
we named the processBlock() method. Before this request reaches your plugin,
however, there are several layers it must go through:
● the lowest software level: the audio device driver,
● the operating system kernel,
● the audio driver API available on the system,
● the host application, for example, a DAW, and
● the audio plugin format glue code.
How this works in detail is not important. However, you can immediately see some
interesting implications of this setup.
● Going through all these layers takes time.
● The shorter the portion of the audio requested, the shorter the latency
between a user action and its audible impact.
● The longer the portion of frames requested by the audio device, the fewer
times we have to go through all these layers, leaving us more time for
processing.
● This processing does not involve any GUI rendering or user input handling; it
happens on a dedicated audio thread, which is a high-priority system thread.
All communication with the audio thread must obey the rules of thread-safe
programming. We will discuss this in more detail shortly.
But the key takeaway is this: during audio playback, the host must output samples in
blocks at a constant rate. This rate is fixed, and so the block size is fixed too. Each
block represents a portion of sound to play back. Because the block size and the
sample rate are fixed, a block has a fixed length in seconds. To keep the audio
stream continuous, each block to be output must be computed in a time shorter than
the time this block represents.
When the host takes too much time to process a block, the audio hardware may run
out of samples to play. This results in an unpleasant glitch in the audio output, an
audio dropout.
In the pro audio world, a glitch is a no-go. If an audio app is glitching under normal
system load, it is basically unusable if not dangerous; in a live concert, for example,
or when wearing headphones, a glitching music app can easily cause hearing
impairment and damage equipment.
Ok, we know that audio processing is carried out on the audio thread, and to
complete processing within a deadline, the host must provide the samples within a
specific interval. But who determines this interval?
Depending on the DAW, the user can choose a sampling rate and a buffer size either
in the program itself or in the operating system's settings. If the user does not set
them explicitly, reasonable defaults are used.
The shorter the buffer, the smaller the latency between user action and its audible
effect. However, the shorter the latency, the less time there is for the plugins to
complete their processing. This is a typical engineering trade-off between latency and
performance.
For example, suppose the user has a guitar plugged into an audio interface, wants to
apply a distortion plugin to it, and hear the result the moment they play a note. In that
case, it’s unacceptable for them to hear the result after, say, 20 milliseconds. When
playing a guitar, quick feedback is crucial for playing evenly, making decisions, and
correcting on the fly. If the user would like to play alongside a backing track, a lack of
real-time sound would make it very difficult to hit the strings at the right time. They
would not be able to focus on playing itself. That would effectively prohibit them from
using this effect plugin for their guitar practice. For this setup to work, they probably
need a total latency of less than 8 milliseconds. Consider that we have a long
processing chain here: analog-to-digital conversion at the input, system audio
callback, host audio callback, plugin callback, and then all the way back to the DAC
at the output, and finally to the speakers or headphones, which may have their own
internal analog or digital processors. An 8-millisecond round-trip latency means that
the buffer size must be less than 5 milliseconds. Which means your plugin should
complete its processing of a 5-millisecond audio signal in much less than 5
milliseconds.
How does the plugin know its deadline? Well, the plugin knows the sampling rate at
which it operates; it receives this information in prepareToPlay(). In
processBlock(), it obtains a buffer of samples. The length of this buffer, converted
to seconds or milliseconds, yields the processing deadline. For example, at 48 kHz, a
buffer with 240 frames corresponds to around 5 milliseconds. Thus, your
processBlock() should take much less than 5 milliseconds to process 240 frames
of audio at 48 kHz.
Why do I keep saying "much less than 5 milliseconds"? Cannot our processing take
up all the time between subsequent calls to processBlock()? Well, the host uses
more than one plugin for audio processing. Typical DAW projects have dozens, if not
hundreds, of plugins loaded at the same time. They are all processed in order on the
audio thread. That means that your plugin won’t have the luxury of 5 milliseconds to
process 240 samples. Instead, it must complete its processing in a fraction of that
time. The shorter the processing time for particular plugins, the more plugins the user
can load into their session without glitching.
If your plugin causes the host to miss the audio deadline, it will cause an audible
glitch. This will negatively impact the user experience and, consequently, the
commercial success of your plugin. Plugins that cause glitches under normal
processing conditions simply won't be used.
How can we ensure that our plugin meets the audio deadline every time? Clearly,
what we put in the processBlock() method determines whether we succeed or fail
in fulfilling this requirement. Indeed, there is a list of DOs and DON'Ts that we should
follow in the processBlock() implementation. This list, unfortunately, is neither
intuitive nor easy to discover on one's own. That's why we'll discuss the constraints of
the code within processBlock() later on in this lesson.
To recap what we learned so far, the audio device calls the plugin host at regular
intervals. It does so on the audio thread: a system thread of execution dedicated only
to processing audio. The host should serve audio samples to play within the interval
period. If it fails, we get a glitch. The host invokes all audio plugins in order to process
sound. Thus, the more plugins in a DAW session, the less time each plugin has to
complete processing. The inverse is also true: the faster the plugins process the
audio, the more plugins a user can load.
The remainder of this lesson is focused on meeting the audio deadline on each call to
processBlock(). We will discuss the specific rules we should follow to ensure our
plugins always meet this deadline.
Let's start with a clarification. Although you can often hear that audio plugins should
be “fast”, that’s not entirely true. Audio processing must complete within a fixed
interval. Thus, audio plugins should be predictably fast.
Let me explain what I mean by an example.
Modern processors typically have between 8 and 16 cores, which results in between
8 and 32 available hardware-supported threads.
If you decided to process audio using multithreading techniques to utilize all of these
threads instead of one, you would probably get a considerable speedup. Let’s say
your plugin is capable of processing an hour of audio material in 20 minutes, i.e., 3
times real time. If you used 16 threads, the processing time of an hour would drop to
less than 2 minutes. That gives us 48 times real-time speed; neat, right? On average,
your multithreaded plugin would be incredibly fast.
However, let’s think about what would happen in a single processBlock() call.
There, you would need to wait for the worker threads to complete their processing.
How long would you need to wait? Because the operating system must keep other
tasks and applications, like a web browser, responsive, you cannot reliably predict the
wait time. It is non-deterministic. We can perform measurements and obtain an
average, but we are not interested in the average: we are interested in the worst-case
scenario. Your plugin will trigger a glitch if it misses just one audio deadline, so it’s
essential to meet the deadline on every call to processBlock(). And because
waiting for another thread on commercial operating systems has unbounded
completion time, we generally cannot use other threads to process incoming audio in
the audio callback.
Ensuring that the processing deadline is always met is termed real-time
programming. Since no consumer operating system gives real-time guarantees,
audio plugins that run in those systems cannot provide those guarantees either.
Thus, they are often referred to as near-real-time, rather than true real-time.
However, all real-time programming techniques still apply.
This deadline severely restricts what we can do on the audio thread. We can only do
what is guaranteed to complete before the audio deadline. This has three
implications:
1. We cannot do anything that, in the worst case, takes longer to complete than
the buffer interval.
2. We cannot perform any operation that has an unbounded execution time.
3. We cannot perform any operation with an unknown execution time, as it may
fall into one of the above categories.
That sounds simple on the outset, but it’s actually amazing how little you are allowed
to do on the audio thread. We cannot possibly list all things that are allowed and
disallowed, but we can outline a few categories of problems that come up quite often.
I’ll also mention typical solutions to them.
So let’s consider: what you, in general, should not do on the audio thread?
First and foremost, don’t allocate or deallocate memory on the audio thread. That
means avoiding C++ new and delete operators, C malloc() and free() family of
functions, C++ make_unique<>() and make_shared<>, etc. Also, beware of
calling code that may allocate. For example, copying a std::vector<> or calling
push_back() on a std::vector<> is likely to allocate. An innocent little copy
assignment operation may cause your audio to glitch.
Why is allocation and deallocation prohibited? Because memory allocation is an
operating system call that is not guaranteed to be real-time safe on commercial
operating systems.
The solution to the allocation problem is to put all allocations into your processor’s
constructor or prepareToPlay(). If you need to deallocate memory manually, you
should do it in releaseResources() or the destructor of your processor.
Second, don’t take a lock on the audio thread. Locking and unlocking a mutex are
system calls that are not guaranteed to be real-time safe. There are exceptions to
this, but they occur so rarely that we won’t discuss them here. Additionally, waiting to
acquire a mutex is non-deterministic; you don't know in advance how long you will
have to wait before another thread unlocks it.
Instead of using mutexes to access shared data on the audio thread, use lock-free
and wait-free data structures. In C++, for small data types like a float, use
std::atomic<>. However, be aware that for larger data types, std::atomic<>
may use a lock internally; thus, you should always use
static_assert(std::atomic<DataType>::is_always_lock_free);
if you want to share a DataType instance with the audio thread. If a DataType
instance is too large to be lock-free, your code with this static_assert() won't
compile. For such types, things get more complicated: you need to use custom
update mechanisms or lock-free FIFO queues. It is outside the scope of this course
to demonstrate how to implement these, but a few resources are available online.
Third, you should not start a new thread or wait for another thread on the audio
thread. That has an unbounded execution time and, thus, will probably cause a glitch.
There’s no general solution to this problem; if you need to perform some heavy work
that’s not real-time-critical, you should probably do it in a background, low-priority
thread and report the result back to your audio thread in a real-time-safe way. That
also means that most real-time audio processing is inherently single-threaded.
Fourth, don’t ever read from a file or write to a file on the audio thread, or perform any
other type of I/O operations, like logging or networking. These are all system calls
with unbounded execution time. Needless to say, reading from a hard drive is very
slow compared to the audio processing rate. Since reading from files and writing to
files are valid concerns for audio plugins, especially for sample-based synthesizers,
there are dedicated techniques to handle them.
The solution to this is to perform I/O operations on a different thread and report the
results to the audio thread in a real-time-safe manner; it must be lock-free and
wait-free on the audio thread.
Fifth, don’t use any algorithm with unpredictable or poor worst-case execution time. If
your algorithm occasionally takes longer to complete, your plugin may occasionally
glitch, which is unacceptable.
The solution to this problem is more nuanced: use an algorithm with better
worst-case performance, even if it means using an algorithm that has worse
average-case performance. It is a good rule of thumb to remember: audio
programming doesn’t care about the average-case performance; it only cares about
the worst-case performance. Keep it in mind when benchmarking your audio
processing code.
Sixth and final, don’t call any system functions or any third-party code that’s not
guaranteed to complete within a known, bounded time. Operating system and
third-party code documentation must be clear if a given function is real-time safe. If
it’s not mentioned explicitly, don’t use it. That’s harsh, but getting a glitch is even
harsher.
As a solution, look for real-time-safe ways to complete the desired operation or
perform it on a separate thread and report the result back to the audio thread, as we
already discussed.
To summarize, audio processing in plugins is performed on the audio thread, which is
a high-priority system thread. It is subject to near-real-time constraints. Not only is
your processBlock() implementation required to complete within a time shorter
than represented by the processed audio buffer, but it must also do so on every call.
Otherwise, your plugin will glitch, which can have disastrous consequences. That
means, you should, in general, not do anything that has too long or unbounded
execution time. In particular, you should not
1. allocate or deallocate memory,
2. take locks,
3. start a thread or wait for another thread,
4. perform any I/O operations, like reading from a file or networking,
5. use any algorithm with unpredictable or poor worst-case execution time,
6. call any operating system functions or any third-party code that’s not
guaranteed to be real-time safe.
As with any rules, there are exceptions and subtleties to these. However, if you follow
them in your audio processing code, it should be real-time-safe and, thus, not glitch.
In the following lessons, you will see how to implement audio processing in your
plugin based on our tremolo plugin.
References
1. Timur Doumler, "Audio in standard C++" ACCU 2019
[Link]
erence
(accessed September 18, 2025)
2. Ross Bencina, "Real-time 101: Time waits for nothing."
[Link]
s-for-nothing
(accessed September 18, 2025)
3. Fabian Renn-Giles, Dave Rowland "Real-time 101" Audio Developer
Conference 2019.
[Link]
(accessed September 18, 2025)