MQTT Essentials

The Ultimate Guide to the MQTT

Protocol for IoT Messaging
VERSION 2.0
 MQTT Essentials Ebook

Abstract
protocols cannot address with features such as persistent
In the rapidly evolving landscape of IoT, MQTT has emerged sessions, retained messages, Last Will and Testament (LWT),
as the de facto standard protocol for data exchange. MQTT Quality of Service (QoS) levels, and more. After reading this
Essentials is designed to equip decision-makers, solution guide, you’ll be ready to use MQTT to optimize connectivity
architects, and IoT professionals with a strategic and practical and lay the proper data foundation to enable any IoT or IIoT use
understanding of MQTT and how to execute it for scalable, case.
reliable, and seamless data movement. Delve into how MQTT
can help your organization overcome the challenges other IoT

TABLE OF CONTENTS

Abstract................................................................................. 2 Chapter 19: MQTT Payload Format Description and

Chapter 1: Introduction to MQTT............................................ 3 Content Type ........................................................................ 65
Chapter 2: Mastering the Basics of MQTT............................... 7 Chapter 20: MQTT Request-Response Pattern ....................... 66
Chapter 3: MQTT Topics, Subscriptions, Chapter 21: MQTT Topic Alias ............................................... 69
QoS, and Persistent Messaging.............................................. 14 Chapter 22: Enhanced Authentication in MQTT ...................... 70
Chapter 4: MQTT Publish/Subscribe Architecture Chapter 23: MQTT Flow Control ............................................ 73
(Pub/Sub) ............................................................................. 16 Chapter 24: MQTT Topic Tree & Topic Matching: Challenges and
Chapter 5: MQTT Client and MQTT Broker Connection Best Practices Explained........................................................ 74
Establishment ....................................................................... 21 Chapter 25: Additional Reading for Mastering MQTT............... 76
Chapter 6: MQTT Publish, MQTT Chapter 26: Next Steps – Choosing the Right MQTT Broker..... 87
Subscribe & Unsubscribe ....................................................... 24
Chapter 7: MQTT Topics and Wildcards ................................. 30
Chapter 8: MQTT Quality of Service (QoS) 0,1, & 2.................. 34
Chapter 9: MQTT Persistent Sessions and Clean Sessions ..... 37
Chapter 10: MQTT Retained Messages .................................. 39
Chapter 11. MQTT Last Will and Testament (LWT).................. 41
Chapter 12: MQTT Keep Alive and Client Take-Over................ 43
Chapter 13: Introduction to MQTT 5 Protocol ......................... 46
Chapter 14: Key Reasons to Upgrade to MQTT 5 from
MQTT 3.1.1 ........................................................................... 51
Chapter 15: MQTT Session Expiry and Message
Expiry Intervals ..................................................................... 53
Chapter 16: MQTT 5’s Improved Client
Feedback & Negative ACKs .................................................... 57
Chapter 17: MQTT User Properties......................................... 59
Chapter 18: MQTT Shared Subscriptions ............................... 62

2
Chapter 1: Introduction to MQTT

MQTT is a lightweight messaging protocol originally designed for communication in constrained networks with limited bandwidth
and compute resources. Developed with simplicity and scalability in mind, MQTT is particularly well-suited for Internet of Things
(IoT) applications where the variety and quantity of devices are growing exponentially.

Below is the official description of the specification:

“MQTT is a Client Server publish/subscribe messaging transport protocol. It is lightweight,

open, simple, and designed so as to be easy to implement. These characteristics make
it ideal for use in many situations, including constrained environments such as for
communication in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a
small code footprint is required and/or network bandwidth is at a premium."

Citation from the official MQTT 3.1.1 specification

[Link]

 MQTT Essentials Ebook

MQTT Publish/Subscribe Architecture

MQTT uses a binary message format for communication between clients and servers (brokers). This is in contrast to other
protocols that use text-based formats, such as HTTP or SMTP.

MQTT Fixed Header format example.

Image Source: [Link]

The binary format used by MQTT is designed to reduce bandwidth and compute resources required by clients to
the size of messages and increase the efficiency of exchange data.
communication. By using a binary format, the protocol can 2. Bidirectional communication: MQTT allows devices
minimize the amount of data that needs to be transmitted and to send and receive data from the server, allowing for
reduce the processing power required to interpret messages. bidirectional data exchange with other components in
This makes MQTT well-suited for use in low-bandwidth or the network.
low-power environments, such as IoT devices with limited 3. Scalable: MQTT can scale to support millions of devices
resources. It’s also used in enterprise systems, where real- or “things” in an IoT or IIoT ecosystem.
time data communication is necessary. 4. Reliable message delivery: MQTT specifies different
Quality of Service (QoS) levels to ensure reliable
Another important aspect of the protocol is that MQTT is message delivery.
extremely easy to implement on the client side. Ease of use 5. Message Buffering and Session Resumption: MQTT
was a key concern in the development of MQTT, and this supports persistent sessions between devices and
makes it a perfect fit for constrained devices with limited servers, enhancing message reliability by ensuring
resources. that messages are delivered to clients even after
disconnections.
Benefits of MQTT 6. Security features: MQTT supports TLS encryption for
message confidentiality and authentication protocols for
MQTT offers several key benefits:
client verification.
1. Lightweight and efficient: MQTT minimizes the network

4
Real-World Applications and Use Cases of The Origin and History of MQTT
MQTT: An Overview
In 1999, Andy Stanford-Clark of IBM and Arlen Nipper of
MQTT is used extensively in IoT, Industrial IoT (IIoT), and Arcom (now Cirrus Link) developed the MQTT protocol to
M2M applications. It has been adopted by companies enable minimal battery loss and bandwidth usage when
such as BMW, Air France-KLM, Liberty Global, Mercedes connecting with oil pipelines via satellite. The inventors
Benz, Hytera, Awair, and Matternet, as showcased in specified several requirements for the protocol, including:
HiveMQ’s customer success stories. These companies
have successfully leveraged MQTT in automotive, • Simple implementation
telecommunications, energy, public safety, and connected • Quality of Service data delivery
product domains. • Lightweight and bandwidth-efficient
• Data agnostic
Here are a few examples: • Continuous session awareness

1. Smart homes: MQTT is used to connect various devices

These goals are still at the core of MQTT. However, the
in a smart home, including smart thermostats, light
primary focus of the protocol has changed from proprietary
bulbs, security cameras, and other appliances. This
embedded systems to open Internet of Things (IoT) use
allows users to control their home devices remotely
cases.
using a mobile app.
2. Industrial automation: MQTT is used to connect
Over the next ten years, IBM used the protocol internally until
machines and sensors in factories and other industrial
they released MQTT 3.1 as a royalty-free version in 2010. This
settings. This allows for real-time monitoring and control
shift in focus from proprietary embedded systems to open IoT
of processes, which can improve efficiency and reduce
use cases created confusion about the acronym MQTT.
downtime.
3. Agriculture: MQTT is used in precision agriculture to
monitor soil moisture levels, weather conditions, and While it formerly stood for MQ Telemetry
crop growth. This helps farmers optimize irrigation and Transport, where MQ referred to the MQ
other crop management practices. Series, a product IBM developed to support
4. Healthcare: MQTT is used to connect medical devices MQ telemetry transport, MQTT is no longer
and sensors, such as glucose meters and heart rate an acronym. It is now simply the name of the
monitors, to healthcare providers. This allows for remote protocol.
monitoring of patients, which can improve patient
outcomes and reduce healthcare costs.
When Andy and Arlen created this protocol in 1999, they
5. Transportation: MQTT is used in connected cars and
named it after the IBM product. Although many sources
other transportation systems to enable real-time
label MQTT as a message queue protocol, this is not entirely
tracking and monitoring of vehicles. This can improve
accurate. While it is possible to queue messages in certain
safety and help optimize traffic flow.
cases, MQTT is not a traditional message queuing solution.

Now that we have a general understanding of what MQTT is

In 2011, IBM contributed MQTT client implementations to
and its characteristics, let’s dive into its history and how it
the newly founded Paho project of the Eclipse Foundation,
came to be a popular messaging protocol. We will explore
an independent, non-profit corporation that provides a
some of the elements and characteristics of MQTT after
community for open-source software projects. This was a
learning about its origins.
significant development for the protocol because it created a

[Link]

 MQTT Essentials Ebook

more supportive ecosystem for MQTT. By contributing MQTT The Role of OASIS in Standardizing MQTT
client implementations to an open-source project like Paho,
In 2014, OASIS announced that it would take over the
IBM allowed developers to access the protocol and build
standardization of MQTT, with the goal of making it an open
their applications on top of it. This move helped to increase
and vendor-neutral protocol. Founded in 1993 as a non-pro
the visibility and adoption of MQTT among the developer
fit, OASIS (Organization for the Advancement of Structured
community.
Information Standards) is an international consortium
that develops open standards for the Internet and related
In 2012, HiveMQ became acquainted with MQTT and built the
technologies.
first version of their software that same year. In 2013, HiveMQ
released their software to the public.
It has developed numerous important standards for industries
such as cloud computing, security, and IoT, including AMQP,
SAML, and DocBook. The standardization process took
around one year, and on October 29, 2014, MQTT became an
officially approved OASIS standard.

OASIS’ involvement in MQTT has been critical to its success

as a widely adopted IoT protocol. As a neutral, third-party
organization, OASIS ensures that the protocol is maintained
as an open standard that can be implemented by anyone
without licensing fees or proprietary restrictions.

Additionally, OASIS provides a forum for the community

to come together and collaborate on improvements to the
protocol, which has resulted in the development of MQTT
5, the latest version of the protocol with new features for
improved reliability and scalability.

In March 2019, OASIS ratified the new MQTT 5 specification.

This version introduced new features to MQTT that are
required for IoT applications deployed on cloud platforms,
and cases that require more reliability and error handling to
implement mission-critical messaging.

Different Versions of MQTT: MQTT 5 vs.

MQTT 3
The earlier version of MQTT was 3.1.1 The latest version is
MQTT 5. MQTT 5 has enhancements to improve performance,
increase reliability, and provide greater control over
communication between clients and servers. Some of the
key enhancements include better error reporting, enhanced
Timeline of how MQTT evolved and when HiveMQ released scalability, and improved support for offline message
an earlier version of its MQTT Broker queueing. These improvements ensure that MQTT can handle

6
the ever-increasing demands of modern IoT environments, recipients. MQTT clients publish messages to the broker,
where the number of connected devices and the amount of and other clients subscribe to specific topics to receive
data they generate are growing exponentially. messages. Each MQTT message includes a topic, and clients
subscribe to topics of their interest. The MQTT broker
maintains a subscriber list and uses it to deliver messages to
the relevant clients.
While MQTT 5 presents a substantial update to MQTT
3.1.1, it remains more of an evolution than a revolution,
faithfully retaining all the features that have contributed
to its success. Its lightweightness, push communication,
unique attributes, ease of use, exceptional scalability,
suitability for mobile networks, and decoupling of
communication participants all endure in this latest
version. However, several foundational mechanics have
been added or slightly adjusted, ensuring that while the
new version still has the feel of MQTT, it incorporates An MQTT broker can also buffer messages for disconnected
improvements that reinforce its standing as the most clients, ensuring reliable message delivery even in unreliable
popular Internet of Things protocol to date. network conditions. To achieve this, MQTT supports three
different Quality of Service (QoS) levels for message delivery:
0 (at most once), 1 (at least once), and 2 (exactly once).
The reassuring news for those acquainted with MQTT 3.1.1
is that the fundamental principles and features remain
unchanged in MQTT 5. Despite certain modifications and
There are two versions of the MQTT
expansions, the heart of the MQTT you are familiar with
specification: MQTT 3.1.1 and MQTT 5.
persists in version 5. Noteworthy adjustments include slight
While most commercial MQTT brokers now
changes to pre-existing features such as Last Will and
support MQTT 5, some IoT-managed cloud
Testament and the inclusion of popular HiveMQ features like
services still primarily support MQTT 3.1.1.
TTL and Shared Subscriptions, which you will learn in detail
We highly recommend using MQTT 5 for new
in the chapters ahead.
IoT deployments due to its enhanced features
that focus on robustness and cloud-native
The underlying protocol has experienced minor
scalability.
transformations, and an additional control packet — AUTH
— has been incorporated. However, these changes do
not obscure MQTT’s core identity. In essence, MQTT v5
is still unambiguously MQTT, retaining its recognizable
MQTT Clients
characteristics while enhancing its capabilities.
Many open source MQTT Client Libraries are available
in a variety of programming languages. HiveMQ provides
Chapter 2: Mastering the Basics of
MQTT Client Libraries, which are designed to simplify the
MQTT deployment and implementation of MQTT clients and offer
users top-notch functionality, performance, security, and
At the core of MQTT are MQTT brokers and MQTT clients.
reliability. Some of the programming languages supported
The MQTT broker is an intermediary between senders
include C#, C++, Java, WebSockets, Python, and more.
and receivers, dispatching messages to the appropriate
7

[Link]

 MQTT Essentials Ebook

Eclipse Paho also offers MQTT client libraries for languages started with HiveMQ on-premises MQTT Broker.
like C/C++ and Python. You can find a comprehensive list of • Deploy an MQTT broker on Amazon Web Services (AWS).
MQTT clients at [Link]. Here’s how to get started with HiveMQ on AWS.
• Deploy an MQTT Broker on Microsoft Azure. Here’s how
MQTT Brokers to get started with HiveMQ on Azure.

MQTT Brokers come in various implementations, catering

Step 2: Connect MQTT Clients to Your MQTT Broker
to different needs, such as open-source, commercial, and
managed cloud services. HiveMQ offers two commercial The method an MQTT Client uses to connect to an MQTT
editions: HiveMQ Professional and HiveMQ Enterprise. broker varies based on the broker's setup. Below are a few
HiveMQ also offers HiveMQ Cloud, a managed cloud methods for establishing a connection:
MQTT service, and HiveMQ Community Edition, an open-
source version. In addition, HiveMQ offers an MQTT broker Example 1: Using MQTT CLI for On-Premises MQTT Brokers
embedded in HiveMQ Edge, an open-source industrial device
If you are unaware, the MQTT CLI is an open-source, Java-
connectivity software for IIoT use cases. For an extensive list
based MQTT client tool that enables you to interact quickly
of MQTT brokers, please visit [Link].
and easily with any MQTT broker in various ways. The
MQTT CLI comes in various binary packages that can be
If you are looking to find the ideal MQTT broker for your
downloaded from the documentation homepage on GitHub.
IoT or industry 4.0 use case, read our blog MQTT Broker
Here are the steps:
Comparison – Which is the Best for Your IoT Application?
1. Open a terminal window and go to the tools directory
How to Make an MQTT Broker Communicate of your HiveMQ MQTT Broker. In the tools directory, go
with an MQTT Client? to the mqtt-cli/bin folder and enter mqtt sh to start the
MQTT CLI in shell mode. The MQTT CLI starts and lists
Here’s an overview of how to make an MQTT broker
useful options and commands.
communicate with an MQTT client, such as a sensor, an edge
2. To connect your first MQTT client to your HiveMQ Broker
device, a PLC, etc.:
on localhost and give it a custom identifier, enter con -h
localhost -i testClient1. This command creates the first
Step 1: Install an MQTT Broker of Your Choice
MQTT client with the custom identifier testClient1 and
MQTT brokers are available in commercial, open-source, connects the client to your MQTT broker on localhost.
cloud-managed and general-purpose editions. To get an 3. To connect another MQTT client that you can use to test
MQTT broker to communicate with an MQTT client, first you your installation, open a second terminal window and
need to find an edition of the broker that is right for your use enter mqtt sh (keep your original terminal window open).
case. 4. In the second terminal window, enter con -h localhost
-i testClient2. This command creates a second MQTT
Here are a few examples of how you can install an MQTT client with the customer identifier testClient2 and
broker in different environments: connects the client to your MQTT broker on localhost.
• Use a fully-managed MQTT Broker, like HiveMQ Cloud.
Here’s how to get started with HiveMQ Cloud. Example 2: Using Cloud-Managed MQTT Broker
• Download and install an MQTT broker on a server or Using a cloud-managed MQTT broker such as HiveMQ Cloud,
computer of your choice. Here’s how to get started with you can set up your broker and obtain connection details
the HiveMQ platform. through the console. These details enable MQTT Clients to
• Run an MQTT broker on Docker. Here’s how to get connect to the broker. Here’s an overview of how to connect

8
MQTT Client to HiveMQ Cloud. • Client Initialization: The MQTT client initializes its MQTT
library and sets up the required parameters, such as the
1. Sign up for Serverless FREE HiveMQ Cloud plan.
broker's address and port.
2. Create a free HiveMQ Cloud cluster.
• Connection to the broker: The client establishes a TCP/
3. Create access credentials that your MQTT clients use to
IP connection with the MQTT broker. The default port for
connect to the cluster.
MQTT is 1883 (or 8883 for encrypted connections using
4. Copy the Cluster url, port number, and access
TLS/SSL).
credentials.
• Handshake: Once the TCP/IP connection is established,
5. Use the details in your MQTT client to connect to
an MQTT handshake occurs. This involves exchanging
HiveMQ Cloud. For detailed steps, check out our
control packets to establish the connection and deal
documentation.
with parameters.
• Connect packet: The client sends a "CONNECT" packet
Step 3: Start Publishing MQTT Messages/
Subscribing to an MQTT Topic to the broker, indicating its intention to establish a
connection. This packet includes information such as
Using MQTT CLI or via the console, you can start publishing
client ID, connection flags, and other settings.
MQTT messages and subscribing to MQTT topics. Here’s an
• Acknowledgment (CONNACK): The broker responds with
example using MQTT CLI:
a "Connack" packet, indicating whether the connection
1. In the terminal window of the MQTT client (testClient2), request is accepted or rejected.
enter sub -t testTopic -[Link] command subscribes • Subscriptions (optional): If the client is a subscriber,
testClient2 to all messages that are published with the it can then send "Subscribe" packets to the broker,
topic testTopic. indicating the topics it wants to subscribe to.
2. In the terminal window of another MQTT client • Publishing messages: Once the connection is
(testClient1), enter pub -t testTopic -m Hello. This established, clients can publish messages to topics
command publishes the message Hello from testClient1 or subscribe to topics to receive messages from other
with the topic [Link] message Hello appears clients.
immediately in the terminal window of testClient2. • Keep-Alive: To maintain the connection, clients
periodically exchange "Ping" packets to confirm that the
For detailed steps, check out our documentation. connection is still active.

To learn more about how an MQTT Client communicates with

an MQTT Broker, read our blog MQTT Client, MQTT Broker, MQTT Brokers, like HiveMQ, offer advanced
and MQTT Server Connection Establishment Explained. features for reliable, flexible, scalable and secure
MQTT client connection.
If you are new to MQTT, read our blog post MQTT Tutorial: An
Easy Guide to Getting Started with MQTT.

Mechanism of How an MQTT Client Example Implementation of MQTT

Establishes a Connection with an MQTT To illustrate how MQTT works, below is a simple example
Broker that utilizes HiveMQ Cloud. To test this implementation on
Here's a basic overview of the mechanism of how an MQTT a live cluster, sign up for HiveMQ Cloud, which allows you
client establishes a connection with an MQTT broker: to connect up to 100 IoT devices at no cost. Sign-up without

[Link]

 MQTT Essentials Ebook

credit card information.

As a first-time user of HiveMQ Cloud, you will be automatically directed to the "Getting Started" section within the management
view of your cluster. Here, you can create access credentials and obtain connection details, as you will see later in this example.

Use Case Example: An IoT Application Built Using Raspberry Pi, MQTT, and Temperature Sensor
In this example, we explore the connection of a temperature and brightness sensor to a Raspberry Pi, and then leveraging the
power of MQTT to transmit the sensor data to a designated MQTT broker effortlessly. You will discover how another device, acting
as a control center, can effortlessly receive and process the MQTT data, enabling efficient monitoring and control of your IoT
ecosystem.

Communication between the sensor client and the control center over MQTT

Step 1 - Use the Raspberry Pi as an MQTT Client Connected to Sensors

Once you’ve successfully signed up for HiveMQ Cloud, click on Manage Cluster and head to the Connection Settings section on
the Overview tab of your cluster. There, you’ll discover your unique hostname/Cluster URL. Copy this hostname and replace it in
the code snippet provided below.

To establish a secure connection between your MQTT client and the cluster, creating MQTT client access credentials is essential.
Navigate to the Access Management tab of your HiveMQ Cloud cluster, and enter the username, password and set Permission to
Publish and Subscribe. These credentials will also replace "<your_username>" and "<your_password>" within the code snippet.

10
public class Sensor {

public static void main(String[] args) throws InterruptedException {

final String host = "<your_host>"; // use your host-name, it should look like
'<alphanumeric>.[Link]'
final String username = "<your_username>"; // your credentials
final String password = "<your_password>";

// 1. create the client

final Mqtt5Client client = [Link]()
.identifier("sensor-" + getMacAddress()) // use a unique identifier
.serverHost(host)
.automaticReconnectWithDefaultConfig() // the client automatically
reconnects
.serverPort(8883) // this is the port of your cluster, for mqtt it is the
default port 8883
.sslWithDefaultConfig() // establish a secured connection to HiveMQ Cloud
using TLS
.build();

// 2. connect the client

[Link]().connectWith()
.simpleAuth() // using authentication, which is required for a secure
connection
.username(username) // use the username and password you just created
.password([Link](StandardCharsets.UTF_8))
.applySimpleAuth()
.willPublish() // the last message, before the client disconnects
.topic("home/will")
.payload("sensor gone".getBytes())
.applyWillPublish()
.send();

// 3. simulate periodic publishing of sensor data

while (true) {
[Link]().publishWith()
.topic("home/brightness")
.payload(getBrightness())
.send();

11

[Link]

 MQTT Essentials Ebook

[Link](500);

[Link]().publishWith()
.topic("home/temperature")
.payload(getTemperature())
.send();

[Link](500);
}
}
//4. Simulate Temperature and Brightness sensor data
private static byte[] getBrightness() {
// simulate a brightness sensor with values between 1000lux and 10000lux
final int brightness =
[Link]().nextInt(1_000, 10_000);
return (brightness + "lux").getBytes(StandardCharsets.UTF_8);
}

private static byte[] getTemperature() {

// simulate a temperature sensor with values between 20°C and 30°C
final int temperature = [Link]().nextInt(20, 30);
return (temperature + "°C").getBytes(StandardCharsets.UTF_8);
}
}

Let’s dissect the code snippet provided above to understand its functionality:

1. Creating the MQTT Client: The code initializes the MQTT client, ensuring a unique identifier is used. An automatic reconnect
feature is also enabled to handle potential instability in the sensor’s internet connection.
2. Establishing Connection to "<your_host>": The client connects to the specified host. Notably, a “will” message is set, allowing
the broker to automatically publish a “sensor gone” notification if the sensor loses its connection.
3. Periodic Publication of Simulated Sensor Data: The code periodically publishes simulated brightness and temperature data
using the methods getBrightness() and getTemperature() methods, ensuring a steady stream of information for further
processing.

With this code snippet, you will have created an MQTT client, established a connection to the broker, and started regularly
transmitting the Brightness and Temperature sensor data.

Now, let’s move on to the next step in our implementation process:

12
Step 2 - Implementing the Subscribing Client
In this next step, we focus on creating the subscribing client responsible for consuming the values published on the topics home/
temperature and home/brightness.

Implementing the subscribing client enables the reception of sensor data transmitted via MQTT. This functionality allows you to
process and utilize the received information for various applications efficiently.

public class ControlCenter {

public static void main(String[] args) {

final String host = "<your_host>"; // use your host-name, it should look like
'<alphanumeric>.[Link]'
final String username = "<your_username>"; // your credentials
final String password = "<your_password>";

// 1. create the client

final Mqtt5Client client = [Link]()
.identifier("controlcenter-" + getMacAddress()) // use a unique identifier
.serverHost(host)
.automaticReconnectWithDefaultConfig() // the client automatically
reconnects
.serverPort(8883) // this is the port of your cluster, for mqtt it is the
default port 8883
.sslWithDefaultConfig() // establish a secured connection to HiveMQ Cloud
using TLS
.build();

// 2. connect the client

[Link]().connectWith()
.simpleAuth() // using authentication, which is required for a secure
connection
.username(username) // use the username and password you just created
.password([Link](StandardCharsets.UTF_8))
.applySimpleAuth()
.cleanStart(false)
.sessionExpiryInterval([Link](1)) // buffer messages
.send();

// 3. subscribe and consume messages

[Link]().subscribeWith()
.topicFilter("home/#")

13

[Link]

 MQTT Essentials Ebook

.callback(publish -> {
[Link]("Received message on topic " + [Link]()
+ ": " +
new String([Link](), StandardCharsets.
UTF_8));
})
.send();
}
}

The code snippet above performs the following actions: Chapter 3: MQTT Topics,
1. Creates an MQTT client instance, similar to the sensor Subscriptions, QoS, and Persistent
client, with the client ID prefixed as controlcenter-. Messaging
2. Establishes a connection between the client and the
specified host in <your_host>. To ensure message MQTT's Messaging Model: Topics and Subscriptions
buffering when the control center is offline, a session MQTT’s messaging model is based on topics and
expiry interval of 1 hour is set. subscriptions. Topics are strings that messages are published
3. The client Subscribes to all topics starting with home to and subscribed to. Topics are hierarchical and can contain
using the multi-level wildcard # in the topic filter. multiple levels separated by slashes, like a file path as shown
4. Any incoming messages with their corresponding topic below.
and payload are printed. If the sensor loses connection,
myhome/kitchen/smartdishwasher
the topic home/will and the payload “sensor gone” are
printed.
Subscriptions are used to specify which topics a client is
interested in receiving messages from.
The code in this example was tested on Java 11. You can also
find a publicly hosted example in our GitHub repository, along
When a client subscribes to a topic, it is essentially telling the
with an example tailored for our public broker.
broker that it is interested in receiving messages published
to that topic. The broker then keeps track of the subscription
By implementing both the sensor data publishing client and
and forwards any messages published to that topic to the
the subscribing client, you can establish a seamless MQTT
subscribed client.
communication system where sensor data is published and
consumed by the control center, enabling efficient monitoring
and control of devices.

14 Example: Smart door opening with a mobile device using MQTT

It’s important to note that a client can subscribe to multiple consequences, but duplicated commands would not.
topics at once, and a topic can have multiple subscribers. This • QoS 2: This level provides “exactly once” delivery, where
allows for a flexible and scalable messaging system. messages are confirmed and re-sent until they are
received exactly once by the subscriber. QoS 2 is the
In addition to topics and subscriptions, MQTT also supports highest level of QoS and is typically used in situations
wildcards, which can be used to subscribe to multiple topics where message loss or duplication is completely
that match a certain pattern. The two types of wildcards are unacceptable. With QoS 2, the publisher and broker
the single-level wildcard (+), which matches a single level in a engage in a two-step confirmation process, where the
topic, and the multi-level wildcard (#), which matches all levels broker stores the message until it has been received
after the specified level in a topic. and acknowledged by the subscriber. This level of QoS
is typically used for critical messages such as financial
Overall, MQTT’s messaging model provides a flexible and transactions or emergency alerts.
scalable way to publish and subscribe to messages using
topics and subscriptions. The use of wildcards adds an It’s important to note that higher QoS levels typically require
additional layer of flexibility, allowing for subscriptions more resources and can result in increased latency and
to multiple related topics using a single subscription. network traffic. As a result, it’s important to choose the
Understanding MQTT’s messaging model is crucial, but equally appropriate QoS level based on the specific needs of your
important is the quality of service (QoS) level that you choose application.
to ensure reliable message delivery.
In addition to the three levels of Quality of Service, MQTT also
Understanding MQTT Quality of Service (QoS) Levels supports message persistence, which ensures that messages
for IoT Applications are not lost in the event of a network or server failure.

MQTT supports three levels of Quality of Service (QoS): QoS 0,

QoS 1, and QoS 2. Here is the breakdown of each level:
Understanding MQTT Message Persistence for
Reliable IoT Communication
• QoS 0: This level provides “at most once” delivery, where
messages are sent without confirmation and may be lost.
Message persistence is an important feature in
This is the lowest level of QoS and is typically used in
MQTT. It ensures messages are not lost in the
situations where message loss is acceptable or where
event of a network or server failure. In MQTT,
the message is not critical. For example, QoS 0 might be
message persistence is achieved by storing
appropriate for sending sensor data where occasional
messages on the server until they are delivered to
data loss would not significantly impact the overall
the subscriber.
results.
• QoS 1: This level provides “at least once” delivery, where
MQTT provides three types of message persistence options:
messages are confirmed and re-sent if necessary. With
QoS 1, the publisher sends the message to the broker • Non-persistent: This is the default option in MQTT. In
and waits for confirmation before proceeding. If the this mode, messages are not stored on the server and are
broker does not respond within a set time, the publisher lost if the server or network fails. This mode is suitable
re-sends the message. This level of QoS is typically used for situations where messages are not critical and can be
in situations where message loss is unacceptable, but easily regenerated.
message duplication is tolerable. For example, QoS 1 • Queued persistent: In this mode, messages are stored
might be appropriate for sending command messages to on the server until they are delivered to the subscriber.
devices, where a missed command could have serious If the subscriber is not available, messages are queued

15

[Link]

 MQTT Essentials Ebook

until the subscriber reconnects. Queued persistence

is useful when the subscriber is not always connected MQTT is one such specific messaging protocol
to the network, or if the subscriber needs to receive all that follows the publish-subscribe architecture.
messages, even if they are sent when the subscriber is MQTT uses a broker-based model where clients
offline. connect to a broker, and messages are published
• Persistent with acknowledgment: This mode provides to topics. Subscribers can then subscribe
the highest level of message persistence. In this mode, to specific topics and receive the published
messages are stored on the server until they are delivered messages.
to the subscriber, and the subscriber must acknowledge
receipt of the message. If the subscriber does not
acknowledge receipt, the message is re-sent until the
subscriber acknowledges receipt. This mode is useful
when it is critical to ensure that messages are received
and processed by the subscriber.

To configure message persistence in MQTT, the broker

software used for handling MQTT connections must support
the chosen persistence option. The configuration can be done
through the broker’s configuration files or through its web
interface.
Example of MQTT Publish / Subscribe Architecture
It is important to note that message persistence comes with
a trade-off in terms of performance and storage. The more
persistent the messages, the more storage and processing Decoupling Features of MQTT Pub/Sub
resources are required by the broker. Therefore, it is important
The Pub/Sub architecture offers a unique alternative to
to choose the appropriate persistence level based on the
traditional client-server (request-response) models. In the
specific requirements of the application.
request-response approach, the client directly communicates
with the server endpoint, creating a bottleneck that slows down
Chapter 4: MQTT Publish/Subscribe performance. On the other hand, the pub/sub model decouples
Architecture (Pub/Sub) the publisher of the message from the subscribers. The
publisher and subscriber are unaware that the other exists. As
MQTT Pub/Sub architecture, also known as pub/sub, is
a third component, a broker handles the connection between
a messaging pattern in software architecture. It enables
them. This decoupling produces a faster and more efficient
communication between different components or systems
communication process.
in a decoupled manner. In the Pub/Sub architecture, there
are publishers that generate messages and subscribers
By eliminating the need for direct communication
that receive those messages. However, publish-subscribe is
between publishers and subscribers, pub/
a broader concept that can be implemented using various
sub architecture removes the exchange of IP
protocols or technologies.
addresses and ports. It also provides decoupling,
allowing operations on both components to
continue communication uninterrupted during
publishing or receiving.

16
The pub/sub features three dimensions of decoupling for appropriate subscribers based on the topic hierarchy. The
optimal efficiency: topic structure is hierarchical, with levels separated by a
forward slash (/), allowing subscribers to receive messages
• Space decoupling: Publisher and subscriber do not need
that match a specific topic level or a topic hierarchy. Here’s an
to know each other (for example, no exchange of IP
example of topic hierarchy.
address and port).
• Time decoupling: Publisher and subscriber do not need to
run at the same time.
• Synchronization decoupling: Operations on both
components do not need to be interrupted during
publishing or receiving.

Pub/Sub Decoupling in MQTT Protocol

MQTT decouples the publisher and subscriber

spatially, meaning they only need to know the
broker’s hostname/IP and port to publish or receive
messages. Additionally, MQTT decouples by time, Example of Topic Hierarchy

allowing the broker to store messages for clients

that are not online. Two conditions must be met to
Benefits of Subject-based Filtering in Pub/Sub:
store messages: the client must have connected
with a persistent session and subscribed to a topic • Simple and easy to use
with a Quality of Service greater than 0. • Flexible, allowing for a hierarchical topic structure
• Efficient, as it only forwards messages to subscribers
One of the most significant advantages of Pub/Sub software interested in a particular topic
architecture is its ability to filter all incoming messages
and distribute them to subscribers correctly, eliminating the Drawbacks of Subject-based filtering in Pub/Sub:
need for the publisher and subscriber to know one another’s
• Publishers and subscribers need to agree on the topic
existence.
hierarchy beforehand
• Limited to filtering messages based on topic hierarchy
MQTT Pub/Sub Message Filtering Feature
only
Message filtering is a crucial aspect of the pub/sub
architecture as it ensures subscribers only receive messages Use Case of Subject-based filtering in Pub/Sub:
they are interested in. The pub/sub broker offers several
Subject-based filtering is best suited for use cases where
filtering options, including subject-based filtering, content-
messages are organized into topics and subscribers are
based filtering, and type-based filtering.
interested in a particular subset of those topics.

Option 1: Subject-based Filtering of Pub/Sub

For example, in a smart home system, a subscriber may be
Architecture
interested in receiving updates about the temperature of a
This is the most common filtering option, where the broker specific room. The subscriber would subscribe to a topic like
filters the messages based on the topic or subject. The "smart-home/living-room/temperature," and the
subscribing clients indicate their interest by subscribing to broker would only send messages that match this topic to the
specific topics, and the broker routes the messages to the subscriber.

17

[Link]

 MQTT Essentials Ebook

Message Filtering in MQTT • Allows for filtering based on message content rather than
just topic hierarchy
MQTT uses subject-based filtering of messages. Every
• Flexible, allowing for complex filter expressions
message contains a topic (subject) that the broker
can use to determine whether a subscribing client
Drawbacks of Content-based Filtering in Pub/Sub:
gets the message or not. To handle the challenges
of a pub/sub system, MQTT has three Quality of • Can be more complex to use and set up than subject-
Service (QoS) levels. You can easily specify that a based filtering
message gets successfully delivered from the client • Requires publishers to include additional metadata in the
to the broker or from the broker to a client. However, message to enable filtering
there is the chance that nobody subscribes to the • Performance may suffer when processing large numbers
particular topic. If this is a problem, the broker must of filter expressions
know how to handle the situation. For example, the
HiveMQ MQTT Broker has an extension system that Use Case of Content-based Filtering in Pub/Sub:
can resolve such cases. You can have the broker take
Content-based filtering is best suited for use cases where
action or simply log every message into a database
messages are not organized into topics, and subscribers are
for historical analyses. To keep the hierarchical topic
interested in a specific subset of messages based on their
tree flexible, it is important to design the topic tree
content.
very carefully and leave room for future use cases.
If you follow these strategies, MQTT is perfect for
For example, in a logistics application, a subscriber may be
production setups.
interested in receiving messages only about packages with a
specific tracking number. The subscriber would subscribe to
a filter expression like "tracking-number = '123456'," and the
Option 2: Content-based Filtering of Pub/Sub broker would only send messages that match this expression
Architecture to the subscriber.
In this type of filtering, the broker filters messages based on
their content, which is specified using a filter expression. Option 3: Type-based Filtering of Pub/Sub
Subscribers indicate their interest by subscribing to a specific
Architecture
filter expression, and the broker routes the messages to In type-based filtering, the broker filters messages based
the appropriate subscribers based on the content of the on their type or class. This type of filtering is useful when
messages. working with object-oriented languages where messages are
represented as objects. Subscribers indicate their interest by
subscribing to a specific message type or class, and the broker
How to bring Content-based Filtering in MQTT?
routes messages to the appropriate subscribers based on the
Even though MQTT uses subject-based filtering of
message type.
messages, you can also set up content-based filtering
by using the HiveMQ MQTT Broker and our custom
Benefits of Type-based Filtering in Pub/Sub:
extension system.
• Allows for filtering based on message type, regardless of
the topic or content
Benefits of Content-based Filtering in Pub/Sub: • Simple to use, especially when working with object-
oriented languages
• Provides more granular control over which messages are
• Offers a high degree of flexibility and extensibility
received

18
Drawbacks of Type-based Filtering in Pub/Sub: connected to the broker who have subscribed to
the topic of a particular message, that message will
• Limited to filtering based on message type only
not be delivered to anyone. Therefore, it is vital for
• Publishers and subscribers need to agree on the message
publishers to keep in mind that message delivery
type hierarchy beforehand
is not guaranteed and to design their systems
accordingly.
Use Case of Type-based Filtering in Pub/Sub:

Type-based filtering is best suited for use cases where

messages are organized into a class hierarchy, and
MQTT Pub/Sub’s Scalability Feature
subscribers are interested in a specific type or subset of
messages based on their class. Scalability is one of the significant benefits of using the Pub/
Sub architecture. The traditional client-server model can limit
For example, in a financial application, a subscriber may be scalability, particularly when dealing with large numbers of
interested in receiving messages only about stock prices. The clients. However, with the pub/sub model, the broker can
subscriber would subscribe to a message type like “stock- process messages in an event-driven way, enabling highly
price,” and the broker would only send messages of this type parallelized operations. This means that the system can handle
to the subscriber. a greater number of concurrent connections without sacrificing
performance.
These filtering options provide flexibility and granularity in
deciding which messages are sent to which subscribers. In addition to event-driven processing, message caching
Depending on the use case, you can use one or more of these and intelligent message routing also contribute to improved
filtering options to ensure that subscribers receive only the scalability in Pub/Sub. By caching messages, the broker
messages they are interested in. can quickly retrieve and deliver them to subscribers without
additional processing. Intelligent routing, on the other hand,
However, it’s important to note that the pub/sub model may ensures that messages are delivered only to the subscribers
not be suitable for all use cases, and there are challenges that need them, reducing unnecessary network traffic and
to consider, such as ensuring that both the publisher and further improving scalability.
subscriber know which topics to use for subject-based
filtering and dealing with instances where no subscriber As MQTT follows the pub/sub architecture,
reads a particular message. You need to be aware of how the scalability comes naturally to this protocol, making
published data is structured beforehand. it ideal for several IoT use cases. Despite its
advantages, scaling up to millions of connections
For subject-based filtering, both publisher and can still pose a challenge for Pub/Sub. In such
subscriber need to know which topics to use. Also, cases, clustered broker nodes can be used to
with message delivery, the publisher can’t assume distribute the load across multiple servers, while
somebody is listening to the messages that are sent. load balancers can ensure that the traffic is evenly
This is an issue because, in publish-subscribe model, distributed. Check out how HiveMQ Broker can
the publisher sends messages to the broker without scale to 200 million concurrent connections using
knowing who the subscribers are or whether they this method.
are currently connected to the broker. The broker is
responsible for delivering messages to all connected Now that we have covered the basic concepts of Publish/
subscribers who subscribe to the appropriate topic. Subscribe architecture, let’s look at the benefits of using it for
However, if there are no subscribers currently IoT communication.

19

[Link]

 MQTT Essentials Ebook

What Are the Key Benefits of MQTT Pub/Sub receives only the messages of interest. As discussed
Architecture in IoT and IIoT? earlier, pub/sub provides three filtering options: subject-
based, content-based, and type-based filtering. Each
The pub/sub model offers several benefits, making it a popular
option has its benefits and drawbacks, and the choice of
choice for various applications. Here are some of the key
filtering method will depend on the use case. MQTT uses
advantages of using pub/sub architecture:
subject-based filtering of messages and every message
Improved scalability: The pub/sub architecture is highly contains a topic that the broker uses to determine
scalable, making it suitable for applications that handle many whether a subscribing client receives the message or not.
clients and messages. The broker acts as a central hub for 3. Security: Security is a crucial aspect of any messaging
all messages, allowing it to handle many clients without system, and pub/sub is no exception. MQTT allows for
compromising performance. several security options, such as user authentication,
access control, and message encryption, to protect the
Increased fault tolerance: The decoupled nature of pub/ system from unauthorized access and data breaches.
sub architecture also provides improved fault tolerance. In a 4. Scalability: Pub/Sub architecture must be designed with
traditional client-server model, all connected clients lose their scalability in mind, as the number of subscribers can grow
connection if the server goes down. In contrast, in pub/sub, exponentially in a large-scale system. MQTT provides
the broker can store messages until the client reconnects, features such as multiple brokers, clustering, and load
ensuring no messages are lost. balancing to ensure that the system can handle a large
number of subscribers and messages.
Flexibility: The pub/sub architecture is flexible and can be 5. Message Ordering: In a pub/sub system, message
used in a variety of applications, ranging from low-bandwidth, ordering can be challenging to maintain. As messages
high-latency networks to high-speed, low-latency networks. are sent asynchronously, it’s difficult to ensure that
The MQTT protocol, which is based on pub/sub architecture, subscribers receive messages in the correct order.
supports various quality-of-service levels, providing the However, MQTT provides QoS levels that ensures
flexibility to choose the appropriate level for your application. messages are successfully delivered from the client to
the broker or from the broker to a client.
Common Challenges in Pub/Sub Architecture
Because MQTT works asynchronously, tasks are not
and How to Overcome Them
blocked while waiting for or publishing a message. Most
While pub/sub architecture offers several benefits, such
client libraries are based on callbacks or a similar model,
as scalability, flexibility, and decoupling of components, it
making the flow of messages usually asynchronous.
also presents some challenges that must be addressed to
In certain use cases, synchronization is desirable and
ensure a successful implementation. Below are some of the
possible, and some libraries have synchronous APIs to
most common challenges of using pub/sub and solutions to
wait for a specific message.
overcome them:
6. Real-time Constraints: In some use cases, real-time
1. Message Delivery: One challenge of using pub/sub is
constraints are critical, and pub/sub architecture may
ensuring that messages are delivered to subscribers.
not be the best choice. For example, a request/response
In some instances, no subscribers may be available to
architecture may be a better option if low latency is
receive a particular topic, resulting in the message being
essential.
lost. To overcome this, MQTT provides quality of service
(QoS) levels.
You can address the challenges of using pub/sub through
2. Message Filtering: Another challenge of pub/sub is
careful design and implementation. Developers can build
filtering messages effectively so that each subscriber
scalable, secure, and efficient messaging systems by

20
understanding these challenges and utilizing MQTT’s features have helped it gain widespread adoption in IoT, mobile, and
effectively. other distributed applications.

MQTT vs. Message Queues Using MQTT, architecture engineers, and companies can build
systems reliably and efficiently communicate data in various
There is a lot of confusion about the name MQTT and
real-world scenarios. With its decoupling by space and time,
whether the protocol is implemented as a message queue or
asynchronous messaging, subject-based filtering, and Quality
not. We will try to shed some light on the topic and explain
of Service (QoS) levels, MQTT provides a robust set of features
the differences. We mentioned earlier that MQTT refers to
to help developers overcome the challenges of building
the MQSeries product from IBM and has nothing to do with
distributed systems. Overall, the pub/sub architecture and
“message queue“. Regardless of where the name comes from,
MQTT protocol are valuable tools for developers who want to
it’s useful to understand the differences between MQTT and a
build efficient and scalable distributed systems.
traditional message queue:

A message queue stores messages until they are consumed. Chapter 5: MQTT Client and MQTT
When you use a message queue, each incoming message is Broker Connection Establishment
stored in the queue until it is picked up by a client (often called
a consumer). If no client picks up the message, the message The two main components of the MQTT protocol are the

remains stuck in the queue and waits to be consumed. In a client and the broker. An MQTT client can be any device that

message queue, it is not possible for a message not to be runs an MQTT library and connects to an MQTT broker over a

processed by any client, as it is in MQTT if nobody subscribes network. The publisher and subscriber labels refer to whether

to a topic. the client is publishing or subscribed to receive messages. The

MQTT broker, on the other hand, is responsible for receiving

A message is only consumed by one client. Another big all messages, filtering them, and sending them to subscribed

difference is that in a traditional message queue a message clients. The broker also handles client authentication

can be processed by one consumer only. The load is and authorization and holds all clients’ session data with

distributed between all consumers for a queue. In MQTT the persistent sessions. Let’s dive deeper into foundational MQTT

behavior is quite the opposite: every subscriber that subscribes components.

to the topic gets the message.

One of the key features of the MQTT protocol is its efficient

Queues are named and must be created explicitly. A queue is and lightweight approach to exchanging messages between

far more rigid than a topic. Before a queue can be used, the IoT devices. The foundation of this communication is

queue must be created explicitly with a separate command. the MQTT connection, which enables devices to securely

Only after the queue is named and created is it possible to and reliably exchange data with the MQTT broker. In this

publish or consume messages. In contrast, MQTT topics are section, we will explore the process of establishing an

extremely flexible and can be created on the fly. If you can MQTT connection and the different parameters involved. By

think of any other differences that we overlooked, we would understanding how MQTT connections work, you can optimize

love to hear from you in the comments. your IoT deployment for better performance, security, and
scalability.

To summarize, the publish/subscribe (pub/sub) architecture

provides a flexible and scalable way of building distributed The MQTT protocol is based on TCP/IP, meaning the client and

systems that can handle many connected clients. MQTT’s the broker must have a TCP/IP stack.

lightweight and efficient pub/sub messaging characteristics

21

[Link]

 MQTT Essentials Ebook

For those unaware, NAT is a common networking technology

that routers use to allow devices on a private network to
access the internet through a single public IP address. NAT
works by translating the IP addresses of devices on the private
network to the public IP address of the router and vice versa.

In the case of MQTT, clients behind a NAT router

can still communicate with the MQTT broker
MQTT connections are always between one client and one because the broker has a public IP address and
broker, and clients never connect directly to other clients. To can connect with the client through the NAT.
initiate a connection, the client sends a CONNECT message However, some potential issues can arise with
to the broker, which responds with a CONNACK message and NAT, such as configuring port forwarding or
a status code. Once the connection is established, the broker opening firewall ports to allow incoming traffic to
keeps it open until the client sends a disconnect command or reach the MQTT broker. Additionally, some NAT
the connection breaks. implementations may have limitations on the
number of concurrent connections that can be
established, which could affect the scalability of
the MQTT system.

Now that we understand how MQTT clients behind a NAT

establish a connection with the broker, let’s take a closer look
at the MQTT CONNECT command message and its contents.
This section will explore the MQTT connection through a NAT
and how the MQTT client initiates a connection by sending How Does MQTT Client Initiate a Connection with the
a CONNECT message to the broker. We will delve into the CONNECT Message?
details of the MQTT CONNECT command message and focus Now let’s examine the MQTT CONNECT command message,
on some essential options, including ClientId, Clean Session, which the client sends to the broker to initiate a connection. If
Username/Password, Will Message, and Keep Alive. Moreover, this message is malformed or too much time elapses between
we will discuss the broker’s response to a CONNECT message, opening a network socket and sending the CONNECT message,
which is a CONNACK message containing two data entries: the the broker terminates the connection to deter malicious clients
session present flag and a connect return code. that can slow the broker down. In addition to other details
specified in the MQTT 3.1.1 specification, a good-natured
MQTT Connection Through a NAT MQTT 3 client sends the following content.
In many cases, MQTT clients live behind routers that use
network address translation (NAT) to convert private network Let’s focus on some of the essential options:
addresses (such as 192.168.x.x or 10.0.x.x) to public-
facing addresses. As mentioned, the MQTT client starts the
connection by sending a CONNECT message to the broker.
Since the broker has a public address and maintains the
connection open to enable bidirectional sending and receiving
of messages (after the initial CONNECT), MQTT clients located
behind NAT routers will have no difficulties.

22
While users of an MQTT library may find some of the Alternatively, some brokers like HiveMQ offer SSL certificate
information in a CONNECT message useful, certain details may authentication, eliminating the need for username and
be more relevant to implementers of the library. For a complete password credentials altogether. Taking these precautions
understanding of all the information contained in the message, ensures that your MQTT communication remains secure and
refer to the MQTT 3.1.1 specification. protected from potential security threats.

Let’s look at some of the elements the MQTT CONNECT What is Will Message in CONNECT MQTT Packet?
packet contains, such as ClientId, Clean Session, Username/
The MQTT Last Will and Testament (LWT) feature includes
Password, Will Message, Keep Alive, etc.
a last will message that notifies other clients when a client
disconnects unexpectedly. This message can be specified by
What is ClientId in CONNECT MQTT Packet?
the client within the CONNECT message as an MQTT message
The ClientId is a unique identifier that distinguishes each and topic. When the client disconnects abruptly, the broker
MQTT client connecting to a broker and enables the broker to sends the LWT message on the client’s behalf. Learn more
keep track of the client’s current state. To ensure uniqueness, about MQTT Last Will and Testament in Part 9 of this series.
the ClientId should be specific to each client and broker. MQTT
3.1.1 allows for an empty ClientId if no state needs to be What is Keep Alive in CONNECT MQTT Packet?
maintained by the broker. However, this connection must have
The MQTT Keep Alive feature allows the client to specify a
the clean session flag set to true, or the broker will reject the
time interval in seconds and communicate it to the broker
connection.
when establishing a connection. This interval determines the
longest period the broker and client can communicate without
What is CleanSession in CONNECT MQTT Packet?
sending a message. To ensure the connection remains active,
The CleanSession flag indicates whether the client wants the client sends regular PING Request messages, and the
to establish a persistent session with the broker. When broker responds with a PING response. This method allows
CleanSession is set to false (CleanSession = false), considered both sides to determine if the other is still available. Learn
a persistent session, the broker stores all subscriptions for the more about MQTT Keep Alive functionality in Part 10 of this
client and all missed messages for the client that subscribed series.
with a Quality of Service (QoS) level 1 or 2. In contrast, when
CleanSession is set to true (CleanSession = true), the broker When connecting to an MQTT broker from an MQTT 3.1.1
doesn’t retain any information for the client and discards any client, the Keep Alive interval is essential. However, some
previous state from any persistent session. MQTT libraries have additional configuration options,
such as the way queued messages are stored in a specific
What is Username/Password in CONNECT MQTT Packet? implementation.

MQTT provides the option to include a username and

MQTT Broker Response With a CONNACK Message
password for client authentication and authorization. However,
it’s important to note that sending this information in plain When a broker receives a CONNECT message, it is obligated to
text poses a security risk. To mitigate this risk, we highly
recommend using encryption or hashing (such as through TLS)
to protect the credentials. We also recommend using a secure
transport layer when transmitting sensitive data.

23

[Link]

 MQTT Essentials Ebook

respond with a CONNACK message. It’s essential to pay attention to the connect returnCode,
The CONNACK message contains two data entries: as it can help diagnose connection issues. For example,
if the returnCode indicates an authentication failure, the
• The session present flag
client can attempt to reconnect with the correct credentials.
• A connect return code
Understanding the sessionPresent flag and connect returnCode
is crucial for successful MQTT connections.
What is sessionPresent Flag in a CONNACK Message?

The sessionPresent flag informs the client whether a To summarize, understanding the roles of MQTT clients
previous session is still available on the broker. If the client and the broker and the connection establishment process
has requested a clean session, the flag will always be false, is essential for anyone interested in working with the MQTT
indicating there is no previous session. protocol. MQTT client libraries make adding MQTT support
to applications and devices easy without implementing the
However, if the client requests to resume a previous session, protocol from scratch. MQTT brokers are responsible for
the flag will be true if the broker still has stored session receiving, filtering, and sending messages to subscribed
information. This flag helps clients determine whether they clients and handling client authentication and authorization.
need to re-subscribe to topics or if the broker still has the With this knowledge, you can build scalable and efficient IoT
subscriptions from the previous session. systems using MQTT.

What is returnCode Flag in a CONNACK Message?

Chapter 6: MQTT Publish, MQTT
The returnCode is a status code that informs the client about Subscribe & Unsubscribe
the success or failure of the connection attempt. This code
can indicate various types of errors, such as invalid credentials MQTT PUBLISH Message
or unsupported protocol versions. In MQTT, a client can publish messages immediately when
it connects to a broker. The messages are filtered based on
Here are the returnCodes at a glance: topics, and each message must contain a topic that the broker
can use to forward the message to interested clients. The
payload of each message includes the data to transmit in byte
Return Code Return Code Response
format, and the sending client can choose to send any type of
data, including text, numbers, images, binary data, and even
0 Connection accepted
full-fledged XML or JSON.
Connection refused, unacceptable
1
protocol version

2 Connection refused, identifier rejected

3 Connection refused, server unavailable

Connection refused, bad user name or

4
password

5 Connection refused, not authorized

Example of MQTT Payload Format

24
 MQTT is data-agnostic, which means the
payload can be structured according to the 4. Publish Complete (PUBCOMP): The broker

specific use case of the client. The payload is finally sends a PUBCOMP message to

the message’s main content and is what the confirm that it has successfully received and

clients subscribe to, receive, and process. processed the message. This is the fourth and
final stage of the process.

A PUBLISH message in MQTT has several attributes that

determine its behavior including the packet identifier, topic These four messages are part of the MQTT protocol’s Quality

name, quality of service, retain flag, payload, and DUP flag. of Service (QoS) mechanisms, which ensure reliable message

Let’s take a look at each. delivery. The QoS level determines the number of messages
exchanged between the client and the broker.
What is MQTT PacketId or Packet Identifier?
It’s worth noting that the packet identifier uniquely identifies
The Packet Identifier (PacketId) is an essential attribute in
a message as it flows between the client and broker. The
MQTT. It is used to identify the specific message and to
packet identifier is only relevant for QoS levels greater than
ensure that messages are delivered in the order they were
zero. This holds true not only for PUBLISH, but for SUBSCRIBE,
sent, particularly when QoS levels greater than zero are used.
UNSUBSCRIBE and CONNECT messages.
The Packet ID is assigned by the client and is included in the
PUBLISH, PUBREL, PUBREC, and PUBCOMP messages. When
The client library and/or the broker is responsible for setting
the broker receives a PUBLISH message, it assigns a Packet
this internal MQTT identifier. When a QoS level greater than
ID to the message and sends a PUBACK message to the client
zero is used, the client must wait for a PUBACK or PUBREC
containing the Packet ID of the PUBLISH message. The client
message from the broker before it can send the next message.
uses the PUBACK message to confirm that the broker has
The client should also keep track of the Packet IDs it has
received the message.
sent and received to ensure that messages are not lost or
duplicated. Overall, the Packet ID is essential to MQTT’s
In the MQTT protocol, messages related to publishing
reliability mechanism and helps ensure that messages are
and acknowledgment of messages are divided into
delivered correctly and efficiently.
several stages:

What is MQTT Topic Name?

1. Publish (PUBLISH): This is the first stage of the
MQTT uses the topic name as a fundamental concept. It
process and involves an MQTT client publishing a
structures this name hierarchically using forward slashes as
message to the broker. The message contains a
delimiters and creates a simple string. It’s similar to a URL
topic and a payload.
path but without the protocol and domain components. MQTT
2. Publish Received (PUBREC): After receiving the
topics are used to label messages and provide a way for
PUBLISH message, the broker sends a PUBREC
clients to subscribe to specific messages.
message to acknowledge that it has received the
message. This is the second stage of the process.
For example, a device that measures temperature might
3. Publish Release (PUBREL): Once the client receives
publish its readings to the topic "sensors/temperature/
the PUBREC message, it sends a PUBREL message
livingroom". A client interested in these readings can subscribe
to release the broker from the responsibility of
to this topic and receive updates as they’re published.
keeping the message in memory. This is the third
MQTT provides two types of wildcards to use with topic
stage.
subscriptions:
25

[Link]

 MQTT Essentials Ebook

• "+" (plus sign) is used to match a single level in the that a message will be delivered. The message is sent
hierarchy. For example, a subscription to "sensors/+/ once, and if it is lost or not received by the recipient, it will
livingroom" would match “sensors/temperature/ not be resent.
livingroom” and “sensors/humidity/livingroom”, but not • QoS 1 (at least once): This level ensures that a message
“sensors/temperature/kitchen”. is delivered at least once, but it may be delivered multiple
• "#" (hash sign) is used to match multiple levels in the times in the case of network issues or failures.
hierarchy. For example, a subscription to “sensors/#” • QoS 2 (exactly once): This level provides the highest
would match “sensors/temperature/livingroom”, level of assurance for message delivery. The message
“sensors/humidity/kitchen”, and “sensors/power/ is guaranteed to be delivered exactly once, but this level
meter1”. requires more communication between the sender and
receiver, which can increase latency and network traffic.
Subscribing to a large number of topics can have a significant
impact on broker performance. This is because every message Choosing the appropriate QoS level depends on the specific
that is published to a topic subscribed by a client must be use case. For example, QoS 0 might be suitable for non-critical
delivered to that client. If many clients subscribe to many data, while QoS 2 might be necessary for critical data requiring
topics, this can quickly become a heavy burden on the broker. high-reliability levels.

Using wildcards to subscribe to multiple topics with a single It’s important to note that the QoS level can impact the
subscription can also impact performance. When a client performance of the broker and network, so it’s recommended
subscribes to a topic with a wildcard, the broker must evaluate to use the appropriate level for the specific use case.
every message published to a matching topic and determine
whether to forward it to the client. If the number of matching What is MQTT Retain Flag?
topics is large, this can strain the broker’s resources.
The Retained Flag is an important feature that determines
whether the broker saves a message as the last known good
To avoid performance issues, it’s important to use topic
value for a specified topic. When the retained flag is set to true,
subscriptions efficiently. One approach is to use more
the broker will save the most recent message that matches the
specific topic filters whenever possible, rather than relying on
specified topic, regardless of whether there are any subscribed
wildcards. Another approach is to use shared subscriptions,
clients.
which allow multiple clients to share a single subscription to
a topic. This can help reduce the number of subscriptions and
When a new client subscribes to a topic with a retained
messages that the broker must handle. Finally, monitoring
message, the broker sends the last retained message (on
the broker’s performance and adjusting its configuration as
that topic) to the client. This allows clients to receive the
necessary is important to ensure optimal performance.
most recent and relevant information even if they have not
subscribed to that topic before.
What is Quality of Service (QoS) in MQTT?
We touched upon Quality of Service Level (QoS) of an MQTT It’s important to note that, like many of the other elements,
message earlier. To refresh, QoS is indicated by a number that the use of retained messages can also impact the broker’s
ranges from 0 to 2. Each level provides a different level of performance, especially if there are many retained messages.
reliability and assurance for message delivery. Additionally, if a retained message is updated frequently, it can
result in increased network traffic and potentially affect the
network’s performance.
• QoS 0 (at most once): This level provides no guarantee What is MQTT Payload?

26
The payload is the actual content of the message and can 4. Feedback: The publishing client receives a confirmation
contain any kind of data. MQTT is data-agnostic, meaning message from the broker indicating that the message
it can handle different data types, including images, text in was published successfully. However, the client does not
any encoding, encrypted data, and binary data. However, receive feedback on how many subscribers received the
it’s important to note that the payload size can impact message or whether anyone is interested in it.
network performance and memory usage on the client and The client that initially publishes the message is only
broker. Therefore, keeping payloads as small as possible is concerned about delivering the PUBLISH message to the
recommended, especially when publishing messages with a
high frequency.

What is MQTT DUP Flag?

The MQTT DUP Flag indicates that a message is a duplicate
and has been resent because the intended recipient (client or
broker) did not acknowledge the original message. It is only
relevant for messages with QoS greater than 0. When a client How MQTT PUBLISH works

or broker receives a message with the DUP flag set, it should

broker. Once the broker receives the PUBLISH message, it is
ignore the message if it has already received a message with
the responsibility of the broker to deliver the message to all
the same message ID. The client or broker should process the
subscribers. The publishing client does not get any feedback
message normally if they have not previously received it.
about whether anyone is interested in the published message
or how many clients received the message from the broker.
The MQTT protocol (MQTT client library or broker) handles
resend and duplicate mechanism automatically, but it’s
important to note that this can impact network performance
How to Subscribe to MQTT Topics?
and increase network traffic. Publishing a message doesn’t make sense if no one ever
receives it. This is where subscribing comes into play. Once a
How do MQTT brokers handle messages from clients? client publishes a message to an MQTT broker, the message
must be delivered to interested clients. Clients that want to
When a client publishes a message to an MQTT broker, the
receive messages on topics of interest send a SUBSCRIBE
broker performs several tasks to ensure the message is
message to the broker. The SUBSCRIBE message is simple and
delivered according to the QoS level specified by the client.
contains a unique packet identifier and a list of subscriptions.
Here’s what happens:

1. Message reception: The broker reads the message sent by

the client and verifies its syntax and format.
2. Acknowledgement: The broker sends an acknowledgment
message to the client to confirm receipt of the message.
The level of acknowledgment depends on the QoS level
requested by the client.
3. Processing: The broker determines which clients have
subscribed to the topic of the message and sends a copy
of the message to each of them. The broker may also
retain the message as the last known good value for that Example of MQTT SUBSCRIBE Packet
topic, depending on the value of the Retained flag.

27

[Link]

 MQTT Essentials Ebook

Packet Identifier: The packet identifier is unique and identifies Packet Identifier: The SUBACK message includes the
a message as it flows between the client and broker. The client same packet identifier that the client included in the
library or the broker is responsible for setting this internal SUBSCRIBE message, which enables the client to match the
MQTT identifier. acknowledgment to the original request.

List of Subscriptions: A SUBSCRIBE message can contain Return Code: The SUBACK message also includes one return
multiple subscriptions for a client. Each subscription includes code for each topic/QoS-pair specified in the SUBSCRIBE
a topic and a QoS level. The topic in the SUBSCRIBE message message. The return codes are binary values that indicate
can contain wildcards that make it possible to subscribe whether the broker has granted or rejected the subscription
to a topic pattern instead of a specific topic. If there are request for each topic.
overlapping subscriptions for one client, the broker delivers the
message with the highest QoS level for that topic. The return codes for QoS levels are as follows:

• QoS 0: This means the subscription request has been

Overall, MQTT allows clients to subscribe to specific topics,
granted at QoS 0. The broker delivers the messages to the
receive messages published to those topics, and process the
client as soon as they are available and with no quality
payloads according to their specific use case. The packet
guarantees.
identifier and QoS level in the SUBSCRIBE message ensure that
• QoS 1: This means the subscription request has been
messages are delivered reliably and with the appropriate level
granted at QoS 1. The broker delivers messages at least
of quality.
once, meaning that the broker sends the message to
the client at least one time. The client sends a PUBACK
Once a client sends a SUBSCRIBE message with the list of
message back to the broker after receiving the message,
desired topics and QoS levels to an MQTT broker, the
which acts as an acknowledgment.
broker responds with a SUBACK message that confirms the
• QoS 2: This means the subscription request has been
subscription and indicates the maximum QoS level that the
granted at QoS 2. The broker delivers messages exactly
broker will deliver. Let’s look deeper into SUBACK.
once, which means that the broker guarantees that
the message is delivered once and only once to the
What is MQTT Suback?
client. The client sends a PUBREC message back to the
Once the client sends a SUBSCRIBE message to the broker broker after receiving the message, which acts as an
with the topics and corresponding QoS levels, the broker acknowledgment. The broker sends a PUBREL message
acknowledges the subscription request by sending a SUBACK to the client after receiving the PUBREC message, and
message to the client. The SUBACK message confirms the the client sends a PUBCOMP message to the broker after
receipt of the SUBSCRIBE message and indicates whether the receiving the PUBREL message.
broker has accepted or rejected each subscription.
If the broker rejects any of the subscriptions in the SUBSCRIBE
message, the SUBACK message contains a failure return code
for that specific topic. The reason for the failure could be that
the client has insufficient permission to subscribe to the topic,
the topic is malformed, or another reason.

The failure return code is represented by 0x80 and indicates

that the subscription is not accepted by the broker. This can
happen if the client does not have sufficient permission to
Example of MQTT SUBACK Packet
28
subscribe to the topic, the topic is malformed, or there is How to Use Unsubscribe in MQTT to Revoke
another issue with the subscription request. When a client Subscriptions?
receives a failure return code, it should retry the subscription
In MQTT, clients can unsubscribe from the topics they have
with a different topic or QoS level or take appropriate action to
subscribed to by sending an UNSUBSCRIBE message to
address the issue with the subscription request.
the broker. Similar to SUBSCRIBE, this message includes a
packet identifier to uniquely identify it and a list of topics to
The SUBACK message is an acknowledgment message from
unsubscribe from.

Return Code Return Code Response

0 Success - Maximum QoS 0

1 Success - Maximum QoS 1

2 Success - Maximum QoS 2

128 Failure

Example of MQTT UNSUBSCRIBE Packet

Packet Identifier: Similar to the SUBSCRIBE message, the

packet identifier in the UNSUBSCRIBE message serves as an
internal MQTT identifier for message flow between the client
and broker. It ensures that the client and broker can keep
track of the message and its corresponding acknowledgment
messages.
How MQTT SUBSCRIBE, SUBACK, and PUBLISH work

List of Topics: The list of topics in the UNSUBSCRIBE message

the broker to the client to confirm the subscriptions that have
can contain one or multiple topics from which the client wants
been granted or rejected. The packet identifier enables the
to unsubscribe. It is not necessary to specify the QoS level
client to match the acknowledgment to the original request,
since the broker will unsubscribe the topic regardless of the
while the return codes indicate the QoS levels at which the
QoS level with which it was originally subscribed.
broker has granted the subscriptions.

What is MQTT Unsuback?

After a client has subscribed to topics of interest and received
messages published to those topics, it may eventually need After receiving the UNSUBSCRIBE message, the broker sends

to unsubscribe. Let’s now explore the counterpart of the an UNSUBACK acknowledgment message to confirm the

SUBSCRIBE message, the UNSUBSCRIBE message, and removal of the client’s subscriptions. This message includes

the corresponding UNSUBACK message that confirms the the packet identifier of the UNSUBSCRIBE message and serves

unsubscription. as an acknowledgment that the broker has successfully

removed the topics from the client’s subscription list.

29

[Link]

 MQTT Essentials Ebook

To summarize, MQTT provides a flexible and data-agnostic

approach to publishing messages between clients and
brokers. By using topics to filter messages, clients can
quickly and easily subscribe to the content that interests
them. The payload of each message can be customized to
meet each client’s specific needs, and MQTT’s support for
various data types makes it a versatile solution for many
use cases. Additionally, understanding the attributes of a
PUBLISH message, such as the QoS level and retain flag, can
help clients and brokers ensure that messages are delivered
Example of MQTT UNSUBACK Packet efficiently and reliably.

Packet Identifier: The packet identifier in the UNSUBACK

Chapter 7: MQTT Topics and
message is the same as the one in the corresponding
Wildcards
UNSUBSCRIBE message. This ensures that the client can
identify the acknowledgment message and correlate it to the What Are MQTT Topics and Their Role in
original UNSUBSCRIBE message. MQTT Message Filtering?
In MQTT, Topic refers to a UTF-8 string that filters messages
Return Codes: The UNSUBACK message includes a list of
for a connected client. A topic consists of one or more levels
return codes for each topic/QoS-pair that was unsubscribed. A
return code of 0 indicates a successful removal, while a return
code of 17 indicates an unsuccessful removal due to an invalid
or malformed topic. Other return codes may also be specified
for different error scenarios.

Learn MQTT Topic Basics

separated by a forward slash (topic level separator).

In comparison to a message queue, MQTT topics are very
lightweight. The client does not need to create the desired
topic before they publish or subscribe to it. The broker accepts
each valid topic without any prior initialization.
How MQTT UNSUBACK works

Examples of MQTT Topics

After receiving the UNSUBACK from the broker, the client can
assume that the subscriptions in the UNSUBSCRIBE message Here’re some examples of MQTT Topics:
are deleted. 1. myhome/groundfloor/livingroom/temperature:
This topic represents the temperature in the living room
These details provide a comprehensive understanding of how of a home located on the ground floor.
clients can unsubscribe from topics and how brokers confirm 2. USA/California/San Francisco/Silicon
the removal of those subscriptions through the UNSUBSCRIBE Valley: This topic hierarchy can track or exchange
and UNSUBACK messages, respectively. information about events or data related to the Silicon
Valley area in San Francisco, California, within the United
States.
30
3. 5ff4a2ce-e485-40f4-826c-b1a5d81be9b6/ MQTT Wildcard – Single Level: +
status: This topic could be used to monitor the status
The single-level wildcard is represented by the plus symbol
of a specific device or system identified by its unique
(+) and allows the replacement of a single topic level. By
identifier.
subscribing to a topic with a single-level wildcard, any topic
4. Germany/Bavaria/car/2382340923453/
that contains an arbitrary string in place of the wildcard will be
latitude: This topic structure could be utilized to share
matched.
the latitude coordinates of a particular car in the region of
Bavaria, Germany.

Best Practices for Using MQTT Topics

Here’re some best practices for using MQTT Topics:

• Each topic must contain at least one character.

• Topic strings can include empty spaces to allow for more Example of how to use MQTT Wildcard Single Level +
readable or descriptive topics.
• Topics are case-sensitive, meaning "myhome/ For example, a subscription to myhome/groundfloor/+/

temperature" and "MyHome/Temperature" are considered temperature can produce the following results:

as two different topics.

• The forward slash alone is a valid topic and can be used
to represent a broad topic or serve as a wildcard for
subscribing to multiple topics simultaneously.

MQTT topics are key in establishing

communication between MQTT clients and MQTT Wildcard – Multi Level:
brokers. They enable efficient filtering and
The multi-level wildcard covers multiple topic levels. It is
routing of messages based on their content.
represented by the hash symbol (#) and must be placed as the
Properly defining and structuring topics is crucial
last character in the topic, preceded by a forward slash.
in ensuring effective data exchange and handling
When a client subscribes to a topic with a multi-level wildcard,
within MQTT-based systems.

MQTT Wildcards and How to Use Them With

Topic Subscriptions? Example of how to use MQTT Wildcard Multi Level #
In MQTT, wildcards provide a powerful mechanism for
subscribing to multiple topics simultaneously. When a client it receives all messages of a topic that begins with the pattern
subscribes to a topic, it can either subscribe to the exact topic before the wildcard character, regardless of the length or depth
of a published message or utilize wildcards to broaden its of the topic. If the topic is specified as “#” alone, the client
subscription. It’s important to note that wildcards can only be receives all messages sent to the MQTT broker.
used for subscription and not for publishing messages. There
are two types of wildcards: single-level and multi-level.

31

[Link]

 MQTT Essentials Ebook

3. $SYS/broker/clients/total: Represents the total count

of clients, both connected and disconnected, that have
interacted with the MQTT broker.
4. $SYS/broker/messages/sent: Provides the count of
messages sent by the MQTT broker.

MQTT Topic wildcard hash example 5. $SYS/broker/uptime: Reflects the duration the MQTT
broker has been running.

However, it’s important to consider that subscribing with These $SYS topics offer valuable insights into the
a multi-level wildcard alone can be an anti-pattern if high internal workings and performance of the MQTT
throughput is expected. Subscribing to a broad topic can result broker, enabling administrators and developers to
in a large volume of messages being delivered to the client, monitor and analyze crucial statistics.
potentially impacting system performance and bandwidth
usage. Follow best practices to optimize topic subscriptions By understanding the purpose and significance of topics
and avoid unnecessary message overload. starting with $, you can effectively leverage this convention
to gain deeper visibility into the behavior and performance of
MQTT Topics Beginning with $ their MQTT infrastructure.

In MQTT, topic naming flexibility is vast, allowing you to

Exploring the Dynamic Nature of MQTT Topics
choose names that suit your needs. However, there is one
important exception to be aware of: topics that start with a $ These are the basics of MQTT message topics. As you can
symbol have a distinct purpose. These topics are not included see, MQTT topics are dynamic and provide great flexibility.
in the subscription when using the multi-level wildcard (#) as a When you use wildcards in real-world applications, there are
topic. Instead, topics beginning with $ are reserved for internal some challenges you should be aware of. We have collected
statistics of the MQTT broker, providing valuable insights into the best practices that we have learned from working
its operation. extensively with MQTT in various projects and are always open
to suggestions or a discussion about these practices. Use
Publishing messages to topics starting with $ is not permitted, the comments to start a conversation, Let us know your best
as these topics serve as a means for the MQTT broker to practices or if you disagree with one of ours!
expose internal information and statistics to clients. While
there is currently no official standardization for these topics, it MQTT Best Practices
is common to use the prefix $SYS/ to denote such information, Avoid Leading Forward Slash
although specific implementations of brokers may vary.
While MQTT allows a leading forward slash in topics (e.g., /
myhome/groundfloor/livingroom), it introduces an unnecessary
One recommended resource for understanding $SYS topics is
topic level with a zero character at the front. This can cause
available in the MQTT GitHub wiki.
confusion (having a zero character at the front) without
providing any benefit. Hence, it’s recommended to exclude the
Here are a few examples of $SYS topics and the information
leading forward slash.
they can provide:

1. $SYS/broker/clients/connected: Indicates the number of Never use spaces in an MQTT Topic

clients currently connected to the MQTT broker.
A space is the natural enemy of every programmer. Spaces in
2. $SYS/broker/clients/disconnected: Shows the number of
topics can hinder readability and debugging efforts, particularly
clients that have disconnected from the MQTT broker.
32
during troubleshooting scenarios. Moreover, UTF-8 has many Embrace Extensibility
different white space types. We advise against using spaces
Topics in MQTT provide inherent flexibility, allowing for future
and uncommon characters altogether in MQTT topics.
expansion and new features. Consider how your topic structure
can accommodate future enhancements or the addition of
Keep MQTT topics short and concise
new sensors or functionalities. Design your topics to facilitate
Remember that each topic is included in every message in extensibility without substantially changing the overall topic
which it is used. To optimize network traffic and conserve hierarchy. For example, if your smart-home solution adds new
valuable resources, strive to make your topics concise. This sensors, it should be possible to add these to your topic tree
is especially crucial when dealing with resource-constrained without changing the whole topic hierarchy.
devices, where every byte counts.
Use specific topics, not general ones
Use only ASCII characters, and avoid non-printable
Differentiate your topics to reflect specific data streams or
characters
entities. Avoid the temptation to use a single topic for multiple
To ensure consistent and accurate representation of topics, types of messages. For instance, if you have three sensors
it’s advisable to stick to ASCII characters. Non-ASCII UTF-8 in your living room, create topics like myhome/livingroom/
characters may display incorrectly, making identifying typos temperature, myhome/livingroom/brightness, and myhome/
or character set-related issues challenging. Unless essential, livingroom/humidity instead of using a generic topic like
refrain from using non-ASCII characters in your MQTT topics. myhome/livingroom. This practice promotes clarity and
enables the utilization of advanced MQTT features such as
Embed a unique identifier or the Client Id in topics retained messages.

To enhance message identification and enforce authorization,

Documentation
consider embedding a unique identifier or the client ID of the
publishing client in the topic. This allows you to determine Maintain comprehensive documentation detailing your
the message sender and control publishing permissions. For MQTT topics, including their purpose, expected message
example, a client with the client1 ID can publish to client1/ payload, and any associated conventions or guidelines. This
status but not to client2/status. aids in onboarding new team members and fosters better
collaboration.
Avoid Subscribing to Wildcards (#)
Continuous Improvement
Sometimes, it is necessary to subscribe to all messages that
are transferred over the broker. For example, to persist all Regularly review and optimize your topic structure based
messages into a database. Do not subscribe to all messages on evolving requirements and feedback from your MQTT
on a broker by using an MQTT client and subscribing to a ecosystem. Embrace a continuous improvement mindset to
multi-level wildcard. Frequently, the subscribing client is not ensure efficient and scalable MQTT communication.
able to process the load of messages that results from this
method (especially if you have a massive throughput). Our Security Considerations
recommendation is to implement an extension in the MQTT
Ensure that your topic structure and naming conventions don’t
broker. For example, with the HiveMQ extensions, you can
inadvertently expose sensitive information. Implement proper
hook into the behavior of HiveMQ and add an asynchronous
access controls and authentication mechanisms to protect
routine to process each incoming message and persist it to a
your MQTT communications.
database.

33

[Link]

 MQTT Essentials Ebook

By following these best practices, you can enhance your MQTT providing the client with the ability to select a service level that
infrastructure’s readability, maintainability, and security. aligns with both the network reliability and the application’s
requirements. MQTT’s inherent capability to handle message
To summarize, MQTT topics serve as the backbone of re-transmission and ensure delivery, even in unreliable network
flexible and efficient message communication in MQTT. By conditions, makes QoS essential for facilitating seamless
understanding the intricacies and applying best practices, communication in such challenging environments. By offering
you can optimize your MQTT implementations for maximum different QoS levels, MQTT empowers clients to optimize
performance and scalability. their network usage and achieve the desired balance between
reliability and efficiency.
Chapter 8: MQTT Quality of Service
(QoS) 0,1, & 2 Now that we understand the significance of Quality of Service
(QoS) in MQTT, let’s delve into the inner workings of QoS and
MQTT Quality of Service (QoS) is an agreement between the explore how it operates to ensure reliable message delivery in
message sender and receiver that defines the level of delivery varying network conditions.
guarantee for a specific message.
How does QoS 0 work in MQTT?
MQTT provides three levels of QoS: At the lowest level, QoS 0 in MQTT offers a best-effort
• At most once (QoS 0) delivery mechanism where the sender does not expect an
• At least once (QoS 1) acknowledgment or guarantee of message delivery. This
• Exactly once (QoS 2) means that the recipient does not acknowledge receiving the
message, and the sender does not store or re-transmit it. QoS
How to Examine Message Delivery in MQTT? 0, commonly called “fire and forget,” functions akin to the
underlying TCP protocol, where the message is sent without
When discussing QoS in MQTT, it’s important to consider
further follow-up or confirmation.
message delivery from the publishing client to the broker and
from the broker to the subscribing client. These two aspects of
message delivery have subtle differences.

The client that publishes a message to the broker defines the

QoS level for the message during transmission. The broker
then transmits the message to subscribing clients using
the QoS level defined by each subscribing client during the
subscription process. If the subscribing client defines a lower Quality of Service level 0: delivery at most once

QoS level than the publishing client, the broker will transmit the
How does QoS 1 work in MQTT?
message with the lower QoS level.
In QoS 1 of MQTT, the focus is on ensuring message delivery
Understanding how message delivery works in MQTT sets at least once to the receiver. When a message is published
the foundation for appreciating the significance of Quality with QoS 1, the sender keeps a copy of the message until it
of Service (QoS) levels in ensuring reliable communication receives a PUBACK packet from the receiver, confirming the
between the publishing client, broker, and subscribing client. successful receipt. If the sender doesn’t receive the PUBACK
packet within a reasonable time frame, it re-transmits the
Why is Quality of Service (QoS) Important? message to ensure its delivery.

Quality of Service (QoS) is crucial in MQTT due to its role in

34
 How does QoS 2 work in MQTT?

QoS 2 offers the highest level of service in MQTT, ensuring

that each message is delivered exactly once to the intended
recipients. To achieve this, QoS 2 involves a four-part
handshake between the sender and receiver.
Upon receiving the message, the receiver can process it
Quality of Service level 1: delivery at least once

immediately. For example, if the receiver is an MQTT broker,

it distributes the message to all subscribing clients and
responds with a PUBACK packet to acknowledge the receipt of
the message.

MQTT Quality of Service level 2: delivery exactly once

It’s important to note that in QoS 1, if the publishing client

When a receiver gets a QoS 2 PUBLISH packet from a sender, it

processes the publish message accordingly and replies to the
sender with a PUBREC packet that acknowledges the PUBLISH
packet. If the sender does not get a PUBREC packet from the
receiver, it sends the PUBLISH packet again with a duplicate
(DUP) flag until it receives an acknowledgment.

MQTT PUBACK packet

sends the same message again, it sets a duplicate (DUP) flag.

However, this flag is used for internal purposes and is not
processed by the broker or client. Regardless of the DUP flag,
the receiver still sends a PUBACK packet to acknowledge the
receipt of the message, ensuring the sender is aware of the
successful delivery.

This approach of QoS 1 strikes a balance between reliability MQTT PUBREC Packet
and efficiency, guaranteeing that the message reaches the
receiver at least once while allowing for potential duplicates to Once the sender receives a PUBREC packet from the receiver,
be handled appropriately. the sender can safely discard the initial PUBLISH packet.
The sender stores the PUBREC packet from the receiver and
responds with a PUBREL packet, the receiver discards all
stored states and replies with a PUBCOMP packet.

35

[Link]

 MQTT Essentials Ebook

Key Considerations for QoS in MQTT

While understanding QoS in MQTT, there are several important
aspects to keep in mind:

Downgrade of QoS

The QoS levels defined by the sender and receiver can differ.
The client sending the message to the broker defines the QoS
level, while the broker uses the QoS defined by the receiver
during subscription. For example, if the sender uses QoS 2
MQTT PUBREL Packet and the receiver subscribes with QoS 1, the broker delivers the
message to the receiver with QoS 1. This can result in multiple
After the receiver gets the PUBREL packet, it can discard deliveries of the same message to the receiver.
all stored states and answer with a PUBCOMP packet (the
same is true when the sender receives the PUBCOMP). Until Packet identifiers are unique per client
the receiver completes processing and sends the PUBCOMP
Packet identifiers used for QoS 1 and 2 are unique between
packet back to the sender, the receiver stores a reference to
a specific client and a broker within an interaction. However,
the packet identifier of the original PUBLISH packet. This step
they are not unique across all clients. Once a flow is complete,
is important to avoid processing the message a second time.
the packet identifier becomes available for reuse. This is why
the packet identifier does not need to exceed 65535 , as it
After the sender receives the PUBCOMP packet, the packet
is unrealistic for a client to send more messages than that
identifier of the published message becomes available for
without completing an interaction
reuse.
When the QoS 2 flow is complete, both parties are sure that the
Best Practices While Using MQTT Quality of
Service (QoS) 0,1, & 2
We are often asked for advice about how to choose the correct
QoS level. Selecting the appropriate QoS level depends on your
specific use case. Here are some guidelines to help you make
an informed decision:

Use QoS 0 when:

• You have a completely or mostly stable connection

between sender and receiver. A classic use case for QoS
MQTT PUBCOMP Packet
0 is connecting a test client or a front end application to
an MQTT broker over a wired connection.
message is delivered and the sender has confirmation of the
• You don’t mind if a few messages are lost occasionally.
delivery.
The loss of some messages can be acceptable if the
data is not that important or when data is sent at short
If a packet gets lost along the way, the sender is responsible
intervals
to retransmit the message within a reasonable amount of time.
• You don’t need message queuing. Messages are only
This is equally true if the sender is an MQTT client or an MQTT
queued for disconnected clients if they have QoS 1 or 2
broker. The recipient has the responsibility to respond to each
and a persistent session.
command message accordingly.

36
Use QoS 1 when: What are Persistent Sessions?
• You need to get every message and your use case can To receive messages from an MQTT broker, a client
handle duplicates. QoS level 1 is the most frequently used establishes a connection and creates subscriptions to the
service level because it guarantees the message arrives desired topics. In a non-persistent session, if the connection
at least once but allows for multiple deliveries. Of course, between the client and broker is interrupted, the client loses its
your application must tolerate duplicates and be able to subscriptions and needs to re-subscribe upon reconnection.
process them accordingly. This can be burdensome for resource-constrained clients. To
• You can’t bear the overhead of QoS 2. QoS 1 delivers address this issue, clients can request a persistent session
messages much faster than QoS 2. when connecting to the broker.

Use QoS 2 when: Persistent sessions store all relevant client

• It is critical to your application to receive all messages information on the broker, ensuring that
exactly once. This is often the case if a duplicate delivery subscriptions and messages are retained
can harm application users or subscribing clients. Be even when the client is offline. The session is
aware of the overhead and that the QoS 2 interaction identified by the clientId provided by the client
takes more time to complete. during the connection establishment process.

Queuing of QoS 1 and 2 Messages What’s Stored in a Persistent Session?

All messages sent with QoS 1 and 2 are queued for In a persistent session, the broker stores the following

offline clients until the client is available again. However, information (even if the client is offline). This information

this queuing is only possible if the client has a persistent becomes immediately available to the client upon

session. reconnection:

• Session existence (even if there are no subscriptions):

The broker retains information about the existence of the
session, allowing the client to resume its previous state
Chapter 9: MQTT Persistent Sessions
upon reconnection.
and Clean Sessions
• All client’s subscriptions: The broker stores the list of

Persistent sessions in MQTT allow a client to maintain its topics to which the client has subscribed. This ensures

subscription and message state across multiple connections. the client does not need to re-subscribe to the same

When a client establishes a persistent session with an MQTT topics every time it reconnects, saving valuable time and

broker, the broker stores the client’s subscription information resources.

and any undelivered messages intended for the client. This • Flow of all messages in a Quality of Service (QoS) 1 or 2

way, if the client disconnects and reconnects later, it can where the client has not yet confirmed: The broker keeps

resume communication seamlessly. This chapter will dive track of unacknowledged messages sent to the client at

deep into how persistent sessions in MQTT enhance QoS QoS 1 or QoS 2 levels. These messages are stored in the

by enabling reliable message delivery, providing QoS level broker’s message queue and will be delivered to the client

guarantees, and facilitating efficient reconnection for clients, upon reconnection, ensuring reliable message delivery.

even in the presence of intermittent connectivity or client • All new QoS 1 or 2 messages that the client missed

disconnections. while offline: If the client was offline when QoS 1 or QoS
2 messages were published to subscribed topics, the

37

[Link]

 MQTT Essentials Ebook

broker stores these missed messages. Once the client examining the session present flag, the client can determine
reconnects, it receives the queued messages, preventing whether to establish a new session or reconnect to an existing
any loss of important information. one.
• All QoS 2 messages received from the client that
are awaiting complete acknowledgment: For QoS 2 Persistent Session on the Client Side:
messages sent by the client, the broker keeps track of Ensuring Local Message Persistence and
their acknowledgment status. If any of these messages Acknowledgment
are not fully acknowledged, the broker holds them until In addition to the broker storing a persistent session,
the acknowledgment is complete. each MQTT client also plays a role in maintaining session
continuity. When a client requests the server to retain session
Clean Sessions in MQTT and How to Use data, it assumes the responsibility and must store the
Them to Start or End a Persistent Session following information:

When establishing a connection to the broker, clients can • Flow of all messages in a QoS 1 or 2 where the broker
enable or disable a persistent session by setting the value has not yet confirmed: The client keeps track of
of the cleanSession flag. Here’s how it works: When the messages it has sent to the broker at QoS 1 or QoS 2
cleanSession flag is set to true, the client explicitly requests levels. These messages are stored locally until the broker
a non-persistent session. In this scenario, if the client acknowledges their receipt or completion. By maintaining
disconnects from the broker, all queued information and these unconfirmed messages, the client ensures that it
messages from the previous persistent session are discarded. can retransmit any messages if necessary and achieve
The client starts with a clean slate upon reconnection. the desired level of reliability.
• All QoS 2 messages received from the broker awaiting
On the other hand, when the cleanSession flag complete acknowledgment: When the broker sends
is set to false, the broker creates a persistent QoS 2 messages to the client, they are received
session for the client. This means that the broker and processed by the client. However, until the
preserves all relevant information and messages client acknowledges the successful processing of
even if the client goes offline. The session these messages, the client stores them locally. This
remains intact until the client explicitly requests ensures that the client can handle any interruptions
a clean session. If the cleanSession flag is set or disconnections while maintaining the reliability and
to false and the broker already has a session integrity of message delivery.
available for the client, it will utilize the existing • By storing these message-related details on the client
session and deliver any previously queued side, MQTT clients can actively maintain session
messages to the client upon reconnection. persistence and ensure the successful processing of
messages even in challenging network conditions.

How Does A Client Know if a Session is Already MQTT Session Management Best Practices:
Stored? Enhancing Message Delivery and Resource
Efficiency
In MQTT version 3.1.1 and beyond, the CONNACK message
sent by the broker in response to the client’s connection When working with MQTT, it’s essential to consider the
request includes a session present flag. This flag serves as best practices for session management to optimize your
an indicator for the client, informing it whether a previously implementation. Here are some guidelines to help you
established session is still available on the broker. By determine when to use a persistent session or a clean
session:

38
Best Practices for MQTT Persistent Sessions • Use Case Specifics: The appropriate solution for
managing message storage duration varies based on
• Ensure Message Reliability: If your client needs to receive
your use case. Consider factors such as the importance
all messages from a specific topic, even when they are
of retaining messages, message expiration policies, and
offline, a persistent session is the way to go. This ensures
regulatory or compliance requirements.
that the broker queues the messages for the client and
delivers them promptly when the client reconnects.
By following these best practices and considering the
• Resource Optimization: If your client has limited
above-mentioned factors, you can effectively manage MQTT
resources, leveraging a persistent session is beneficial.
sessions, optimize message persistence, and ensure reliable
Storing the client’s subscription information on the
communication in your MQTT-based solutions.
broker facilitates a quick restoration of interrupted
communication, reducing the burden on constrained
Remember to incorporate these guidelines into your
clients.
implementation based on your unique requirements and
• Resume Publishes: A persistent session is necessary if
application needs.
your client needs to resume publishing Quality of Service
(QoS) 1 and 2 messages after reconnecting. The broker
To summarize, understanding and effectively utilizing
retains these messages until the client returns online,
persistent sessions, queuing mechanisms, and proper session
ensuring their reliable delivery.
management practices, we can harness the full potential
of MQTT and build robust, scalable, and reliable IoT and
Best Practices for MQTT Clean Session
messaging applications.
• Publish-Only Clients: A clean session is suitable if your
client only needs to publish messages and does not
Chapter 10: MQTT Retained
require subscriptions to topics. In such cases, the broker
Messages
does not need to store session information or attempt to
transmit QoS 1 and 2 messages, simplifying the session A retained message is a normal MQTT message with the
management process. retained flag set to true. The broker stores the last retained
• Avoid Offline Message Retrieval: A clean session suffices message and the corresponding QoS for that topic. Each client
if your client does not need to receive missed messages that subscribes to a topic pattern that matches the topic of the
while offline. It eliminates the overhead of storing and retained message receives the retained message immediately
delivering messages that the client did not subscribe to after they subscribe. The broker stores only one retained
during its offline period. message per topic.

MQTT Message Storage Duration: How Long How are Retained Messages Different from
Does the MQTT Broker Store Messages? Normal MQTT Messages?
People often ask how long the broker stores session
information and messages. The answer depends on various Retained messages are a valuable feature in
factors and considerations: MQTT that mitigates uncertainty regarding
• Memory Limit Constraint: Typically, the primary constraint message publication. By enabling the retention
on message storage is the memory limit of the operating of the most recent message on a topic,
system hosting the broker. Monitoring and allocating subscribers can stay informed about the current
sufficient resources to handle the expected message state, even during periods of inactivity.
volume is crucial.

39

[Link]

 MQTT Essentials Ebook

In MQTT, it is important to understand that the publisher to myhome/#, Client B will receive the retained message
of a message cannot guarantee that the subscribing client for myhome/livingroom/temperature immediately after
will receive the message. The publisher’s responsibility lies subscribing to myhome/#. By recognizing the retained flag
in ensuring the safe delivery of the message to the broker. set to true, the subscribing client can process the retained
Similarly, the subscribing client cannot determine when the messages according to its requirements.
publishing client will send a new message related to their
subscribed topics. The time interval between messages can Retained messages are crucial in providing newly-subscribed
vary significantly, ranging from a few seconds to several clients with immediate status updates upon subscribing to
minutes or even hours. As a result, the subscribing client a topic, eliminating the need to wait for subsequent updates
remains unaware of the current topic status until a new from publishing clients. Essentially, a retained message
message is published. This is where retained messages play a represents the last known valid value for a particular topic. It
vital role. does not necessarily have to be the latest value, but it must be
the most recent message with the retained flag set to true.
Retained messages provide a solution to the challenge
mentioned above. When a client publishes a message with Something else important to emphasize, retained messages
the “retained” flag set to true, the broker retains the message. operate independently of persistent sessions, which we
Consequently, any client subscribing to the corresponding discussed in Part 7 of the series. Once the broker stores a
topic will receive the most recent retained message, even if no retained message, there’s only one way to remove it. We will
recent publications have occurred. discuss that shortly in an upcoming section.

In essence, retained messages offer subscribers a snapshot of Now that you understand the structure of retained messages,
the last-known state of a topic, ensuring access to the latest let’s learn more about managing them.
relevant information regardless of publishing frequency.
How to Send a Retained Message in MQTT?
Structure of a Retained Message in MQTT As a developer, sending a retained message is a
An MQTT retained message is a standard message that straightforward process. To mark a message as retained, all
includes the retained flag set to true, signifying its importance. you need to do is set the retained flag of your MQTT publish
When a client publishes a retained message, the broker stores message to true. This flag signals the broker to retain the
it along with the corresponding Quality of Service (QoS) for message and make it available to subscribers. The good news
a specific topic. Subscribing clients immediately receive the is that most MQTT client libraries provide a convenient and
retained message when they subscribe to a topic pattern that user-friendly way to enable this flag, streamlining the process.
matches the retained message’s topic. By leveraging this feature, developers can ensure that critical
information persists and remains accessible to subscribers,
It is worth noting that the broker retains only one message per even if they join the network later or experience temporary
topic. connectivity issues. Retained messages offer a powerful
mechanism for sharing important data and enabling seamless
Even if a subscribing client uses wildcards in their topic communication in MQTT-based systems.
pattern, they can still receive a retained message that may not
be an exact match for the topic. How to Delete Retained Messages in MQTT?
There is only one way to delete the retained message of a
For instance, if Client A publishes a retained message to
topic. To achieve this, simply publish a retained message with
myhome/livingroom/temperature and later Client B subscribes

40
a zero-byte payload to the topic where the retained message Closing the Loop: The Lasting Impression of
is stored. When the broker receives this special retained Retained Messages in MQTT’s Ecosystem
message, it identifies it as a request for deletion and promptly
As you can see, retained messages play a crucial role in MQTT
removes the retained message associated with that topic. As
communication by addressing the challenge of uncertain
a result, new subscribers will no longer receive the previously
message delivery and providing immediate access to the last-
retained message for that particular topic.
known state of a topic. By enabling the retention of the most
recent message on a topic, subscribers can stay informed
It’s worth noting that in many cases, explicitly deleting retained
about the current status, even during periods of inactivity.
messages may not be necessary. This is because each new
retained message automatically overwrites the previous one
Retained messages are beneficial for providing status
for the same topic. Therefore, if you publish a new retained
updates, ensuring newly-subscribed clients receive relevant
message on a topic, it will replace and supersede any existing
information without having to wait for the subsequent
retained message, effectively achieving the same outcome
message publication. By leveraging retained messages, MQTT
as deleting the previous message. This behavior ensures
empowers efficient and reliable communication between
that subscribers receive the most up-to-date and relevant
clients and brokers, enhancing the overall effectiveness of IoT
information, eliminating the need for manual deletion in most
and messaging applications.
scenarios.

Why and When Should You Use Retained Chapter 11. MQTT Last Will and
Messages? Testament (LWT)
Retained messages offer valuable benefits in various Last Will and Testament (LWT) is a powerful feature in
scenarios, particularly when you need newly-connected MQTT that allows clients to specify a message that will be
subscribers to receive messages promptly without waiting for automatically published by the broker on their behalf, if or
the next message publication. when an unexpected disconnection occurs. It provides a
reliable means of communication and ensures that clients can
This is particularly advantageous for delivering real-time gracefully handle disconnections without leaving topics in an
status updates of components or devices on specific topics. inconsistent state. This feature is particularly valuable when
For instance, let’s consider the example of a device named clients must notify others of their unavailability or convey
device1, whose status is published on the topic “myhome/ important information upon an unexpected disconnection.
devices/device1/status”. By utilizing retained messages, new
subscribers to this topic instantly receive the device’s status What is the Purpose of Last Will and
(such as online or offline) immediately after subscribing. Testament (LWT) in MQTT?
In scenarios where unreliable networks are prevalent, it
Similarly, this applies to clients that transmit data periodically,
is common for MQTT clients to experience occasional
such as temperature readings, GPS coordinates, and other
unintended breaks, which can happen due to loss of
relevant information. Without retained messages, newly-
connection or depleted batteries. Understanding the type
subscribed clients would remain unaware of the latest updates
of disconnection (graceful - with a disconnect message, or
between message intervals. By leveraging retained messages,
ungraceful - without a disconnect message) is crucial for
you can seamlessly provide connecting clients with the most
taking appropriate actions.
recent and accurate value, ensuring they have immediate
access to critical information.

41

[Link]

 MQTT Essentials Ebook

The Last Will and Testament feature in

MQTT offers a solution for clients to respond
effectively to ungraceful disconnects and
ensure proper handling of such events.

The LWT allows clients to notify others about their unexpected

disconnections. When a client connects to a broker, it can
specify a last-will message. This message follows the
structure of a regular MQTT message structure, including a
topic, retained message flag, Quality of Service (QoS), and CONNECT MQTT Packet
payload. The broker stores this message until it detects an
ungraceful disconnect from the client. Upon detecting the
When does the MQTT Broker Send the LWT
disconnection, the broker broadcasts the last will message to
Message?
all subscribed clients of the corresponding topic. The broker According to the MQTT 3.1.1 specification, the broker sends
discards the stored LWT message if the client disconnects a client’s Last Will and Testament (LWT) message in the
gracefully using the DISCONNECT message. following situations:

1. I/O error or network failure: If the broker detects any

issues with the input/output or network connection, it will
distribute the LWT message.
2. Failed communication within Keep Alive period: If the
client fails to communicate with the broker within the
specified Keep Alive period, the LWT message is sent.
In Part-10 of our MQTT Essentials, we will explore the
concept of MQTT Keep Alive time and delve into its
significance.
3. Client closes connection without DISCONNECT: When the
client terminates the network connection without sending
DISCONNECT MQTT Packet a DISCONNECT packet, the broker ensures the LWT
message is distributed.
4. Broker closes connection due to protocol error: If the
By utilizing LWT, you can implement various strategies to
broker closes the network connection due to a protocol
handle client disconnections and inform other clients about the
error, it will send the LWT message.
offline status.

Understanding when and why the broker sends the Last Will
How to Configure a Last Will and Testament
and Testament (LWT) messages lays the groundwork for
(LWT) Message for an MQTT Client?
implementing best practices in leveraging this feature, which
To specify an LWT message for an MQTT client, you include we will delve into in the next section.
it in the CONNECT message, which is used to initiate the
connection between the client and the broker.

42
When to Use Last Will and Testament (LWT) in Chapter 12: MQTT Keep Alive and
MQTT? Client Take-Over
LWT proves invaluable for alerting subscribed clients about an
Keep Alive is a feature of the MQTT protocol that allows
abrupt disconnection of a client. It becomes a powerful tool
an MQTT client to maintain its connection with a broker by
for storing and communicating client state on specific topics
sending regular control packets called PINGREQ to the broker.
when combined with retained messages.
Let’s dive into MQTT Keep Alive’s critical role in mobile
networks, and in maintaining a robust and efficient MQTT
For instance, by setting a lastWillMessage with Offline
connection.
payload, enabling the lastWillRetain flag, and specifying the
lastWillTopic as client1/status, followed by publishing an
What is MQTT Keep Alive and Why It’s
Online retained message to the same topic, client1 can keep
Important?
newly-subscribed clients informed about its online status.
Should client1 disconnect unexpectedly, the broker publishes The Keep Alive mechanism in MQTT ensures the connection’s
the LWT message with Offline payload as the new retained liveliness and provides a way for the broker to detect if a client
message, ensuring that clients subscribing to the topic while becomes unresponsive or disconnected.
client1 is offline receive the LWT message and stay up to date
on its current status. When a client establishes a connection with an MQTT broker,
it negotiates a Keep Alive value, which is a time interval
LWT not only notifies subscribed clients about unexpected expressed in seconds. The client must send a PINGREQ packet
disconnections but also assists in maintaining the system’s to the broker at least once within this interval to indicate its
integrity by providing valuable information on client states. presence and keep the connection alive. Upon receiving a
Combining LWT with retained messages allows you to create PINGREQ packet, the broker responds with a PINGRESP packet,
a robust solution that stores and communicates the latest confirming that the connection is still active.
client state on specific topics, ensuring reliable updates for
all subscribers. This approach enables seamless integration The MQTT Keep Alive mechanism is important
and synchronization between clients, enhancing the overall for connection monitoring, efficient resource
resilience and functionality of the MQTT network. utilization, detecting network failures, graceful
disconnection, etc.
The Importance of Last Will and Testament in
MQTT Now, let’s establish why the kept alive feature is so important
by delving into the issue of half-open TCP connections and
The Last Will and Testament (LWT) feature in MQTT is crucial
how it poses a challenge within MQTT, particularly in mobile
in ensuring efficient communication and maintaining system
networks.
integrity in the event of unexpected client disconnections.
By combining LWT with retained messages, developers
The Problem of Half-Open TCP Connections in MQTT
can store and communicate client state on specific topics,
providing valuable information to subscribed clients. LWT The problem of half-open TCP connections arises within
empowers MQTT networks with enhanced resilience, seamless MQTT, which relies on the Transmission Control Protocol
integration, and reliable updates, making it a powerful tool (TCP) to ensure “reliable, ordered, and error-checked” packet
for various applications. By understanding the benefits and transfer over the internet. Despite TCP’s robustness, there are
best practices of LWT, you can leverage this feature to create instances where the synchronization between communicating
robust and effective MQTT solutions. parties can falter due to crashes or transmission errors.

43

[Link]

 MQTT Essentials Ebook

In TCP, this state of an incomplete connection is referred to as "The Keep Alive ... is the maximum time
a half-open connection, where one side of the communication interval permitted to elapse between the point
remains unaware of the other side’s failure. The connected at which the Client finishes transmitting one
side persistently attempts to send messages while eagerly Control Packet and the point it starts sending
awaiting acknowledgments. the next. It is the responsibility of the Client
to ensure that the interval between Control
Andy Stanford-Clark, the inventor of the MQTT protocol, Packets being sent does not exceed the Keep
highlights that the issue of half-open connections becomes Alive value. In the absence of sending any
more pronounced in mobile networks. While TCP/IP other Control Packets, the Client MUST send a
theoretically notifies users when a socket breaks, practical PINGREQ Packet."
scenarios, particularly on mobile and satellite links, involve the
“faking” of TCP over the air with added headers at each end.
This practice can lead to a phenomenon known as a “black As long as messages are transmitted frequently within the

hole” TCP session, where the connection appears open but, in Keep Alive interval, there is no need to send an additional

reality, discards any transmitted data. message to verify the connection status. However, if the
client remains inactive during the Keep Alive period, it must
send a PINGREQ packet to the broker as a confirmation of its
availability and to ensure that the broker is still accessible.
"Although TCP/IP, in theory, notifies you when a socket
breaks, in practice, particularly on things like mobile and
If a client fails to send any messages or a PINGREQ packet
satellite links, which often “fake” TCP over the air and put
within one and a half times the Keep Alive interval, the broker
headers back on at each end, it’s quite possible for a TCP
is responsible for disconnecting the client. Likewise, the client
session to “black hole,” i.e. it appears to be open still, but
should close the connection if it does not receive a response
is just dumping anything you write to it onto the floor."
from the broker within a reasonable timeframe.

Andy Stanford-Clark on the topic "Why is the keep-alive

By employing the Keep Alive mechanism, MQTT enhances
needed?" Source
connection stability, mitigates the risks associated with half-
open connections, and facilitates efficient communication
between brokers and clients in various network conditions.
How Does MQTT Keep Alive Work?
To address the challenge of half-open connections and How Does Keep Alive Ensure Connection
enable continuous assessment of connection status, MQTT Vitality?
incorporates a vital feature called Keep Alive. This mechanism
Let’s examine the flow of Keep Alive messages to gain a
guarantees that the connection between the MQTT broker
deeper understanding of the Keep Alive mechanism. The Keep
and client remains active and that both parties know their
Alive feature utilizes two packets: PINGREQ and PINGRESP.
connection status.

What is PINGREQ in MQTT Keep Alive?

When a client establishes a connection with the broker, it
specifies a time interval in seconds known as the Keep Alive
duration. This duration sets the maximum allowed time gap
during which the broker and client may not exchange any
communication. According to the MQTT specification, the
Keep Alive interval is defined as follows:

44
When a client wants to signal its continued presence and 12 minutes, and 15 seconds. Conversely, setting the Keep
activity to the broker, it sends a PINGREQ packet. This packet Alive interval to 0 effectively deactivates the Keep Alive
is a “heartbeat” message indicating that the client is still mechanism, removing its influence on the connection’s
alive. If the client has not sent any other type of packet, such stability and management.
as a PUBLISH or SUBSCRIBE packet, it must send a PINGREQ
packet to the broker. The client can choose to send a PINGREQ What is Client Take-Over in MQTT?
packet at any time to verify the ongoing vitality of the network
In the MQTT protocol, when a client becomes disconnected, it
connection. Notably, the PINGREQ packet does not contain any
typically attempts to reconnect. However, there are instances
payload.
when the broker still maintains a half-open connection for that
client. In such cases, when a client, which the MQTT broker
What is PINGRESP in MQTT Keep Alive?
perceives as online, initiates a reconnection and performs a
`client take-over, the broker takes necessary action. It promptly
terminates the previous connection associated with the same
client (identified by the client identifier) and establishes a fresh
connection with the client. This intelligent behavior ensures
that half-open connections do not impede the disconnected
client from successfully re-establishing its connection. By
seamlessly managing client take-over, MQTT guarantees
smooth connectivity and resilient communication in the face of
intermittent network interruptions.

How Does Keep Alive and Client Take-Over

Upon receiving a PINGREQ packet from a client, the broker is Enhance MQTT Performance?
obligated to respond with a PINGRESP packet. The PINGRESP The Keep Alive feature and client take-over mechanism are
packet serves as an acknowledgment from the broker to the vital components of MQTT that ensure reliable and efficient
client, confirming its availability and continued connection. communication in various scenarios. By implementing Keep
Like the PINGREQ packet, the PINGRESP packet does not Alive messages through PINGREQ and PINGRESP packets,
include a payload. MQTT enables clients to actively signal their presence and
verify network connectivity. This mechanism prevents half-
How Can I Customize Keep Alive Settings for open connections and allows for timely detection of inactive or
Optimal Performance? lost connections.
• If the broker does not receive a PINGREQ packet or any
other packet from a client within the expected time frame, Furthermore, client take-over facilitates seamless
the broker will close the connection and dispatch the Last reconnection for disconnected clients. When a client initiates
Will and Testament message (LWT) message if the client a reconnection, the MQTT broker intelligently closes any
has specified one. existing half-open connection associated with that client and
• The MQTT client is responsible for setting an appropriate establishes a fresh connection. This process ensures that
Keep Alive value. For instance, the client can adjust the disconnections do not hinder the client’s ability to regain
keep-alive interval based on its current signal strength, connectivity and resume communication smoothly.
optimizing the connection for its specific circumstances.
• Importantly, the maximum Keep Alive interval is 18 hours,

45

[Link]

 MQTT Essentials Ebook

MQTT clients must set appropriate Keep Alive values, considering factors such as signal strength and network conditions. This
allows for optimal management of the Keep Alive mechanism and ensures efficient resource utilization.

Understanding the intricacies of the Keep Alive feature and client take-over in MQTT empowers developers to build robust and
resilient MQTT applications. By leveraging these capabilities, MQTT facilitates the creation of reliable, real-time IoT and messaging
solutions that thrive even in challenging network environments.

Chapter 13: Introduction to MQTT 5 Protocol

MQTT has solidified its footprint by connecting numerous constrained devices across multiple deployments, establishing a vast
network of connected systems and standalone devices. From connected cars and manufacturing systems, logistics to enterprise
chat applications, and mobile apps, MQTT’s widespread use has spurred demands for its evolution. MQTT 5 answers this call,
promising an extensive array of exciting new features and improvements.

In this chapter, we provide an in-depth exploration of MQTT Unlocking the Power of IoT with MQTT 5: New
5. We discuss topics ranging from foundational changes in Features and Improvements
the protocol to user properties, shared subscriptions, payload
One of the significant improvements in MQTT 5 is the
format description, request-response pattern, topic alias,
introduction of enhanced authentication mechanisms. These
enhanced authentication, and flow control.
provide a more robust security framework critical in today’s
world, where the risk of cyber-attacks is always looming. With
Design Goals of MQTT 5
MQTT 5, users have more options for securing their devices
The OASIS Technical Committee (TC), tasked with defining and data, including using more sophisticated encryption
and standardizing MQTT, faced the complex challenge of algorithms and key management techniques.
adding long-desired features without increasing overhead or
decreasing ease of use. They sought to improve performance In addition, MQTT 5 introduced Shared Subscriptions, a feature
and scalability without inducing unnecessary complexity. After that allows load balancing of messages across multiple
much deliberation, the TC decided on a host of functional client instances. This ensures that people can manage many
objectives for MQTT 5, aimed at enhancing scalability, messages effectively without overloading individual clients.
formalizing common patterns such as capability discovery and The shared subscriptions feature is practical in scenarios
request-response, and enabling extensibility mechanisms like where many devices are transmitting data concurrently.
user properties.

46
Further, MQTT 5 also introduces the concept of message communication in the foreseeable future.
properties, allowing additional metadata to be included with
each message. This is useful for providing context about the As technology continues to evolve, and as the needs of IoT
transmitted data, such as timestamps, location information, or systems grow and change, so will MQTT. It’s critical for
device status. anyone working in the IoT space to stay abreast of these
developments to ensure they are leveraging the full power
As described, the transition from MQTT 3.1.1 to MQTT 5 was of MQTT and other related technologies to deliver the most
not a simple version number update but rather a significant effective, reliable, and secure IoT solutions possible.
leap in the protocol’s capabilities, addressing several areas
of improvement. The result is a more robust, reliable, and What Properties Are Available in MQTT 5 Header &
scalable protocol better suited to modern IoT applications’ Reason Codes?
needs. Undoubtedly, one of the most transformative and flexible
features introduced in MQTT 5 is the ability to incorporate
It’s important to understand these changes and improvements custom key-value properties in the MQTT header. This
when considering MQTT for IoT applications, as they can capability can potentially revolutionize many deployments,
significantly affect the performance and reliability of your IoT and is of such significance that it warrants an entire chapter
system. So, whether you’re a developer, a systems integrator, dedicated to it. Like protocols such as HTTP, MQTT clients and
or an end user, the step to MQTT 5 is a key advancement in brokers can add an unlimited number of custom or pre-defined
evolving IoT communication protocols. headers to carry metadata. This metadata is adaptable and
can be tailored to specific application data, with pre-defined
While MQTT 5 brings numerous benefits, it also headers predominantly employed in the execution of most new
demands careful implementation. Like any MQTT features.
technology upgrade, it is critical to evaluate
the implications on your specific use case Additionally, MQTT packets now include Reason Codes,
and ensure the transition is controlled and signifying the occurrence of a pre-defined protocol error.
managed. Understanding the benefits and Traditionally found on acknowledgment packets, these Reason
potential challenges of MQTT 5 will help ensure Codes facilitate error interpretation and potentially aid in
a successful implementation, maximizing devising remedies by both clients and brokers.
the potential benefits of this powerful IoT
communication protocol. Occasionally referred to as Negative Acknowledgments, these
Reason Codes can be found on a range of MQTT packets, such
As we look to the future, MQTT remains a vital player in IoT. as:
With MQTT 5, developers now have even more flexibility in
• CONNACK
designing and implementing solutions that can handle the
• PUBACK
diverse requirements of modern IoT systems.
• PUBREC
• PUBREL
The evolution of MQTT doesn’t stop at version 5. The
• PUBCOMP
MQTT Technical Committee is already working on further
• SUBACK
enhancements and features that will continue to advance the
• UNSUBACK
protocol’s capabilities. This ongoing work demonstrates the
• AUTH
strong commitment of the MQTT community to the continued
• DISCONNECT
growth and advancement of this technology, ensuring
that MQTT will remain a relevant and powerful tool for IoT
47

[Link]

 MQTT Essentials Ebook

The range of Reason Codes for Negative Acknowledgments wholly conformed to the MQTT specification, comprehensively
varies widely, spanning from “Quota Exceeded” to “Protocol supporting all features. Pivoting toward the subject of
Error”. It is the joint responsibility of clients and brokers to unsupported features, MQTT 5 presents an ingenious
decode and comprehend these newly introduced Reason resolution. It enables those implementations that are not
Codes, further enriching the overall MQTT experience. entirely complete, often seen in SaaS offerings, to indicate the
broker’s inability to support certain features. The onus is then
Having established the custom key-value properties and placed upon the client to ensure these unsupported features
Reason Codes, let’s explore how MQTT handles unsupported are not utilized. The broker employs pre-defined headers in the
features and uses CONNACK return codes. CONNACK packet, sent in response to the client’s CONNECT
packet, to communicate the lack of support for specific
Responding to Unsupported Features with features. These headers can also be leveraged to notify the
CONNACK Return Codes client that they are not permitted to use certain features.

As MQTT’s popularity surged, various MQTT implementations

began to emerge, proposed by a diverse range of companies.
What are the Pre-Defined Headers Available in
However, not all these implementations adhere strictly to
MQTT 5?
the complete MQTT specification. Certain features, such as Here are the pre-defined headers available in MQTT v5 for
QoS 2, retained messages or persistent sessions may not indicating unimplemented features or features not authorized
consistently be implemented. In contrast, HiveMQ remains for client use:

Pre-Defined Header Data Type Description

Retain Available Boolean Are retained messages available?

The maximum QoS the client is allowed to use for

Maximum QoS Number
publishing messages or subscribing to topics

Wildcard available Boolean If Wildcards can be used for topic subscriptions

If Subscription Identifiers are available for the MQTT

Subscription identifiers available Boolean
client

If Shared Subscriptions are available for the MQTT

Shared Subscriptions available Boolean
client

Defines the maximum message size a MQTT client

Maximum Message Size Number
can use

The Keep Alive Interval the server supports for the

Server Keep Alive Number
individual client

48
These return codes represent a significant stride forward persistent session concept. In MQTT 5, all sessions persist
in expressing the permissions of individual MQTT clients. unless the “Session Expiry Interval” is set to 0. Session
However, this new capability carries an inherent trade-off; deletion takes place either after the interval timeout or when
MQTT clients must implement the interpretation of these the client reconnects using Clean Start.
codes independently, and must ensure that application
developers avoid using features unsupported by the broker, or AUTH Packet in MQTT 5?
those the client lacks permission for.
MQTT 5 brings forth a new packet type: the AUTH
packet. Serving as a vital tool in implementing non-trivial
As a note of assurance, HiveMQ is fully
authentication mechanisms, we anticipate this packet to be
compliant with all MQTT 5 features, ensuring
integral in production environments. It’s crucial to understand
these custom headers would only be utilized
that this novel packet can be dispatched by both brokers and
at the administrator’s discretion for setting
clients post-connection establishment. It enables the use of
permissions in deployments.
intricate challenge/response authentication methods, such
as SCRAM or Kerberos as outlined in the SASL framework
and is also compatible with cutting-edge IoT authentication
Enhanced Session Management: From Clean methods like OAuth. Importantly, the AUTH packet allows for
Session to Clean Start in MQTT 5 the re-authentication of MQTT clients without requiring the

The concept of “Clean Session,” a prominent feature in termination of the connection.

MQTT 3.1.1, is now succeeded by “Clean Start” in MQTT v5.

The Clean Session feature was highly used in MQTT 3.1.1
MQTT 5: Enriching MQTT with UTF-8 String
by clients that either had temporary connections or didn’t
Pairs
subscribe to messages. Upon connecting to the broker, the To accommodate the custom headers, a new data type was
client was required to send a CONNECT packet with either the necessitated, and thus, the UTF-8 string pairs were introduced.
Clean Session flag enabled or disabled. If enabled, it indicated Essentially, a string pair is a key-value structure wherein both
to the broker that all client data should be discarded when the the key and value are of the String data type. At present, this
underlying TCP connection was severed or upon the client’s data type is primarily used for custom headers.
decision to disconnect from the broker. Moreover, if a previous
session was associated with the client identifier on the broker, This novel addition expands the spectrum of MQTT data types
a Clean Session CONNECT packet compelled the broker to utilized on the wire to a total of seven:
discard the prior data.
1. Bit
2. Two Byte Integer
MQTT 5 introduces the Clean Start option, signaled by the
3. Four Byte Integer
Clean Start flag in the CONNECT message. With this flag, the
4. UTF-8 Encoded String
broker dismisses any prior session data, and the client initiates
5. Variable Byte Integer
a new session. However, the session isn’t automatically
6. Binary Data
cleared when the TCP connection is closed between the client
7. UTF-8 String Pair
and the server. To prompt the deletion of the session post-
disconnection, a new header field, Session Expiry Interval,
For most application users, Binary Data and UTF-8
must be set to 0.
Encoded Strings remain the go-to data types within
their MQTT library APIs. However, with the onset
This revised Clean Start functionality enhances and simplifies
of MQTT 5, the UTF-8 String Pairs are anticipated
MQTT’s session handling, offering more flexibility and easier
to gain frequent usage.
implementation compared to the former Clean Session/
49

[Link]

 MQTT Essentials Ebook

While the remaining data types might not be directly visible to already burdened MQTT clients. Consider a scenario where an
users, they are leveraged on the wire to construct valid MQTT MQTT client takes 11 seconds to process a message received
packets by MQTT client libraries and brokers. from a broker to acknowledge the packet after processing.
There is no tangible advantage if the broker retransmits the
How Does MQTT 5 Enhance Communication message after a 10-second timeout. This approach merely
with Bi-directional DISCONNECT Packets? consumes valuable bandwidth and further overwhelms the

Under the MQTT 3.1.1 framework, clients could gracefully MQTT client.

terminate their connection by dispatching a DISCONNECT

packet before closing the underlying TCP connection. With the advent of MQTT 5, retransmission of MQTT messages

However, there was no provision for the MQTT broker to inform for healthy TCP connections is disallowed for both brokers

the client of a potential issue that necessitates closing the and clients. However, brokers and clients must resend

TCP connection. This shortcoming has been addressed in the unacknowledged packets when the TCP connection has been

new protocol version. severed. Thus, the QoS 1 and 2 guarantees remain as vital as
they were in MQTT 3.1.1.

In the enhanced MQTT, the broker is authorized to transmit a

DISCONNECT packet before severing the socket connection. If your use case depends on retransmission packets (for

This provision empowers the client to understand the cause instance, if your implementation fails to acknowledge packets

behind the disconnection and devise a suitable response under certain circumstances), we recommend reevaluating this

accordingly. While the broker is not mandated to disclose the strategy before upgrading to MQTT v5.

exact reason (for security purposes, for instance), this feature

benefits developers as it provides valuable insight into why the
How Does MQTT 5 Simplify Authentication?
broker terminated a connection. In the MQTT 3.1.1 protocol, MQTT clients needed to provide
a username when utilizing a password in the CONNECT
Another valuable addition is that DISCONNECT packets can packet. This proved inconvenient for some use cases where
now carry Reason Codes, simplifying the process of revealing a username was unnecessary. One prominent example
the disconnection’s rationale, such as in the case of invalid includes using OAuth, which leverages a JSON web token
permissions. for authentication and authorization information. With MQTT
3.1.1, static usernames were often utilized given that the
Revamping QoS 1 and 2 in MQTT 5: critical information was contained within the password field.
Eliminating Retries for Healthy Connections
MQTT employs persistent TCP connections (or similar With the advent of MQTT 5, the protocol has introduced more
protocols with identical assurances) for the underlying refined ways to handle tokens, for instance, via the AUTH
transport. A robust TCP connection ensures bi-directional packet. However, using the password field of the CONNECT
connectivity with exactly-once and in-order guarantees, packet is still feasible. The major improvement here is
meaning all MQTT packets dispatched by clients or brokers that users can simply leverage the password field without
will be received at the other end. When the TCP connection is the obligation to fill out the username. This adjustment
disrupted while the message is in transit, QoS 1 and 2 ensure offers a more streamlined and straightforward approach to
message delivery across multiple TCP connections. authentication in specific scenarios.

Under the MQTT 3.1.1 protocol, re-delivery of MQTT messages While the core of the MQTT protocol remains relatively
was permissible even when the TCP connection was healthy. unchanged, subtle refinements have been implemented under
This often proved detrimental in practice, risking overloading the surface, laying the groundwork for many new features

50
in version 5 of this widely adopted IoT protocol. As a user 6. Streamlined Migration
leveraging MQTT libraries, these modifications may seem 7. Improved features, including
minor and do not radically alter the way MQTT is utilized. 1. Negative Acknowledgments
However, for developers working on MQTT libraries and [Link] Aliases for Efficiency
brokers, these changes, particularly those related to protocol 3. User Properties for Customization
nuances, are critical and demand attention. 4. Payload Format Indicators

Some shifts in specified behavior, such as message Here is the rationale behind how and why MQTT 5 is better
retransmission, necessitate revisiting deployment design compared to MQTT 3.
decisions during the transition to MQTT 5.
1. How MQTT 5 is Better Than MQTT 3 in Error
Chapter 14: Key Reasons to Upgrade Handling
to MQTT 5 from MQTT 3.1.1 MQTT 5 significantly enhances error handling mechanisms,
contributing to the robustness and reliability of the system.
The Internet of Things (IoT) has evolved rapidly over the last One of the notable introductions is the session and message
decade, with MQTT emerging as the de facto protocol for the expiry feature. This allows developers to set a defined time
seamless and efficient communication of IoT devices. As the limit for each message and session. If a message isn’t
scale and complexity of IoT systems grow, the MQTT protocol delivered within this time frame, it’s automatically deleted. This
is also evolving to meet these demands. In this light, the MQTT feature is particularly useful in ensuring that network latency
5 upgrade presents significant improvements catering to or outages do not lead to delivering outdated or irrelevant
modern IoT applications’ expanding needs. commands to IoT devices.

Now, let’s explore why your organization or development team 2. How MQTT 5 is Better Than MQTT 3 in
should consider upgrading to MQTT 5 from MQTT 3.1.1. Offering Greater Scalability
Scalability is a crucial requirement in the modern world of
Why Upgrade to MQTT 5 from MQTT 3?
growing IoT networks. MQTT 5 addresses this by standardizing
MQTT 5 represents a substantial enhancement to the MQTT shared subscriptions. This allows multiple MQTT client
protocol, developed with valuable insights from MQTT instances to share the same subscription on the broker. It’s a
users. It incorporates features essential for contemporary powerful feature for load-balancing MQTT clients deployed on
IoT applications, tailor-made for cloud-based deployments. a cloud cluster. This scalability makes MQTT 5 a robust choice
These advancements promote resilience, dependable error for enterprises with large deployments and those planning to
management for executing crucial messaging, and facilitate scale up their IoT systems.
the straightforward integration of MQTT messages into
existing computational frameworks. 3. How MQTT 5 is Better Than MQTT 3
in Offering Greater Flexibility and Easier
In this chapter, we present seven reasons why your Integration
organization or development team should consider upgrading
Taking customization to a new level, MQTT 5 introduces
to MQTT 5; they include:
User Properties, a feature that allows for adding key-
1. Better Error Handling for More Robust Systems value properties to the headers of MQTT messages. This
2. More Scalability for Cloud Native Computing capacity means that application-specific information can
3. Greater Flexibility and Easier Integration be incorporated directly into each message, enhancing the
4. The Rise of a Single IoT Standard processing of these messages. For instance, a meta-tag that
5. Future-Proofing includes the unique identifier of the sending client or the
51

[Link]

 MQTT Essentials Ebook

firmware version of the device platform can be added to the

message header, facilitating analysis and processing by the 7. How is MQTT 5 Better Than MQTT 3 in
receiver. Offering Improved Features
In this section, we delve into an array of MQTT5’s enhanced
Additionally, MQTT 5 simplifies the process for the receiver features. From bolstering system robustness with Negative
by incorporating Payload Format Indicators. Now it’s easier to Acknowledgments to enabling customization with User
differentiate between binary or text data, as MQTT 5 includes Properties and boosting efficiency with Topic Aliases, MQTT
a MIME-style content type descriptor. This feature provides 5 has significantly emphasized usability, flexibility, and
immense value across a plethora of use cases. Consider, for performance. Moreover, Payload Format Indicators have been
example, a toll road control system transmitting images of introduced to facilitate easy handling of diverse data types.
license plates for image recognition processing. In contrast, Let’s explore each of these advancements in detail:
other messages might require a different processing style,
like those including location coordinates. By specifying the Negative Acknowledgments
format, MQTT 5 ensures efficient and appropriate processing
In a further bid to bolster system robustness, MQTT 5
of diverse data types.
incorporates the concept of negative acknowledgments.
The broker can reject specific messages under predefined
4. How MQTT 5 is Better Than MQTT 3 as an
conditions or restrictions, such as exceeding the maximum
IoT Standard
message size, maximum quality of service (QoS), or using
With its rich feature set, MQTT 5 has cemented its place
unsupported features. This proactive measure guards against
as the go-to choice for diverse IoT use cases, successfully
MQTT clients that might start sending erroneous or malicious
addressing the limitations of previous versions. We anticipate
messages, significantly enhancing the overall system’s
exponential MQTT adoption across industries in the coming
security.
years, hinting at MQTT’s impending status as the universal IoT
standard. Topic Aliases for Efficiency
In large systems with complex topic structures, topic strings
5. How MQTT 5 is Better Than MQTT 3 in
Future-Proofing Your Applications can become long, increasing network and processing load.
MQTT 5 introduces topic aliases, which lets you replace
By addressing the limitations of MQTT 3, MQTT 5 paves the
these long topic strings with integer values. This can
way for future enhancements in IoT applications. Its flexible
considerably reduce the demand on the network and the
and robust features make it adaptable to upcoming technology
processing overhead, thereby boosting system efficiency and
trends, ensuring your IoT systems stay current and ready for
performance.
future advancements.

User Properties for Customization

6. How MQTT 5 is Compatible With its
Predecessors MQTT 5 goes a step further in customization by introducing
User Properties. These allow developers to add key-value
One of the practical advantages of MQTT 5 is its compatibility
properties to the message header of an MQTT message. This
with its predecessors. It supports MQTT 3.1 and MQTT 3.1.1
level of customization enables you to include application-
features, allowing a mix of MQTT 3 and MQTT 5 clients. This
specific information within each message. This data could
makes the migration process seamless, enabling organizations
be used to enrich message processing and integration, giving
to gradually transition to MQTT 5 without disrupting existing
developers more flexibility and control over their applications.
IoT operations.

52
Payload Format Indicators
With the diverse types of data in IoT applications, it’s crucial
to have a mechanism that simplifies data handling and
processing. MQTT 5 meets this need with payload format
indicators. These indicators identify whether the payload is
binary or text and include a MIME-style content type. This
helps improve data processing efficiency and makes the
system more adaptable to various data use cases.

Chapter 15: MQTT Session Expiry and

Message Expiry Intervals The Session Expiry Interval can be
defined in the connect packet
In this chapter, we focus on two closely-related features: the
Session Expiry Interval and the Message Expiry Interval. Message Expiry Interval in MQTT 5
These functionalities represent key improvements in MQTT 5,
Within MQTT 5, clients can set a unique Message Expiry
and a thorough understanding of them is crucial for optimal
Interval in seconds for each PUBLISH message. This interval
protocol utilization.
establishes the duration the broker preserves the PUBLISH
message for subscribers that match the topic but are
How do Session Expiry Interval and Message
currently offline. If the interval isn’t defined, the broker must
Expiry Interval Work in MQTT 5?
indefinitely hold the message for matching subscribers, yet
Let’s break down and analyze these two integral expiry interval disconnected subscribers. Furthermore, if the ‘retained=true’
features individually. option is selected during the PUBLISH message, the interval
also dictates the length of time a message is retained on a
Session Expiry Interval in MQTT 5
particular topic.
The Session Expiry Interval is a parameter a client sets
during the CONNECT packet stage, specified in seconds. This
parameter indicates the duration the broker retains the client’s
session information. If this interval is set to zero, or if the
CONNECT packet does not specify an expiry value, the session
data is promptly deleted from the broker as soon as the client’s
network connection terminates.

Notably, the maximum session expiry interval is UINT_

MAX (4,294,967,295), enabling an offline session to Publish Packet with Message Expiry Interval set to 120 seconds.
persist for an extended duration of just over 136 years The retainFlag is set to "true", so the message will also be
following client disconnection. retained for 120 seconds.

53

[Link]

 MQTT Essentials Ebook

Why Were the Expiry Intervals Introduced in the session to be discarded would have to connect with the
MQTT 5? cleanSession=true flag set.

Understanding the functionality of these expiry intervals

Consider scenarios where some IoT devices never reconnect
merely scratches the surface. It’s also essential to explore the
due to decommissioning, destruction, or leftover sessions
reasons that prompted incorporating these features into the
from inadequate cleanups post-load-testing. These residual
MQTT 5 specification, along with their practical applications.
sessions could impose an unnecessary burden on a broker’s
persistence. Enterprise-grade MQTT brokers like HiveMQ
In earlier chapters, we captured the spirit of the OASIS
come equipped with sophisticated administrative tools such
committee’s deliberation process, which culminated in the
as the HiveMQ Control Center to facilitate the management of
establishment of MQTT 5 as a standard in March 2019.
idle sessions. Nonetheless, it is still crucial to discern which
HiveMQ, a proud member of this committee, actively gathered
sessions can be effectively removed.
user feedback, comprehended the needs of long-standing
users of the MQTT 3.1 and 3.1.1 versions, and determined
Enter the session expiry interval feature of MQTT
how best to evolve the protocol. The main driving force was to
5. This intuitive feature enables users to specify
introduce features that expanded the realm of possibilities for
a sensible duration, after which the broker
users and enhanced the simplicity of their interactions with the
automatically purges an idle session, liberating
protocol.
valuable resources.

MQTT 5’s Session Expiry Interval

Beyond this automated cleanup capability, the session expiry
The Session Expiry Interval in MQTT 5 masterfully fulfills two interval has significantly simplified session state management.
critical needs simultaneously. Earlier versions of MQTT offered Ian Craggs graciously provided two diagrams illustrating
a single avenue for removing persistent sessions as defined by the reduction in complexity in state transitions. This visual
the specification: a client bearing the same client ID as representation underscores how this new feature aids in
streamlining state transitions and bolstering user efficiency.

54
 De-cluttering of state transition in MQTT 3.1.1 and MQTT 5 (courtesy of Ian Craggs)

MQTT 5’s Message Expiry Interval The longevity of message relevance to IoT devices can exhibit
significant fluctuations. Take the example of the connected
Much like its counterpart, the session expiry’s interval
car: traffic updates maintain their relevance only briefly. Yet,
inception was fueled by a need for an automated maintenance
when we consider over-the-air firmware upgrades, these need
mechanism. Consider the myriad of IoT devices, such as
to be executed even if the car remains offline for an extended
connected cars designed to weather extended periods of
period, stretching into several weeks.
internet disconnection. MQTT offers a lifeline for these
scenarios with persistent sessions and message queueing.
The Message Expiry Interval in MQTT 5 emerges
as the perfect tool to manage these differing
Messages crafted for offline devices are stored on the broker,
time frames, adding to the protocol’s versatility.
waiting for the device to regain connectivity for delivery.
In grand-scale deployments, where the count of connected
devices escalates into thousands or even millions, it’s
imperative to constrain the offline message queue for each
client individually.
55

[Link]

 MQTT Essentials Ebook

Example: traffic jam alerts usually become obsolete after 1-2 hours, but a firmware update should be available for many weeks. Message
Expiry Interval is the perfect feature to define these different periods of time.

By assigning an optimal message expiry interval to messages Important Tips for MQTT 5 Session Expiry
with a limited relevance window and leaving perpetual Interval and the Message Expiry Interval
relevance messages without an expiry, we ensure effective
Here’re some important information:
utilization of broker resources for clients that could be
offline for a considerable duration. This strategy also spares • Both the Session Expiry Interval and the Message Expiry
clients from being inundated with irrelevant messages upon Interval are instrumental in refining resource management
reconnection. on the MQTT broker.
• Reflecting on the past, many MQTT 3 users expressed
For retained messages, the message expiry a need for expiry features. HiveMQ responded to this
interval operates similarly, guaranteeing that demand by introducing session and message expiry as
these messages are only dispatched to new supplementary features in our MQTT platform, predating
subscribers for a specified period. their standardization in MQTT 5.
• The equivalent for cleanSession=true in the CONNECT
packet of MQTT 3 is sessionExpiry=0 (or absent) and
One crucial aspect to note, however, is that when a client’s cleanStart=true in MQTT 5.
session expires, all queued messages for that client also • Likewise, cleanSession=false from the MQTT 3 CONNECT
expire in sync with the session, irrespective of their individual packet finds its counterpart in MQTT 5 as a sessionExpire
message expiry status. This “GOTCHA” is an important value exceeding zero, coupled with cleanStart=false.
reminder of the interconnectedness of sessions and their • MQTT brokers, such as HiveMQ, offer the capability to
queued messages in the MQTT protocol. configure a maximum value for these expiry intervals
on the server side. This feature is handy in multi-vendor
projects, particularly when the broker operator may lack
authority over the MQTT client settings.

56
The advent of MQTT 5 has ushered in a host of new features transparency and allow centralized system control by virtue of
designed to enhance the protocol’s usability, flexibility, and the broker.
efficiency. The Session Expiry Interval and the Message Expiry
Interval are prime examples of this, acting as invaluable tools Feedback on Connection Establishment in
in resource management and ensuring the smooth functioning MQTT 5
of your MQTT broker. Both these features truly embody the With MQTT version 5, it is now possible for MQTT brokers
user-centric evolution of the MQTT standard, demonstrating its to give additional feedback to MQTT clients on connection
responsiveness to the demands and challenges of its users. establishment. A number of different properties can be added
to the connection acknowledgment packet that tells the client
Chapter 16: MQTT 5’s Improved Client which features the broker supports or the client is allowed to
Feedback & Negative ACKs use. This includes the following MQTT features:

• Retained messages
In this chapter, we focus on MQTT 5’s Enhanced Client
• Wildcard subscriptions
Feedback and Negative ACKs(NACKs). We will dissect
• Subscription identifiers
this method that MQTT 5 implements for MQTT brokers to
• Shared subscriptions
acknowledge clients, examining their potential to simplify the
• Topic aliases
work of developers and operations teams, and making the
• Maximum quality of service level the client can use
protocol more robust and efficient.

In addition to notifying the client about enabled and disabled

Who Needs More Client Feedback While Using
MQTT? features, the new properties in the CONNACK packet also allow
the server to give the client feedback on the limits granted to it
Over the years, MQTT has become a staple in numerous IoT
by the broker. These limits include:
projects. As noted in this eBook earlier, the OASIS committee
thoroughly considered the feedback from actual protocol users • Keep Alive

while crafting the new MQTT 5 protocol standard. Among the • Session expiry interval

predominant grievances was a glaring lack of transparency, • Maximum packet size

primarily due to the insufficient return codes and limited means • Maximum number of topic aliases the client can send

to communicate specific limitations or circumstances from

the broker to the client. This deficiency resulted in increased Beyond supporting all of these limits, HiveMQ allows you

debugging complexity, particularly in multi-vendor projects. to configure a maximum for the limits and to disable MQTT
features that are not needed in your use case.

Whether it’s delving into the causes of client disconnects,

examining why messages fail to reach their designated targets,
Better Reason Codes in MQTT 5
or guaranteeing consistency in MQTT client deployments In previous iterations, namely MQTT version 3.1 and 3.1.1, the
across various teams, MQTT 3 users often find the need to assortment of available reason codes was somewhat limited,
augment the basic protocol features with technology, such with only five codes pertaining to unsuccessful operations.
as the HiveMQ Extension SDK. MQTT 5 has been deliberately However, MQTT 5 substantially enhances this aspect, offering
designed to address these challenges more efficiently and in a an expanded set of over 20 reason codes dedicated to
standardized manner by introducing the following features. unsuccessful scenarios.

MQTT 5’s Specific Feature Set Moreover, MQTT 5 introduces the possibility of integrating

Let’s focus on several MQTT 5 features that provide more reason codes into packets that previously lacked this attribute
in versions 3.1 and 3.1.1. These packets include UNSUBACK,
57

[Link]

 MQTT Essentials Ebook

PUBACK, PUBREC, PUBREL, DISCONNECT, and PUBCOMP. This a comprehensive understanding of why the connection was
enhancement further strengthens the protocol’s transparency severed.
and troubleshooting efficiency, illustrating the forward strides
made in MQTT 5. This streamlined method of connection termination not only
adds clarity for the client but also substantially simplifies
Enhancing Clarity with Contextual Reason pinpointing the reason behind a broker-initiated connection
Strings in MQTT 5 closure. This significant enhancement in MQTT 5 dramatically

While adding additional reason codes certainly bolsters the improves the communication transparency between broker and

feedback quality to clients, these codes often fall short in client.

providing specific contexts. This is where MQTT 5 introduces

the concept of ‘reason strings’ to bridge the gap, described in
What are Negative Acknowledgments in
the specifications as, “… a human readable string designed for
MQTT 5
diagnostics …" In MQTT 5, the communication mechanisms have been greatly
improved with the addition of negative acknowledgments.
Reason strings offer a comprehensive context that developers Multiple packet types and message flows can now respond
and operation teams need to swiftly pinpoint the cause of an with a negative acknowledgment message, enhancing the
event. In essence, a reason string is a human-readable string messaging infrastructure’s overall feedback loop.
meticulously designed for diagnostic purposes. This valuable
tool aids in understanding the nuances of events in a way that Contrasting with MQTT 3, where the UNSUBACK packet sent
reason codes alone can’t provide. to the client lacked a payload, leaving the client uninformed
in case of an unsuccessful UNSUBSCRIBE attempt, MQTT
However, it’s worth noting that reason strings, despite 5 fundamentally changes this. The UNSUBACK packet in
their usefulness in development and diagnostics, can be MQTT 5 includes a reason code informing the client about its
deactivated within HiveMQ’s configuration. This flexibility UNSUBSCRIBE attempt’s status, providing clear and actionable
caters to scenarios where the exposure of such detailed feedback. Several potential failure reasons are enumerated,
information might not be preferred. such as the nonexistence of the initial subscription or a lack of
client authorization to unsubscribe.
Server-Sent Disconnect Packets in MQTT 5
Acknowledgement packets from the publish flow, specifically
In MQTT 3.1 and 3.1.1, when a client surpasses a broker-
PUBACK, PUBREL, PUBREC, and PUBCOMP, have also been
defined limit, the broker responds by abruptly closing the
enhanced with the ability to send a negative acknowledgment
client’s connection. However, this action doesn’t provide
message. This ability is crucial when the server cannot process
any direct insight to the client regarding the reason for the
a message sent by the client, for example, due to the client
connection termination, which can lead to confusion and
not having the required authorization to publish on a certain
troubleshooting difficulties.
topic. The enhanced ack packet now equips the client with all
the necessary information to adapt and rectify the situation,
In contrast, MQTT 5 significantly improves this process by
reducing the need to contact operations or support teams to
introducing server-sent disconnect packets. These allow the
understand the issue. This marks a significant leap forward in
broker to deliver a DISCONNECT packet to the client before
maintaining efficient and smooth communication within MQTT
terminating the connection. Each server-sent disconnect
5 applications.
packet includes a reason code and a corresponding reason
string. These two elements work together to give the client

58
 Exceeding the maxPacketSize is one of the new reason codes for negative acknowledgments.

MQTT 5: Refining MQTT Communication

• MQTT brokers such as HiveMQ let you set a server side In essence, User Properties are the user-defined
max value configuration for the mentioned limits. This properties that help you add metadata to MQTT
is extremely useful in multi-vendor projects in which the messages and help transmit additional user-defined
operator of the broker may not have control over the information.
settings of the MQTT clients.
• The improved feedback for clients significantly simplifies
diagnosis for development and operations.
Let’s dive in.
• The added transparency also improves the interoperability
between different MQTT clients and brokers.
MQTT 5 has been meticulously crafted to bridge the existing
chasm between the offerings of MQTT 3.1.1 and the evolving
Notably, the advancements in feedback for clients
expectations of users from this de facto standard protocol for
substantially streamline the diagnostic process for both
the Internet of Things. With MQTT 5, we’re ensuring that MQTT
development and operations teams. This enhanced feedback
continues to lead the pack as the IoT protocol par excellence
mechanism resolves issues faster and prevents potential
for many more decades.
bottlenecks in your MQTT systems.

How User Properties in MQTT 5 Work?

Moreover, the transparency introduced with these
improvements fosters better interoperability between different In MQTT 5, User Properties fundamentally operate as
MQTT clients and brokers. This attribute is fundamental in straightforward UTF-8 string key-value pairs. Their utility lies
creating robust and versatile IoT ecosystems, and it further in their ability to be affixed to nearly every category of MQTT
emphasizes the strides MQTT 5 has made toward enhancing packet, with the sole exceptions of PINGREQ and PINGRESP.
communication, debugging, and overall system control. This broad application extends to various control packets like
PUBREL and PUBCOMP.

Chapter 17: MQTT User Properties

The power of User Properties is in their uncapped potential
In this chapter, we guide you through another groundbreaking - as long as the maximum message size isn’t exceeded, you
feature: User Properties. This addition is poised to are free to employ an infinite number of User Properties. This
revolutionize how you interact with MQTT, creating a more opens up vast possibilities for enriching MQTT messages
customized and insightful user experience.

59

[Link]

 MQTT Essentials Ebook

with additional metadata, facilitating a fluid transmission of initially seem minor, the practical implications of possessing
information between the publisher, broker, and subscriber. a mechanism to transfer metadata across the complete
MQTT ecosystem are indeed substantial. To illustrate this
Conceptually, this feature closely mirrors the transformative potential, let’s delve into three common
role of headers in HTTP. It’s this resemblance use cases that underscore the need for a feature like User
that allows User Properties to inject a level of Properties—a need articulated repeatedly by users eagerly
customizable complexity into MQTT 5, helping awaiting the introduction of this component in the MQTT
to create a protocol that is not only more robust specification.
but also more adaptable to user needs.
Saving Resources with Payload Metadata Using User
Properties in MQTT 5
Why Were User Properties Introduced in
MQTT 5? In environments where MQTT serves as a connector between
diverse systems developed by different teams or vendors,
MQTT 3 users identified two significant limitations: the
variability in payload structures is quite commonplace. Clients
protocol’s constrained extensibility and the complexity of
may transmit data in many formats, including JSON, XML, or
creating multi-vendor deployments. To rectify these concerns,
compressed formats such as Protobuf.
MQTT 5 introduced the User Properties feature, effectively
mitigating these challenges.
The advent of User Properties in MQTT 5 opens the door to
appending metadata to messages, encapsulating specific
User Properties provide an avenue for enhancing flexibility by
details such as the markup language and version employed
enabling users to transport virtually any piece of information
to encode the payload. This metadata provision obviates the
across the entire MQTT system. This capability ensures
need for the receiving client, or in certain instances, the broker,
that the MQTT protocol no longer restricts but promotes
to unpack the payload and cycle through an array of possible
customized enhancements. This leap forward in functionality
parsers until the appropriate one is located.
paves the way for users to augment standard protocol
features, tailoring them to meet their specific requirements.
Instead of this cumbersome process, each message
arrives equipped with its parsing information, streamlining
In doing so, MQTT 5 ensures that the protocol evolves in step
interpretation and significantly reducing the computational
with its users, facilitating greater adaptability and easing the
load across the entire system. This efficient use of resources
integration of multi-vendor deployments.
amplifies the overall performance and speed of the MQTT

Practical Use Case Examples of MQTT 5 User network, showcasing the transformative power of User
Properties Properties.

While the intricacies of the User Properties feature might

60 Including metadata about the used markup language in the payload can significantly relieve the system.
Increased Efficiency Through Application Level Routing Using User Properties in MQTT 5
With its proficiency in data transportation and routing, MQTT frequently serves as the backbone for large-scale data processing and
streaming deployments. Such deployments typically involve a multitude of devices, systems, and applications. It’s quite common for
different systems to receive identical messages but for distinct purposes. For instance, one system may display live data while other
archives the same data for long-term storage.

In such scenarios, User Properties can prove invaluable by serving as an additional application-level timestamp for the message.
This attribute lets the broker quickly ascertain whether certain messages should not be passed to a specific subset of subscribers
based on a predefined validity period. This feature introduces an added application-level layer that further refines message relevance
according to the Message Expiry Interval.

Therefore, User Properties in MQTT 5 not only bolster the system’s efficiency but also provide a finer level of control, thereby
maximizing the utility and relevancy of each transmitted message.

Example: user properties provide information for the broker, if messages should be routed to a storage or a display application.

Transparent Traceability in Complex Systems Using User Properties in MQTT 5

The landscape of IoT deployments often presents a maze of intricacy, with individual systems operating in parallel. Such complexity
can cloud the origin of a specific message or the factors contributing to an unsuccessful multi-layer message flow. Under the MQTT
3.1.1 framework, there was no mechanism for a subscriber to discern the identity of a message’s publisher. Although embedding
a unique identifier in the topic is a viable strategy in 1-to-1 scenarios, this approach undermines several pivotal advantages of the
publish-subscribe model.

In this regard, the advent of the User Property feature in MQTT 5 heralds a significant transformation. This innovative addition
enables publishers to effortlessly include relevant self-identifying information, such as a client ID or the region where the publishing
is conducted. Importantly, this information is relayed to all message recipients without necessitating any supplemental business
logic.

Incorporating information about the publisher’s region enhances the system’s traceability while attaching a unique system identifier
to MQTT messages enables comprehensive logging and tracking of the entire message flow from the sender to all subscribers.
Implemented effectively, these identifiers can extend across multiple MQTT message flows, introducing unprecedented transparency
and traceability.
61

[Link]

 MQTT Essentials Ebook

Adding information about the publisher’s region adds traceability to the system.

Such capabilities unlock a new horizon of possibilities, User Properties in MQTT 5 represent a significant step forward
particularly for business-critical applications like premium in protocol extensibility and versatility, opening up exciting
paid services for end customers, where transparency and possibilities for future IoT applications.
traceability become indispensable.
Chapter 18: MQTT Shared
Additional Information on User Properties in Subscriptions
MQTT 5
• User properties serve as UTF-8 string key-value pairs that In this chapter, we’ll delve into an especially interesting

can seamlessly be incorporated into any MQTT message. feature: Shared Subscriptions.

This ability positions them as a versatile and invaluable

addition to the MQTT protocol. Shared subscriptions, a core feature of MQTT

• The implementation potential of User Properties in 5, enable multiple MQTT clients to share a

enhancing MQTT use cases is practically boundless. It single subscription on the broker. In essence,

offers a degree of customization that allows for many this feature allows messages on a topic to be

innovative applications, both in function and scope. distributed among multiple clients, thereby

• Deployments and projects that extend over multiple improving load balancing and fault tolerance in

systems and vendors can leverage this feature an MQTT system.

to maintain consistency and ensure seamless

communication across the entire infrastructure.
• We are exhilarated to see the myriad of inventive ways
Shared subscriptions: Bridging the MQTT Gap
with V5
MQTT users will harness the potential of this deceptively
simple yet impactful feature. MQTT 5 was ingeniously designed to connect the gap
between the functionality offered by MQTT 3.1.1 and users’

62
expectations for the Internet of Things’ de facto standard that shared subscriptions employ a unique topic syntax for
protocol. With the integration of sought-after features such subscribing.
as Shared Subscriptions and Session and Message Expiry
Intervals, MQTT 5 is poised to consolidate MQTT’s standing as Shared subscriptions use the following topic structure:
the go-to IoT protocol for the foreseeable future.
$share/GROUPID/TOPIC

Due to the high demand for shared subscriptions, HiveMQ

The shared subscription consists of 3 parts:
introduced this feature before the MQTT 5 specification was
released. As a result, all MQTT brokers striving for 100% • A static shared subscription identifier ($share)

compatibility with the MQTT 5 specification must support this • A group identifier

feature. • The actual topic subscriptions (may include wildcards)

How MQTT Shared Subscriptions Work A concrete example for such a subscriber would be:

In a standard MQTT subscription, each subscribing client is $share/my-shared-subscriber-group/myhome/groundfloor/+/

privy to a copy of each message broadcasted to that topic. temperature
With shared subscriptions, clients sharing a subscription
in the same group receive messages in rotation, a process How Do MQTT Shared Subscriptions Work in
sometimes referred to as client load balancing. The message
HiveMQ MQTT Broker?
load of a single topic is distributed across all subscribers. A shared subscription group can be conceptually envisaged
as a virtual client acting as a proxy for numerous subscribers
MQTT clients can subscribe to a shared subscription using simultaneously. HiveMQ selects one subscriber from the group
standard MQTT mechanisms. Any standard MQTT clients, and delivers the message to that client. It typically uses a
such as Eclipse Paho, can participate without requiring round-robin approach for distribution. The following picture
any modifications on the client side. It’s important to note demonstrates the principle:

Shared Subscriptions

63

[Link]

 MQTT Essentials Ebook

For instance, a HiveMQ deployment might feature multiple How to Subscribe with Shared Subscriptions
shared subscription groups. These groups can have the in MQTT?
same subscription but different group identifiers. Whenever a
Engaging your clients with shared subscriptions is a
publisher sends a message with a matching topic, one unique
straightforward process. Below, we illustrate how to
client from each group receives the message. For example, the
accomplish this using the MQTT CLI. The given command line
following scenario is possible:
execution code allows two MQTT clients to subscribe to the
same subscription group and topic:

mqtt sub -h [Link] -t

'$share/group1/my-share-topic' -i
client1 -q 1

mqtt sub -h [Link] -t

'$share/group1/my-share-topic' -i
client2 -q 1

Shared Subscriptions multiple groups

In the given scenario, we have two distinct groups, each

With these commands executed, both MQTT clients now
encompassing two subscribing clients within their shared
have a shared subscription to the ‘my-share-topic’ (part of
subscription group. Although these groups subscribe to
the virtual group ‘group1’). In this configuration, each client
the same topics, they are differentiated by unique group
is allocated half of the MQTT messages dispatched over the
identifiers. When a publisher issues a message that aligns with
MQTT broker on the topic ‘my-share-topic’.
a particular topic, a solitary client from each group — and only
one client — is selected to receive the message.
Remember, the MQTT clients can join or depart from the
subscription group whenever they prefer. For instance, should
MQTT Shared Subscription Use Cases
a third client decide to join the group, the distribution of
Shared subscriptions have a multitude of applications, relevant MQTT messages becomes equally divided among all
particularly in high-scalability scenarios. These include: three clients, each receiving one-third of the total.

• Client load balancing for MQTT clients that cannot handle

the load on subscribed topics. For a deeper understanding of the semantics of shared

• Worker (backend) applications that ingest MQTT streams subscriptions, the official HiveMQ Documentation is an

must scale horizontally. excellent resource.

• Optimizing subscriber node-locality for incoming

publishes to reduce HiveMQ intra-cluster node traffic. Scaling MQTT Subscribers with Shared
• The delivery semantics employ QoS 1 and 2 though
Subscriptions
there’s no necessity for guarantees on ordered-topic. Shared subscriptions provide a simple way to integrate
• Addressing hot topics causing scalability bottlenecks due backend systems with MQTT, especially when it is not feasible
to higher message rates. to use HiveMQ’s extension system or dynamic scaling is
necessary. With the shared subscriptions, you can quickly add

64
 subscribers as needed, distributing work in a push fashion. indicates an “unspecified byte stream” while a 1 represents a
Shared subscriptions prove immensely valuable for scaling and “UTF-8 encoded payload.” When the Payload Format Indicator
load-balancing MQTT clients. Furthermore, HiveMQ clusters isn’t provided, it automatically defaults to 0.
offer additional benefits in terms of latency and scalability
through internal optimization of message routing.

For comprehensive details on shared subscriptions

or 1
and HiveMQ clusters, reference the official HiveMQ
Documentation.

In summary, shared subscriptions provide a compelling way

to distribute messages across various MQTT subscribers The Payload Format Description is optional. It can have the
using standard MQTT mechanisms. This feature simplifies values "1" or "0".
implementing MQTT-client load balancing without the need
for any proprietary modifications to your MQTT clients. This is MQTT Content Type
especially beneficial for backend systems or “hot-topics” that
Similar to the Payload Format Indicator, the Content Type
might quickly overwhelm a single MQTT client.
is also optional and can be incorporated in a CONNECT
containing a WILL message or any PUBLISH packet. The value
Chapter 19: MQTT Payload Format for the Content Type must be a UTF-8 encoded string that
Description and Content Type identifies the payload’s nature. When the Payload Format
Indicator is set to 1, ideally, you should have a MIME content
In this chapter, we will focus on Payload Format Indicators,
type descriptor (though it’s not a hard requirement). A valid
which specify the message content type, ensuring easier, more
UTF-8 String is all you need.
efficient parsing and interoperability between systems.

For UTF-8
What is Payload Format Indicator in MQTT?
encoded
The Payload Format Indicator is a fundamental component strings
of any MQTT packet that houses a payload. This includes a Content
CONNECT packet encapsulating a WILL message or a PUBLISH Types can be
packet. This optional byte value has two possible settings: a 0 defined.

Why Describe the Payload Format?

The combined use of Payload Format Indicator
and Content Type facilitates a transparent
description of the payload content for any
application message. This ability sets the stage
for creating and defining industry-wide MQTT
standards for varied payload formats. MQTT
protocol experts view this standardization as the
protocol’s natural progression.

Payload Format Description allows pre-parsing without the need to open the payload 65

[Link]

 MQTT Essentials Ebook

Having the payload content description in the headers can What is Request-Response Pattern in MQTT?
prove incredibly beneficial in individual deployments. It
MQTT is rooted in asynchronous messaging, adopting the
ensures every message is correctly processed without delving
publish-subscribe paradigm. This design allows senders and
into the payload itself. Depending on the content type, different
receivers to function independently of one another, facilitating
messages within a system may need various parsing methods.
one-to-many relationships. It’s vital to grasp that MQTT’s
Moreover, in certain instances, message persistence could
request-response pattern tackles challenges differently from
hinge on the payload’s specific type. As the content-type
synchronous, one-to-one-based protocols like HTTP.
definitions hinge on user design, the potential applications of
this feature appear boundless.
In MQTT, a response typically doesn’t directly
answer a request’s “question”. Instead, the
In summary, the Payload Format Indicator discerns whether
request triggers a specific action in the receiver,
a payload is an undefined byte array or a UTF-8 encoded
and the response communicates the result of
message. When dealing with UTF-8 encoded messages, the
this action.
sender can use the content type to specify the payload’s
nature.
Sounds complicated? Don’t fret; a concrete example will soon
bring this into perspective!
These features set the stage for transparent payload content
definitions across large-scale systems and, potentially,
What is Response Topic in MQTT 5?
entire industries. As the need for pre-parsing actual payloads
diminishes, proper message processing can considerably A response topic is an optional UTF-8 string incorporated
enhance scalability. into any PUBLISH or CONNECT packet. If a value is present
in the response topic, the sender immediately classifies the
Although it’s anticipated that most users will rely on known associated PUBLISH as a request. The response topic field
MIME types to describe the content, they can also use arbitrary indicates the topics where the message receivers’ responses
UTF-8 Strings. are expected. The initial PUBLISH (request) and the response
topic can have multiple or no subscribers. Ideally, the original

Chapter 20: MQTT Request-Response PUBLISH (request) sender should subscribe to the response
topic before dispatching the request.
Pattern

In this chapter, we spotlight two standout elements: Response What is Correlation Data in MQTT 5?
Topic and Correlation Data. Correlation data is optional binary data that trails the
response topic. It helps the request’s sender identify which
Navigating the complexities of modern IoT projects demands specific request a later received response relates to. The
teamwork across diverse vendors and teams. As MQTT correlation data allows the original request sender to manage
has become the protocol par excellence for IoT, enhanced asynchronous responses potentially sent from multiple
interoperability and system transparency emerged as prime receivers. It’s important to note that this data is not relevant
requirements in the MQTT version 5 blueprint. The features to the MQTT broker but serves to identify the relationship
we’re delving into today answer these user needs by providing between sender and receiver.
a standard solution for implementing a request-response
pattern with MQTT. What is Response Information in MQTT 5?
To foster transparent implementation and improved
standardization, the MQTT 5 specification introduced the

66
Response Information property. Through a boolean field in the CONNECT, a client can request response information from the
broker. When set to true, the broker can dispatch an optional UTF-8 String field (response information) in the CONNACK packet to
communicate anticipated response topics.

This feature allows users to globally define a specific part of your topic tree on the broker, accessible by all clients indicating their
intention to use the request-response pattern at the connection establishment.

End-to-End Acknowledgment in MQTT 5

MQTT ensures the complete decoupling of message senders and receivers. Use cases often call for an acknowledgment of message
receipt from the intended recipient. For example, when opening the door of your smart home via a command, the sender (typically a
mobile app) would want to know when and if the message was received and the outcome of the command.

Example: Smart door opening with a mobile device using the MQTT 5 Request Response feature.

The inclusion of the request-response pattern in the MQTT directly into protocol fields, we’ve significantly enhanced
5 specification was largely driven by the need for these application development’s versatility, dynamism, and
“business ACKs.” MQTT users sought the capability to transparency under the request-response pattern.
provide end-to-end acknowledgments between an application
message’s sender and receiver. Adding response topics, Source Code Example of MQTT Request-
correlation data, and response information as protocol Response Workflow
fields significantly bolsters extensibility, dynamism, and Below is a quick code snippet showcasing the HiveMQ MQTT
transparency in application development using the request- Client’s capabilities, providing a taste of the request-response
response pattern. pattern workflow in MQTT. Note that it’s not a complete,
functioning excerpt.
MQTT 3 users previously embraced this pattern. By integrating
response topics, correlation data, and response information You can access a full example on GitHub.

67

[Link]

 MQTT Essentials Ebook

//Requester subscribes to response topic

Mqtt5SubAck subAck = [Link]()
.topicFilter("job/client1234/result")
.send();

//Requester publishes request

Mqtt5PublishResult result = [Link]()
.topic("job")
.correlationData("1234".getBytes())
.responseTopic("job/client1234/result")
.payload([Link]())
.send();

//Responder subscribes to request topic

Mqtt5SubAck subAck = [Link]()
.topicFilter("job")
.send();

//Responder sends response after receiving the request

Mqtt5PublishResult result = [Link]()
.topic([Link]().get())
.payload([Link]())
.correlationData([Link]().get())
.send();

Key Takeaways from Request-Response in MQTT

Here are a few key takeaways of request-response in MQTT:

• The request-response pattern in MQTT significantly differs from its counterpart in

synchronous, client-server-based protocols like HTTP.
• MQTT allows multiple, single, or even no subscribers for requests and responses.
• Correlation data ensures the proper linking between request and response, enhancing
message tracking.
• This pattern facilitates the implementation of “business acknowledgment”
functionality, providing an extensible, dynamic, and transparent solution.

68
Best Practices to Consider While Using Why Use Topic Aliases in MQTT?
Request-Response in MQTT
MQTT plays a pivotal role in your network with its efficiency
Here are a few best practices to consider while using request- in maintaining standing connections between your devices
response in MQTT: and the brokers. The Keep Alive mechanism guarantees the
longevity of connections between clients and the broker,
• Ensure the requester subscribes to the relevant response
swiftly detecting any connection loss that could occur in
topic before sending a request.
unstable networks. PING packets - of just two bytes - need
• Employ unique identifiers within the response topic to
to be sent only every few minutes, enabling MQTT to sustain
avoid confusion.
these connections with minimal power and bandwidth use.
• Ascertain that responders and requesters possess the
necessary permissions to publish and subscribe to
This brings us to the Topic Alias feature, which is particularly
response topics.
beneficial in deployments involving a vast array of connected
• Dedicate a specific section of the topic tree for response
devices transmitting smaller, frequent messages. So, let’s
purposes, and utilize the response information field to
dive into this feature and see how it can optimize your MQTT 5
relay it to clients.
utilization.

As we journey through the transformative features of MQTT

v5, we discover the power of protocol evolution. These
How to Use MQTT Topic Names?
enhancements, like the request-response pattern, not only The MQTT client and the broker have the power to establish a
streamline existing practices but also unlock new realms of Topic Alias for any PUBLISH message, provided they are the
dynamic, transparent, and extensible application development. message’s originator. Similarly, they can control the number
of Topic Aliases permitted for each connection. The upper
Chapter 21: MQTT Topic Alias limit for Topic Aliases, known as the Topic Alias Maximum, is
determined during the connection establishment phase.
In the previous chapter, we unraveled the Request - Response
Pattern. In this chapter, we now direct our attention to another The client indicates its Topic Alias Maximum in the CONNECT
potentially impactful feature: Topic Alias. packet, while the broker does so in the CONNACK packet.
Therefore, the client should only utilize Topic Alias values
What is MQTT Topic Alias? ranging from 1 to the broker-defined Topic Alias Maximum
presented in the CONNACK packet. Similarly, the broker should
Topic Aliases are integer values substituting topic respect the range of 1 to the client-defined maximum from the
names. It enables you to condense a lengthy and CONNECT packet.
frequently utilized topic name into a 2-byte integer.
This helps minimize the amount of bandwidth In the absence of a specified Topic Alias Maximum, a default
consumed during message publication. value of 0 is assumed, which effectively disables the use of
Senders define the Topic Alias value in the PUBLISH Topic Aliases. This ensures clear communication boundaries
message, followed by the topic name. Receivers and precise control of Topic Aliases within your MQTT
then process this message like any other PUBLISH, deployment.
establishing a mapping between the Topic Alias
(integer) and Topic Name (string). Subsequent Use Case Examples of MQTT Topic Alias
PUBLISH messages for the same topic can be sent
MQTT stands out as a lightweight communication protocol
with just the Topic Alias, omitting the topic name.
that efficiently maintains TCP connections between clients

69

[Link]

 MQTT Essentials Ebook

and brokers. Its Keep Alive mechanism minimizes energy and

bandwidth usage, enabling users to establish cost-effective,
always-connected device deployments. Moreover, it allows for Understanding MQTT Topic Alias
real-time delivery of minimal data points, like measurements, • Topic Aliases substitute UTF-8 String topic
negating the need for periodic bulk data transfers – a names with an integer.
requirement of heavier data transfer technologies. • Topic Alias to Topic mapping is relevant only
for a single connection.
This inherent efficiency of MQTT finds utility in numerous • This feature’s support is optional for brokers
applications, such as predictive maintenance, where the and clients.
service’s quality and responsiveness can be enhanced by • Broker and client negotiate to what degree this
transmitting small data points in real time. In these situations, feature is supported during the connection
an elaborate topic name might be more sizable than the establishment.
actual data payload, which a single integer could represent. • Ensure your broker and client implementation
For instance, a topic name like 'data/europe/germany/ supports Topic Aliases if you wish to use this
south/bavaria/munich/schwabing/box-32543y/ feature.
junction/consumption/current' describes the current • When used correctly, Topic Aliases can
power consumption of a specific junction box, with the payload significantly impact the profit margins of your
being a single integer value. business case.

In summary, Topic Alias offers a versatile approach to utilizing

the pub/sub model. When repeatedly publishing messages to
a limited number of topics, particularly in high volumes, Topic
Aliases can significantly conserve network and computing
resources in an efficient manner.

Chapter 22: Enhanced Authentication

in MQTT
Topic Alias can substitute long and complex topic strings with
Modern IoT projects have evolved into large, complex projects,
single integers.
especially when robust security measures are paramount.
These expansive initiatives often involve collaboration between
The Topic Alias feature of MQTT comes to the fore in these
multiple vendors and teams. Adhering to internationally
instances, replacing lengthy and complicated topic strings
accepted standards becomes crucial to streamline the
with single integers. This technique is especially useful
challenges encountered in such projects. Enhanced
when sending numerous small messages over extensive
Authentication helps ensure compliance with these standards.
topic names in real time, offering two primary advantages: It
amplifies performance while significantly reducing network Implementing Challenge-Response
traffic. Hence, Topic Alias becomes a powerful tool in your Authentication
MQTT 5 arsenal, optimizing data transmission and network
By incorporating challenge-response authentication into your
management.
MQTT 5 implementation, you can access industry-standard
authentication mechanisms like the Salted Challenge

70
Response Authentication Mechanism (SCRAM) or the Kerberos protocol. These widely recognized protocols further bolster the
security of your IoT infrastructure by adding a layer of verification.

What is Authentication Flow in MQTT?

The authentication flow in enhanced authentication relies on three MQTT message types: CONNECT, CONNACK (already present in
MQTT v3), and the new MQTT v5 AUTH message. Clients send CONNECT messages, while the server sends CONNACK messages.
Both message types are used once during each authentication process. On the other hand, AUTH messages can be used multiple
times by both the server and the client.

The core of the authentication flow revolves around two message properties: the Authentication Method (identified by byte 21) and
the Authentication Data (identified by byte 22). These properties are set on every message involved in the enhanced authentication
flow.

Authentication Flow

Authentication Method in MQTT the authentication process. It typically involves transferring

encrypted secrets or protocol steps in multiple iterations.
With the Authentication Method the client and server can
The specific content of the data heavily relies on the chosen
select and describe the agreed-upon authentication approach.
mechanism employed in enhanced authentication and is
It is represented by method strings commonly used to identify
specific to the application in use.
SASL (Simple Authentication and Security Layer) mechanisms.
For instance, examples of method strings include SCRAM-
Source Code Example of Enhanced
SHA-1 for SCRAM with SHA-1 or GS2-KRB5 for Kerberos.
Authentication in MQTT

The Authentication Method assigns significance to the In this code snippet, we utilize the HiveMQ extension SDK to

exchanged data during enhanced authentication and should implement enhanced authentication. The purpose is to verify

remain constant throughout the process, ensuring consistency the support of the Authentication Method and determine the

and integrity. state of an MQTT client that is connecting after the exchange
of two AUTH messages.
Authentication Data in MQTT
Authentication Data refers to binary information utilized during
71

[Link]

 MQTT Essentials Ebook

public class MyEnhancedAuthenticator implements EnhancedAuthenticator {

public void onConnect(EnhancedAuthConnectInput input, EnhancedAuthOutput
output) {
final ConnectPacket connectPacket =
[Link]();
// Is the given authentication method supported?
if
(authenticationMethodIsSupported([Link]())) {
// Did the client provide valid authentication data?
if
(validateClientAuthenticationData([Link]())) {
// Send an AUTH message that contains a challenge!
output.
continueAuthentication(prepareServerAuthenticationData());
return;
}
}
// Fail the authentication and disconnect the client.
[Link]();
}
public void onAuth(EnhancedAuthInput input, EnhancedAuthOutput output) {
final AuthPacket authPacket = [Link]();
// Try to validate the response.
if
(validateClientAuthenticationData([Link]())) {
// Allow the client to connect to the server.
[Link]();
return;
}
// Fail the authentication and disconnect the client.
[Link]();
}
}

The significance of Enhanced Authentication cannot be overstated. In a world where the proliferation of interconnected devices has
amplified the importance of secure communication, MQTT 5 steps up to the challenge. This advanced authentication mechanism
empowers organizations to safeguard their IoT infrastructure, sensitive data, and the privacy of their users.

72
Chapter 23: MQTT Flow Control Similarly, an IoT device might connect to multiple MQTT
brokers, each with distinct limitations on managing in-flight
Flow Control is a dynamic feature introduced in MQTT 5 messages from an MQTT client. To seamlessly manage such
designed to regulate message traffic between IoT devices and diversified conditions among MQTT clients and brokers, MQTT
brokers for efficient and stable communication. 5 introduces the Flow Control feature.

IoT deployments encapsulate a wide range of device types. How Flow Control Works in MQTT 5?
For instance, an MQTT client embedded in a compact
The Flow Control feature functions through a negotiation
sensor varies significantly from one incorporated in a high-
between the client and broker to establish in-flight windows
performance backend server regarding processing speed
during the connection. This process involves setting an
and storage capabilities. Consequently, these MQTT clients
optional property known as "Receive Maximum" in the
demonstrate varying tolerance levels for managing in-flight
CONNECT packet, indicative of the maximum number
messages. Here, an in-flight message refers to a PUBLISH
of unacknowledged PUBLISH messages the client can
command with a Quality of Service level of one or two
accommodate. The broker reciprocates with a similar value for
awaiting acknowledgment.
“Receive Maximum” in the CONNACK packet. If the "Receive
Maximum" value is not specified, the default value of 65,535 is
employed.

Client and broker negotiate their receive maximum.

What Are the Advantages of Flow Control in sends more unacknowledged messages than what the Server
MQTT 5? Receive Maximum permits, the broker sends DISCONNECT
with Reason Code 0x93 (Receive Maximum exceeded). This
Flow Control enhances dynamic message flow adjustment for
flexibility allows the client and broker to send fewer in-flight
use cases involving diverse systems and devices, fostering
messages than the corresponding Receive Maximum permits.
transparency and adaptability when multiple teams or vendors
collaborate on a project. No longer is it necessary for all
parties to pre-establish in-flight windows. If an MQTT 5 client

73

[Link]

 MQTT Essentials Ebook

This chapter explains why finding the matching subscriptions

What to Do and What Not to Do among millions of subscribers is a challenge and how an
MQTT broker can overcome this challenge.
• Implementing “Receive Maximum”
remains an optional, yet beneficial
What is Topic Matching in MQTT?
choice.
• Both client and broker can establish MQTT is a publish/subscribe protocol where devices act as

their unique in-flight windows during the MQTT Clients and exchange messages over an MQTT Broker.

connection initiation. MQTT Clients send their data in the PUBLISH control packets

• Flow Control is designed to maintain to the specific topic. The topic is separate from the packet’s

balanced message processing, payload, which allows the broker to avoid analyzing the

preventing the overloading of any packet’s payload. The broker delivers the published message

participating parties. to every client subscribed with a matching topic filter.

• As a feature, Flow Control aligns

seamlessly with MQTT 5’s key For those unaware, the main distinction between the topic and

objectives - enhancing transparency and topic filter is that the topic is used for publishing and cannot

fostering increased flexibility. contain wildcard characters whereas the topic filter can. The
wildcard characters are used to aggregate multiple streams
of data into one and are thus used on the subscriber’s side. It

Chapter 24: MQTT Topic Tree & is possible to create a topic filter without wildcard characters,
then it would only match at most one topic. That case is often
Topic Matching: Challenges and Best
referred to as an exact subscription.
Practices Explained

As both the size and complexity of IoT projects continues to In a nutshell, topic filter can be thought of as a selector for

grow, we talk to many IT architects who are working to solve topics that the PUBLISH packets are sent to. The broker must

the technical challenges to design a data foundation built for be able to find the matching subscriptions for each published

scale. In a large MQTT deployment, there may be thousands or message.

even millions of clients subscribing to different topics.

74
Subscriptions can contain wildcard characters to match a • +/house/kitchen
broad range of topics. Subscriptions with wildcards are often • town/+/kitchen
used when there is uncertainty about the topics that publishing • town/house/+
clients will use. For example, when the publishing clients • +/+/kitchen
include their ID as one topic level, it may be impossible to • town/+/+
reliably receive messages for all such topics without the usage • +/+/+
of wildcard characters. While this is useful for the clients, • town/house/kitchen
finding all the matching subscriptions presents a technical • …
challenge. In some real-life scenarios, brokers check millions
of subscriptions for every published message. The broker must also check the map for all these topic filters.
In production workloads, the broker has to find the matching
MQTT Wildcard Topic Matching Challenge subscriptions for published messages thousands of times
Explained per second, so it needs a specialized data structure for a fast

Since there are many use cases for wildcards, let’s examine the lookup.

technical challenge of capitalizing on wildcard subscriptions.

First, looking at every subscription for every published The Topic Tree
message is not scalable. The number of steps needed to find
The Topic Tree is a data structure used to solve the challenges
the matching subscriptions linearly increases with the number
posed by the above wildcard topic matching problem. The
of subscriptions. Alternatively, the broker could map the
topic of the published message is used to collect the matching
subscriptions to their topic filters and check the map for all
subscriptions present in the topic tree. We start at the root of
filters matching the topic of a published message. This method
the topic tree and proceed through its levels using the topic
is also impractical because the number of potentially matching
segments to select the next node. If the current node has
topic filters is rather large for topics with many levels. For
wildcard subscriptions (with # or +), they are added to the
instance, if a message is published to the topic “town/house/
matching subscriptions. Once there are no more segments to
kitchen”, all the subscriptions with the following topic filters
match in the topic and there are non-wildcard subscriptions in
would match:
the current node, there are exact subscriptions for this topic.
• # An exact subscription filters topics of published messages for
• town/# exact matches.
• town/house/#

MQTT Topic Tree Structure

75

[Link]

 MQTT Essentials Ebook

The broker can continue delivering the published message In summary, MQTT’s topic matching is crucial to its publish/
to the subscribed clients once it has found all matching subscribe protocol, enabling MQTT clients to exchange
subscriptions. This way of storing the subscriptions also messages with the MQTT broker with minimal effort. Topic
reduces memory usage because topic levels shared across filters help select the topics to which PUBLISH packets are
multiple subscriptions are only stored once. sent, and subscriptions with wildcard characters enable broad
topic matching. However, finding all matching subscriptions
presents a technical challenge, it can be solved using a
specialized data structure called the Topic Tree. It is essential
Best Practices to use best practices when designing topics to make them
For your application to filter messages to your agnostic of the implementation that the particular MQTT
specifications regardless of how the MQTT Broker broker may have for topic matching.
defines topic matching, there are a few topic design
considerations that you may leverage. Chapter 25: Additional Reading for
Mastering MQTT
It is good practice to avoid topic levels that do not
add additional information, like using the same topic MQTT Over WebSockets
level across all subscriptions. The most common
We’ve seen that MQTT is ideal for constrained devices and
example of such abuse is using the company name
unreliable networks and that it is perfect for sending messages
as the first level for every subscription. While some
with a very low overhead. Naturally, it would be quite nice
topic levels typically have less variety than others,
to send and receive MQTT messages directly in a browser.
you should omit topic levels that are the same for
For example, on a mobile phone. MQTT over WebSockets is
every topic. Similarly, leading with forward slashes
the answer. MQTT over WebSockets enables the browser to
must be matched, so should be avoided if you don’t
leverage all MQTT features. You can use these capabilities for
want them present on all topics.
many interesting use cases:

Bad practice: • Display live information from a device or sensor.

• Receive push notifications (for example, an alert or critical
• home/livingarea/kitchen
condition warning).
• home/livingarea/bathroom
• See the current status of devices with LWT and retained
• home/garage
messages.
• Communicate efficiently with mobile web applications.
Bad practice:

• /livingarea/kitchen What Does All This Mean from a Technical Point of View?
• /livingarea/bathroom
Every modern browser that supports WebSockets can be a
• /garage
full-fledged MQTT client and offer all the features described in
the MQTT Essentials. The Keep Alive, Last Will and Testament,
Good practice:
Quality of Service, and Retained Messages features work
• livingarea/kitchen the same way in the browser as in a native MQTT client.
• livingarea/bathroom All you need is a JavaScript library that enables MQTT
• Garage over WebSockets and a broker that supports MQTT over
WebSockets. Of course, HiveMQ Broker offers this capability
straight out-of-the-box.

76
How Does It Work? • If you want to integrate MQTT into your existing web
application, check out this step-by-step guide on how to
WebSocket is a network protocol that provides bi-directional
build your own MQTT WebSockets client.
communication between a browser and a web server. The
• To learn more about how to set up your own broker with
protocol was standardized in 2011 and all modern browsers
WebSockets support, read MQTT over WebSockets.
provide built-in support for it. Similar to MQTT, the WebSocket
protocol is based on TCP.
Secure WebSockets
You can leverage Transport Layer Security (TLS) to use
secure WebSockets with encryption of the whole connection.
This method works seamlessly with HiveMQ. However, there
are a few points that you need to keep in mind. For more
information, see the Gotchas section of our user guide.

Types of MQTT Brokers

In MQTT over WebSockets, the MQTT message (for example, There are different types of MQTT brokers available:
a CONNECT or PUBLISH packet) is transferred over the
• Open source: These are license-free brokers that are
network and encapsulated by one or more WebSocket
suitable for small-scale projects.
frames. WebSockets are a good transport method for
• Commercial: These are available via licenses and offer
MQTT because they provide bi-directional, ordered, and
several features and customization, making them suitable
lossless communication (WebSockets also leverage TCP).
for production-level deployment.
To communicate with an MQTT broker over WebSockets,
• Cloud-managed: These are fully-managed MQTT brokers
the broker must be able to handle native WebSockets.
that need minimum maintenance and are easy to deploy.
Occasionally, people use a webserver and bridge WebSockets
• General purpose: These are available both license-free
to the MQTT broker, but we don’t recommend this method.
as well as for licenses and are suitable for small-scale
When using HiveMQ, it is very easy to get started with
projects.
WebSockets. Simply enable the native support in the
configuration.
The below image summarizes features offered by different
types of brokers.
Why Not Use MQTT Directly?

Currently, it is not possible to speak pure MQTT in a browser

because it is not possible to open a raw TCP connection.
Socket API will change that situation; however, few browsers
implement this API yet.

Get Started
If you want to get started with MQTT over WebSockets, here
are some useful resources:

• For testing and debugging, the HiveMQ MQTT WebSocket

client is ideal. The public broker of the MQTT Dashboard
is the default broker of this client. All features of the
client are documented in detail and the source code is
available on GitHub. For in-depth information, download our 2023 Buyer’s Guide:
MQTT Platforms. 77

[Link]

 MQTT Essentials Ebook

Open Source vs. Commercial MQTT Brokers On-premises MQTT Brokers vs. Fully-Managed Cloud
MQTT Brokers
Open-source MQTT brokers have limited scalability, limited
security options, cannot cluster for higher availability, are On-premises MQTT brokers offer more control but require
hard to manage when coded in difficult libraries, and have no substantial setup and maintenance. Fully managed cloud-
overload protection from overactive publishers. based MQTT brokers are easier to deploy, cost-effective, and
have zero maintenance requirements.
On the other hand, commercial MQTT brokers are customizable
according to your needs, provide increased scalability, security, HiveMQ offers an on-prem MQTT broker as well as a fully-
flexibility & reliability, and have overload protection. managed cloud MQTT broker. Explore them now!

HiveMQ offers a community edition MQTT broker as well as a

commercial MQTT broker. Explore them now!

MQTT vs. Other IoT Protocols

MQTT vs. HTTP

MQTT HTTP

MQTT (the OASIS standardization

Full name group decided it would not stand for Hypertext Transfer Protocol
anything)

Publish subscribe (MQTT does have

Architecture Request response
a request/reply mode as well)

Command targets Topics URIs

Underlying Protocol TCP/IP TCP/IP

TLS + username/password (SASL

Secure connections TLS + username/password (SASL support possible)
support possible)

Known connection status (will

Client observability Unknown connection status
messages)

Messaging Mode Asynchronous, event-based Synchronous

78
 MQTT HTTP

The broker can queue messages for

Message queuing Application needs to implement
disconnected subscribers

2 bytes minimum. Header data can 8 bytes minimum (header data is text - compression
Message overhead
be binary possible)

No limit but 256MB is beyond normal use cases

Message Size 256MB maximum
anyway.

Content type Any (binary) Text (Base64 encoding for binary)

Message distribution One to many One to one

Three qualities of service: 0 - fire and

Reliability forget, 1 - at least once, 2 - once and Has to be implemented in the application
only once

To learn more, read our blog, MQTT vs. HTTP

MQTT vs. AMQP

MQTT AMQP

MQTT (the OASIS standardization

Full name group decided it would not stand for Advanced Message Queuing Protocol
anything)

Publish subscribe (MQTT does have Queues, multicast (fanout), publish subscribe,
Architecture
a request/reply mode as well) request reply

Command targets Topics Exchanges, Queues

Underlying Protocol TCP/IP TCP/IP

79

[Link]

 MQTT Essentials Ebook

MQTT AMQP

TLS + username/password (SASL

Secure connections TLS + username/password (SASL support possible)
support possible)

Known connection status (will

Client observability Unknown connection status
messages)

Messaging Mode Asynchronous, event-based Synchronous and asynchronous

The broker can queue messages for

Message queuing Core capability, flexible configuration
disconnected subscribers

Message overhead 2 bytes minimum 8 bytes (general frame format)

Message Size 256MB maximum 2GB theoretical, 128MB max recommended

Content type Any (binary) Any (binary)

Topic matching Level separator: / Wildcards: + # Level separator: . Wildcards: * #

Three qualities of service: 0 - fire and

Two qualities of service: -without acks (=0) - with
Reliability forget 1 - at least once 2 - once and
acks (=1)
only once

Connection
No Yes - channels
“multiplexing”

Message attributes MQTT 5.0 only Yes

Object persistence Yes Yes

To learn more, read our blog, MQTT vs. AMQP

80
 MQTT vs. ZeroMQ

MQTT ZeroMQ

MQTT (the OASIS standardization

Full name group decided it would not stand for ZeroMQ
anything)

Architecture Client/server Peer to peer

Command targets Topics ZeroMQ sockets

Underlying Protocol TCP/IP TCP/IP, UDP, shared memory

TLS + username/password (SASL

Secure connections PLAIN (username/password), CurveZMQ and ZAP
support possible)

Known connection status (will

Client observability None
messages)

Retained messages Yes No

Messaging Mode Asynchronous, event-based Application dependent

The broker can queue messages for

Message queuing Some 0MQ socket types have it
disconnected subscribers

Message overhead 2 bytes minimum 2 bytes minimum

81

[Link]

 MQTT Essentials Ebook

MQTT ZeroMQ

Message Size 256MB maximum 2^63-1 bytes per frame

Message
No Yes
fragmentation

Content type Any (binary) Any

Pub/sub topic
Level separator: / Wildcards: + # Prefix only
matching

Message distribution One to many, one to one Various

Three qualities of service:

Varies between socket types, but the equivalent
0 - fire and forget
Reliability of QoS 2 would have to be implemented in the
1 - at least once
application.
2 - once and only once

To learn more, read our blog, MQTT vs. ZeroMQ

MQTT vs. CoAP

MQTT CoAP

MQTT (the OASIS standardization

Full name group decided it would not stand for The Constrained Application Protocol
anything)

Architecture Client/Server Client/Server

82
 MQTT CoAP

Command targets Topics URIs

Underlying transport TCP UDP

Default insecure TCP

1883 5684
port

Default secure UDP

8883 5684
port

TLS + username/password (SASL

Secure connections DTLS (TLS for UDP)
support possible)

Known connection status (will

Client observability None
messages)

Retained messages Yes No

Synchronous (or asynchronous with observe

Messaging Mode Asynchronous, event-based
extension)

The broker can queue messages for

Message queuing None
disconnected subscribers

Message overhead 2 bytes minimum 4 bytes minimum

Limit of underlying transport - RFC 7252 suggests

Message Size 256MB maximum 1152 bytes for UDP if nothing is known about the
target.

Message
Reliant on TCP No (RFC 7959 proposes an extension for this)
fragmentation

Content type Any (binary) Any

To learn more, read our blog, MQTT vs. CoAP 83

[Link]

 MQTT Essentials Ebook

MQTT CoAP

Pub/sub topic
Level separator: /Wildcards: + # No pub/sub (but extensions have been proposed)
matching

Message distribution One to many, one to one One to many, one to one

Three qualities of service: 0 - fire and

Confirmable or not (roughly equivalent to QoS 1 and
Reliability forget 1 - at least once 2 - once and
0)
only once

Implementing MQTT in C
To get started with the Paho MQTT C Client library, clone the repo and use the following code to install it.

Developers can use different client libraries to implement MQTT in different languages, such as Java, C, C#, etc.

git clone [Link]

make
sudo make install

For more instructions, read our blog Implementing MQTT in C.

Implementing MQTT in Java

Developers can use an MQTT Java Client Library, such as HiveMQ MQTT Client for Java or Eclipse Paho Java MQTT Client Library.
To use Paho Java, install it by downloading it from the Eclipse project page. If you’re using Maven, add the following code to your
[Link] file.

<repositories>
<repository>
<id>Eclipse Paho Repo</id>
<url>[Link]
</repository>
</repositories>

84
Then add the following code to your dependencies section.

<dependency>
<groupId>[Link]</groupId>
<artifactId>[Link].mqttv3</artifactId>
<version>1.0.2</version>
</dependency>

For more instructions, read our blog Implementing MQTT in Java.

Implementing MQTT in C#
You can install the HiveMQ C# MQTT client by using the code:

dotnet add package HiveMQtt

For more instructions, read our blog Implementing MQTT in C#.

Implementing MQTT in Python

You can install Paho MQTT for Python 3.12 in the command line with the code:

pip3 install paho-mqtt<2.0.0

For more instructions, read our blog Implementing MQTT in Python.

MQTT Broker Architecture in Smart Manufacturing

MQTT brokers can be used for several smart manufacturing use cases, such as intra-factory connections, inter-factory connections,
etc.

Here’s an example of inter-factory communication using the HiveMQ MQTT Broker.

To learn more, read our

whitepaper Modernizing
the Smart Manufacturing
Industry with MQTT.

85

[Link]

 MQTT Essentials Ebook

MQTT Broker Architecture in Connected Cars

MQTT brokers can be used for several connected car use cases such as remote monitoring, remote diagnostics, Over-the-Air (OTA)
updates, theft recovery, energy management, etc.

To learn more, read our blog Building New Connected Car Features on the Back of a Robust Connectivity Platform.

MQTT Broker Architecture in Transportation and Logistics

MQTT brokers are well-suited for use in the transportation and logistics industry, where real-time data communication is crucial for
efficiency and safety. Some of the use cases include fleet management, supply chain visibility, warehouse management, driver and
cargo safety, etc.

Here’s an example of how an MQTT broker, like HiveMQ, can help real-time communication between several vehicles and backend
systems.

86
MQTT Broker Architecture in the Energy Industry
MQTT brokers can play a crucial role in optimizing the operations and OT/IT interoperability in the energy industry. MQTT brokers
can be used in energy use cases such as Asset Performance Management, Predictive Maintenance, Asset Tracking, etc.

Here’s an example of how an MQTT broker, like HiveMQ, can be used for Remote Asset Management in Oil & Gas.

Chapter 26: Next Steps – Choosing • Filtering and routing messages: MQTT brokers can filter
messages based on the subscription topic and determine
the Right MQTT Broker
which client(s) should receive the message.
Choosing the right MQTT broker depends on your architectural • Session management: MQTT brokers can maintain
requirements (a.k.a. Non-Functional Requirements). An MQTT session data for all connected clients, including
broker comparison based on these architectural requirements subscriptions and missed messages, for clients with
should give you insight into how to find the best MQTT broker persistent sessions.
for your needs. • Authentication and authorization: The broker is
responsible for authenticating and authorizing clients
Here are some of the functionalities you should look for in an based on credentials provided by the client. The broker
MQTT broker if you want to use it in large-scale production: is extensible, facilitating custom authentication,
authorization, and integration into backend systems. In
• Handling large numbers of concurrent connections
addition to authentication and authorization, brokers may
reliably: Depending on its implementation, a broker
provide other security features, such as encryption of
has the capability to manage millions of MQTT client
messages in transit and access control lists.
connections reliably and securely, with a robust load
• Scalability and monitoring: An MQTT broker must
balancing feature. This facilitates communication
be scalable to handle large volumes of messages
between diverse devices, networks, and software systems
and clients, integrate into backend systems, be easy
in near real-time.
to monitor, and be failure-resistant. To meet these

87

[Link]

 MQTT Essentials Ebook

requirements, the MQTT broker must use state-of-the- Here are some of the functionalities you should look for in an
art event-driven network processing, an open extension MQTT Broker if you want to use it in small-scale projects:
system, and standard monitoring providers. Brokers
• Ease-of-use: For small-scale projects, especially DIY
may also provide advanced features for managing and
projects and PoCs, an MQTT broker should be easy to
monitoring the MQTT system, such as message filtering,
deploy. There are several open-source and fully-managed
message persistence, and real-time analytics.
MQTT brokers that help you connect devices with minimal
• Clustering: MQTT brokers can support clustering, allowing
effort.
multiple instances of the broker to work together to
• Scalability: MQTT brokers should be able to help you
handle large numbers of clients and messages.
scale as you go so your PoCs turn into production-level
• Provides control over how IoT data flows within your data
projects soon.
pipeline: Prominent MQTT brokers can provide you with
• Cost-efficiency: There is pay-as-you-go pricing available
the ability to have control over your data with visualization
for MQTT brokers so you don’t have to constrain yourself
and management tools, powerful security features, and
while working on a PoC.
an integrated policy engine that can validate, enforce, and
• Security: It goes without saying how important it is
transform data in motion.
to secure your device connectivity from end to end,
irrespective of your scale of the project.
HiveMQ Self-Managed Enterprise Broker offers all the above
features. Hundreds of active customers, across the globe trust
HiveMQ Cloud is a fully managed MQTT broker that offers all
HiveMQ MQTT Broker for reliable, flexible, observable, secure,
the features above. Do check it out.
and scalable data communication.

88
HiveMQ is active in the open source community, working with several
organizations to advance the adoption of the MQTT protocol for IoT.

We are a member of the OASIS MQTT and

MQTT-SN Technical Committees.

We are a member of the OASIS MQTT and

MQTT-SN Technical Committees.

We are a member of the OASIS MQTT and

MQTT-SN Technical Committees.

HiveMQ Cloud: HiveMQ Trial:

Develop, test, deploy, and scale Trial the HiveMQ platform with all of the
production IoT use cases without a enterprise features needed for large-scale
large investment. deployment.

89

[Link]

About HiveMQ

HiveMQ empowers businesses to unlock the full potential

of their data with the most trusted edge-to-cloud IoT data
streaming platform. Built on MQTT’s publish/subscribe
architecture for seamless and flexible integration across
operational technology (OT) assets and information
technology (IT) applications, HiveMQ ensures businesses
can efficiently connect, stream, and govern their data in
real-time. With a focus on reliability, scalability, and security,
HiveMQ helps organizations get their data AI-ready—enabling
advanced analytics, predictive maintenance, and digital
transformation. Leading brands like Audi, BMW, Liberty
Global, Mercedes-Benz, Siemens, and Eli Lilly trust
HiveMQ to modernize their operations, accelerate
innovation, and create smarter, data-driven
experiences.

Visit [Link]
to learn more

[Link]