1.
How will the ship agent know which topics to subscribe to, and how will that subscription
config be delivered — will it be preconfigured via the desktop app the same way tags are
configured?
1. Architecture Overview
The Standard Ship Data Framework uses a .NET 8 Windows Service as the onboard data
acquisition agent. It reads directly from ship hardware using industrial protocols and publishes
normalised data to a cloud MQTT broker. There is no local MQTT broker on the ship.
Sensors / Equipment
↓ (Modbus / NMEA / CIP / Serial / HTTP)
.NET 8 Windows Service (Ship Data Agent)
↓ online: publish directly
↓ offline: buffer to Postgres + Timescale DB local instance, drain on
reconnect
Cloud MQTT Broker
↓
[Link] Core API + PostgreSQL + Timescale DB
Analytics / Dashboard / Reporting
The ship agent has four internal layers that handle data from acquisition to transmission:
Layer Responsibility
Driver Layer Reads raw data from hardware — Modbus, NMEA, CIP, Serial, HTTP.
Each protocol has its own driver.
Normalization Maps raw device values to standard fleet parameters. Adds deviceId,
Layer shipId, unit, componentId.
Postgres Buffer Stores readings locally when satellite link is down. Drained in order on
reconnect.
MQTT Publisher Publishes normalised payload to cloud broker on the correct fleet topic.
2. Why There Is No Local MQTT Broker(on ship side the
agent directly connects to MQTT)
In a typical IoT setup with off-the-shelf devices, a local broker is needed because devices are
hardcoded to publish MQTT locally and cannot reach the internet directly. This framework is
different:
1. The ship agent reads from hardware using industrial protocols (Modbus registers, NMEA
sentences, CIP tags) — not MQTT subscriptions.
2. There is no intermediate publisher between devices and the agent. The agent IS the
publisher.
3. Offline resilience is handled by PostgresSQL, not by a local broker with persistent sessions.
4. This eliminates an entire infrastructure component, reduces latency, and simplifies ops.
Key distinction
The ship agent does not subscribe to MQTT anywhere. It only publishes MQTT — to the cloud
broker. All data acquisition happens via driver protocols, not via MQTT.
3. Normalised Topic Structure
3.1 The Problem with Deep Topic Hierarchies
A naive topic design maps every device and parameter into the topic path:
fleet/{shipId}/{componentId}/{deviceId}/{parameterId}
Example:
fleet/IMO-1234567/mainengine/krohne-001/MEMINFLOW
fleet/IMO-1234567/mainengine/moxa-002/RPM
fleet/IMO-1234567/mainengine/moxa-002/LOAD
This explodes combinatorially.
Example:
5 ships x 10 components x 5 devices x 8 parameters = 2,000 topics
The cloud broker stores every subscribed topic in persistent sessions.
Backend services would need to know every individual device ID to subscribe correctly —
which is impossible across a heterogeneous fleet.
3.2 Normalised Design
Topic format
fleet/{shipId}/{componentType}
The topic path carries only what backend services actually need for routing.
Device identity, component location, and parameter name move into the payload where they
belong.
Example:
fleet/IMO-1234567/flowmeter
fleet/IMO-1234567/bridge
fleet/IMO-1234567/shaft-power
fleet/IMO-1234567/AE-Power Counter
Different ship, identical shape:
fleet/IMO-9999999/flowmeter
fleet/IMO-9999999/engine
Topic count formula:
Topic count = Ships x Component Types
Example:
20 ships x 8 component types = 160 topics total
This remains constant regardless of how many sensors each ship carries.
3.3 Payload Structure
All device-specific identity is carried in the payload.
The Normalization Layer inside the ship agent is responsible for constructing this before
publishing.
{
"shipId": "IMO-1234567",
"deviceId": "krohne-001",
"componentId": "mainengine/me1",
"parameter": "MEMINFLOW",
"value": 20,
"unit": "m3/h",
"timestamp": "2026-05-18T12:22:00Z"
}
Field Purpose
shipId Which ship this reading came from. Critical for all downstream storage and
display.
deviceId The specific physical device (krohne-001, moxa-002). Defined in device
registry.
componentId Where on the ship this device is located (mainengine/me1, auxengine/ae2).
parameter The specific measurement (MEMINFLOW, RPM, LOAD, HEADING).
value The normalised reading after driver conversion.
unit Engineering unit (m3/h, rpm, kW, degrees).
timestamp UTC time of acquisition on the ship agent, not publish time.
4. Device Registry (Ship Configuration)
Each ship has a configuration file loaded by the ship agent at startup.
This is the single source of truth for:
What devices exist on this vessel
How to read them
Where to publish their data
There is no localTopic field.
The agent reads from hardware directly via drivers, not from a local broker.
Example registry:
{
"shipId": "IMO-1234567",
"devices": [
{
"deviceId": "krohne-001",
"type": "flowmeter",
"componentId": "mainengine/me1",
"protocol": "Modbus",
"register": "40001",
"pollInterval": 5,
"forwardTopic": "fleet/IMO-1234567/flowmeter"
},
{
"deviceId": "moxa-002",
"type": "engine",
"componentId": "mainengine",
"protocol": "Modbus",
"register": "40010",
"pollInterval": 5,
"forwardTopic": "fleet/IMO-1234567/engine" //the forward topic helps
us to know what topics to subscribe in order to backend to recieve.
},
{
"deviceId": "gps-001",
"type": "nav",
"componentId": "bridge",
"protocol": "NMEA",
"sentence": "GGA",
"pollInterval": 1,
"forwardTopic": "fleet/IMO-1234567/nav"
}
]
}
Key points
1. forwardTopic is always:
fleet/{shipId}/{type}
This creates a consistent topic shape across all ships.
2. protocol + register/sentence replaces localTopic from local-broker designs.
The driver layer uses this to know how to acquire the reading.
3. pollInterval tells the Polling Engine how frequently to read this device.
4. The registry can grow or shrink per ship.
The topic structure never changes.
Registry update process
When a device is replaced at port, the ops team pushes an updated registry to the ship agent
over the existing satellite/4G uplink.
The agent picks it up on next restart.
Ship crew are not involved in this workflow.
5. Offline Handling
When satellite connectivity is lost, the ship agent continues reading from all devices.
Readings are written to Postgres instead of being published to the cloud broker.
On reconnect, the agent drains the buffer in captured order.
This guarantees:
Zero data loss
Correct chronological order in the central database
Connection state behaviour
State Agent behaviour
Online Driver reads device → Normalize → publish directly to cloud broker
Offline Driver reads device → Normalize → write to SQLite buffer
Reconnect Drain SQLite buffer in order → publish to cloud broker → mark published
6. Cloud Backend Wildcard Subscriptions
Because every ship uses the same normalised topic shape, backend services subscribe with
stable wildcards that never change regardless of fleet size or device count.
Service Wildcard What it receives Note
subscription
Flowmeter fleet/+/flowmeter All flowmeter data, all Matches any shipId
service ships
Engine fleet/+/engine All engine data, all Matches any shipId
monitoring ships
Service Wildcard What it receives Note
subscription
Power fleet/+/power All power data, all Matches any shipId
monitoring ships
Per-ship fleet/IMO-1234567/+ All data from one ship Matches any
dashboard component type
Raw ingestion / fleet/# Everything, all ships Catch-all for storage
DB
Scaling behaviour
Adding a new ship requires:
Zero backend subscription changes
Adding a new device type requires:
One new wildcard subscription for that type
Nothing else changes.
7. Access Control
The cloud broker enforces ACL rules at the connection level.
Topic design and access control are kept separate.
The topic structure is NOT designed to restrict access.
The broker ACL handles access restriction.
Examples:
# Ship agent publishes only to its own shipId subtree
fleet/IMO-1234567/# → ship-agent-IMO-1234567 → publish only
# Backend services subscribe to what they need
fleet/+/flowmeter → flowmeter-service → subscribe only
fleet/# → ingestion-service → subscribe only
# Third-party scoped to one vessel
fleet/IMO-1234567/# → client-vessel-xyz → subscribe only
8. Summary
Decision Approach Example Why it scales
No local Postgres handles Ship agent reads Simpler ops, one fewer
broker offline hardware directly infra component
3-level topic fleet/{shipId}/{type} 160 topics for 20 ships x Never grows with sensor
8 types count
Device In payload only deviceId, componentId, Topic stays stable
identity parameter across fleet
Agent config Registry file protocol + register Driver layer owns
replaces localTopic acquisition
Backend Broker wildcards fleet/+/{type} Zero changes for new
routing ships
Final Architectural Shift From v1
The key architectural shift from v1:
localTopic no longer exists
The ship agent acquires data directly from hardware via driver protocols.
MQTT is only ever an outbound transmission concern, not an inbound subscription concern on
the ship side.
so for this we will use device_registery.json as the starting point.
Topic hierarchy and broker routing
Question
What should the topic hierarchy look like — something like
fleet/{ship_id}/{component_id}/{device_id}/{parameter_id} — and how does the
broker route a published message to only the right subscriber?
Why deep hierarchies break
A naive design puts everything in the topic path:
fleet/{ship_id}/{component_id}/{device_id}/{parameter_id}
// Example:
fleet/IMO-1234567/mainengine/krohne-001/MEMINFLOW
fleet/IMO-1234567/mainengine/moxa-002/RPM
fleet/IMO-1234567/mainengine/moxa-002/LOAD
This explodes combinatorially:
5 ships × 10 components × 5 devices × 8 parameters = 2,000 broker topics
Every new sensor = new topic stored on the broker. Every backend service would need to know
individual device IDs to subscribe — which is impossible across a heterogeneous fleet where
Ship A has 3 flowmeters and Ship B has 7.
Solution — normalised 3-level topic
fleet/{ship_id}/{component_type}
Examples — identical shape for every ship regardless of device count:
fleet/IMO-1234567/flowmeter
fleet/IMO-1234567/engine
fleet/IMO-1234567/nav
fleet/IMO-9999999/flowmeter
fleet/IMO-9999999/engine
Topic count = ships × component types — never grows with sensors or parameters.
20 ships × 8 component types = 160 topics, permanently.
How broker routing works
When the ship agent publishes to fleet/IMO-1234567/flowmeter , the broker matches it
against all active subscriptions and delivers to every match:
fleet/IMO-1234567/flowmeter → exact match subscribers only
fleet/IMO-1234567/+ → all component types on this ship
fleet/+/flowmeter → flowmeter data across all ships
fleet/# → everything across all ships
The broker does this natively. No application code needed for routing.
Each backend service subscribes only to what it needs — the broker handles the rest.
Payload structure
device_id and parameter_id are not in the topic path — they go in the payload. The ship
agent's Normalization Layer constructs this before publishing:
{
"shipId": "IMO-1234567",
"deviceId": "krohne-001",
"componentId": "mainengine/me1",
"parameter": "MEMINFLOW",
"value": 20,
"unit": "m3/h",
"timestamp": "2026-05-18T12:22:00Z"
}
Once the message reaches the backend via the broker, deviceId and parameter become
columns in PostgreSQL. The formula engine and analytics layer query and aggregate by these
fields — they never needed broker-level routing on them, only database-level filtering.
Backend wildcard subscriptions
Service Subscription Receives
Flowmeter service fleet/+/flowmeter All flowmeter data, all ships
Engine monitoring fleet/+/engine All engine data, all ships
Per-ship dashboard fleet/IMO-1234567/+ All data from one ship
Raw ingestion / DB fleet/# Everything, catch-all for storage
Adding a new ship = zero changes to backend subscriptions.
Adding a new device = registry entry only, topic structure unchanged.
Rule
Topic levels are for broker routing.
Device identity and parameter name go in the payload for backend processing.
Payload validation
Problem 1 — Empty or malformed payload
The broker routes on topic string only. It delivers empty, null, or malformed JSON payloads
without complaint. The backend receives it and either crashes or silently drops the reading —
no visibility into what was lost.
Problem 2 — Missing fields inside valid JSON
Even if JSON parses correctly, individual fields like deviceId , parameter , or value could be
null or missing. The backend then stores an incomplete row — or worse, passes null into the
formula engine and produces garbage output downstream.
Fix — validate at the ship agent before publishing
The ship agent is the only place that knows what a valid payload looks like per device.
Validation must happen there, not on the backend.
Driver reads value from hardware
↓
Normalization layer builds payload
↓
Validation check — complete and well-formed?
↓ yes ↓ no
Publish to cloud broker Write to SQLite buffer
with error flag + raw value
retry on next poll or alert ops
Required field rules enforced by agent before every publish:
Field Rule
shipId Non-null, matches registry
deviceId Non-null, exists in registry
componentId Non-null, exists in registry
parameter Non-null, known parameter name
value Numeric, not null, within sensor range
timestamp Valid UTC ISO 8601
Second layer — defensive parsing on backend
Even with agent-side validation, the backend never trusts the payload blindly:
if ([Link](payload))
return; // log and discard
var reading = [Link]<SensorReading>(payload);
if (reading?.Value is null || reading?.DeviceId is null)
// write to dead letter table — do not process
The dead letter table holds every message that failed backend validation. No reading is silently
lost — it is quarantined for ops investigation.
Rule
Validate at the ship agent — it is the only place that knows what a valid payload looks like
per device.
Backend parsing is a safety net, not the primary defence.