DAQ & EMBED

Interfacing
Data Acquisition (DAQ)

• Data Acquisition (DAQ) is the process of measuring physical phenomena (like temperature, light, pressure,
vibration) and converting them into digital signals for monitoring, analysis, or control.

Component Role
Measures physical property (e.g.,
Sensor
temperature)
Signal Conditioning Amplifies, filters, or modifies signal
ADC (Analog-to-Digital Converter) Converts analog signal into digital
Microcontroller/DAQ Card Reads data, processes, stores, or transmits
Visualizes or logs the data (e.g., Arduino
Software
IDE, LabVIEW)
Types of DAQ Systems

Type Example Application

PC-based NI DAQ + LabVIEW Industrial R&D
Embedded-based Arduino, ESP32 Education, prototyping
Modular DAQ PXI systems Aerospace, automotive

PC-Based DAQ Systems

Uses a personal computer as the main processing and control hub, with external DAQ devices connected via
USB, PCIe, or Ethernet.
Product Description
Key Features:
Multifunction DAQ with analog I/O
• High sampling rates. NI USB-6009
and digital pins
• Large data storage & visualization.
Educational kit with LabVIEW
• Ideal for lab-scale or industrial use DAQExpress + NI myDAQ
integration
Affordable DAQ for sensors and
LabJack U3-HV
voltage logging
DAQ Systems

• Compact systems with microcontrollers or single-board computers that handle data acquisition, processing, and
sometimes visualization locally.
Platform Description
Basic ADC, I2C/SPI support, large
Arduino UNO / Mega
• Key Features: community
Wi-Fi, Bluetooth, 12-bit ADC,
• Low power ESP32
multi-core processing
• On-device processing High precision ADCs, RTOS,
STM32 Nucleo + Mbed OS
cloud-capable
• Ideal for edge computing & IoT
Embedded Linux with PRUs for
BeagleBone Black
real-time DAQ

• Pros: Teensy 4.1

High-speed 600 MHz Cortex-M7
with powerful ADCs
• Portable, low-cost
• Ideal for prototyping Digital Twins and IoT
• Supports edge AI (ESP32-S3, Jetson Nano)
DAQ Systems

Modular DAQ Systems

• Scalable, plug-and-play systems composed of chassis + I/O modules. Widely used in industrial and high-end
R&D environments.
Platform Description
• Key Features:
Modular DAQ with timing/sync,
• Hot-swappable modules NI PXI / cDAQ
LabVIEW support
• Thousands of I/O combinations Modular switching + measurement
Keysight DAQ970A
• Designed for rugged environments DAQ
Industrial DAQ for structural and
HBM QuantumX
fatigue testing
• Pros: DAQeUSB (ADLINK) Compact DAQ with USB interface
• High precision, modularity
• Suited for mission-critical industrial setups
• Integration with SCADA, MES systems
Emerging/New DAQ Systems

Type Description Example

Texas Instruments SensorTag, Phytec
Wireless DAQ Battery-powered nodes with BLE/Zigbee
DAQ
Sensor-to-cloud systems for remote
Cloud-native DAQ Blynk, Ubidots, Arduino IoT Cloud
monitoring
ML inference on device using onboard
AI-on-Edge DAQ NVIDIA Jetson Nano + IMU sensor DAQ
models
Low-code DAQ GUI and flow-based DAQ programming Node-RED + Raspberry Pi GPIOs
Mass-market USB devices for direct Excel PICO Technology, Digilent Analog
Plug-and-play USB DAQs
logging Discovery 2
Sampling Rate and Signal Aliasing

• Sampling rate is the number of samples taken per second from a continuous analog signal to convert it into a
digital format.
• Measured in Hz (e.g., 1000 Hz = 1000 samples/sec) (Also known as the sampling frequency)

• Nyquist Theorem: To avoid losing information, the sampling rate must be at least twice the maximum frequency
present in the signal.
• This minimum required sampling rate is called the Nyquist rate.
• If your signal contains components up to 200 Hz (e.g., machine vibration), you must sample at ≥ 400 Hz.

Aliasing occurs when a signal is sampled at too low a rate,

Technique Description
causing different signals to become indistinguishable after
digitization. Always sample > 2× max signal
Use proper sampling rate
frequency
• It causes the sampled signal to look like a lower frequency
Analog low-pass filters before
than the original. Apply anti-aliasing filters the ADC to cut off high
• It is irreversible once digitized. frequencies
Adjust rate based on signal type
Dynamic sampling (e.g., slow for temperature, fast
for sound)
Sensing and Communication of Data from the Physical Twin

• In the ecosystem of Digital Twins (DTs), the lifeline between the physical and digital worlds is the sensing and
communication system. A Digital Twin is only as accurate and useful as the data it receives from its real-world
counterpart — the Physical Twin (PT). This makes sensing and communication not merely a technical add-on but
a foundational enabler of the DT’s value proposition.

• Sensors and Their Limits

• Sensing is the activity of detecting, measuring or capturing information from the environment using various
devices.
• It involves converting physical phenomena, such as light, temperature, pressure, sound or chemical processes,
into measurable signals that can be processed, analysed and interpreted.

• Details common sensor types, their operation principles, and limitations such as:
• Sampling frequency issues, Quantization errors, Sensor noise and drift and Immeasurable quantities
Limited Sampling Frequency

• What Is Sampling Frequency?

• Sampling frequency (or sampling rate) is the number of times per second a sensor or data acquisition system
samples (reads) an analog signal.
• It is measured in Hertz (Hz):
E.g., a sampling frequency of 100 Hz means the system takes 100 samples per second.

Problems caused by low sampling: Factors Limiting Sampling Frequency

Problem Description Real-world Example Factor Impact
High-frequency In rotating
Cheap sensors may cap at
behavior appears as equipment, vibration Sensor Hardware Limits
Aliasing 10–50 Hz
incorrect low- spikes may appear
frequency signal inverted Microcontroller ADC Slower microcontrollers
Missing critical Fast-changing speed limit acquisition rate
Undersampling transitions between temperature in
samples furnaces or reactors In IoT, higher sampling =
Network Bandwidth
more bandwidth needed
Lag in autonomous
Poor Virtual twin gets drone response High-frequency logs can
Synchronization outdated, reacts late during rapid Storage or Logging Rate
overflow memory
movement
Protocols
• Set of rules governing the exchange of
data between two entities.

• The key elements of a protocol are

• Syntax: Includes such things as data
format and signal levels.
• Semantics: Includes control information
for coordination and error handling.
• Timing: Includes speed matching and
sequencing.
Protocols and Their Trade-offs in Digital Twins

• In a Digital Twin system, especially in IoT-based implementations, communication protocols define how data is
packaged, transmitted, received, and interpreted between the physical and digital counterparts. Choosing the
right protocol depends on factors like latency, reliability, bandwidth, power consumption, and topology.

Categories of Protocols
1. Device-Level (Sensor to Gateway)
• I2C, SPI, UART, Modbus
• Short-range, high-speed, low-latency wired protocols

2. Network-Level (Gateway to Twin / Cloud)

• MQTT, CoAP, HTTP, AMQP, DDS
• For wide-area, internet-scale communication
Data Acquisition in local environment

Receiver
Gateway
Media
Method Protocol Medium Notes
• Laptop can run
Mosquitto (MQTT
Router (no broker), Node-RED, or
MQTT / Best
Wi-Fi LAN internet
HTTP / TCP
needed)
flexibility a custom Python
server.
For
Bluetooth /
BLE Profile Short-range wearables,
• Raspberry Pi, Jetson
BLE
R small setups Nano, STM32.
Very stable
• Siemens IoT2040
USB/Serial UART / SPI Wired but not (secure)
scalable • Advantech WISE series
Best for fixed (5G)
MQTT /
Ethernet
HTTP
LAN cable industrial • Moxa UC series (Edge)
settings

Sensor → (I²C (1-2) / UART (15)/ Ethernet (100) / Modbus RTU) → Local Gateway → (TCP/IP or other LAN protocol) → Local
Server → Digital Twin Dashboard
Data Flow

• Sensor → (I²C/UART) → Gateway → (TCP/IP) → Digital Twin

I²C (Inter-Integrated Circuit) UART (Universal Asynchronous

• A two-wire communication Receiver-Transmitter)
protocol for short-distance • A two-wire serial
data transfer between communication method for
chips/devices. point-to-point links.
• Data & Clock line • TX & RX
• Master slave architecture • Asynchronous

HTU21D humidity sensor Arduino sensor node →

(I²C) → Raspberry Pi gateway UART → Raspberry Pi
Switch

• A network switch is a device that connects multiple networked devices

and forwards data only to the destination device by mapping hardware
addresses (MAC) to its physical ports, enabling efficient, collision-free
communication.

A LAN switch acts as the traffic controller:

• Each sensor connects to a port on the switch (via Ethernet cable or
Ethernet-over-adapter if needed).
• The switch learns each sensor’s MAC address.
• When a sensor sends data for the Raspberry Pi, the switch sends that data [Sensor 1] \
only to the port the Pi is connected to, instead of spamming every other
sensor. [Sensor 2] \
[Sensor 3] \
• This reduces unnecessary traffic, lowers latency, and helps the Raspberry [ SWITCH ] ---> [ Raspberry Pi ]
Pi receive clean, direct data streams. [Sensor 4] /
[Sensor 5] /
[Sensor 6] /
Gateway (in IoT and networks)

• A gateway (physical or virtual network node) is a bridge or translator between two different systems, typically
between devices that collect data (like sensors) and the cloud or a server that processes it.

• Connectivity Bridging: Different devices speak different

languages (protocols). Gateways translate between Zigbee,
LoRa, BLE, and internet protocols like HTTP and MQTT.
• Centralized Communication: Acts as a single point of contact
for hundreds or thousands of edge devices.
• Enhanced Security: Gateways can encrypt data, authenticate
devices, and protect the network from intrusions.
• Edge Computing: Real-time decisions (e.g., shut down a
system if a sensor detects overheating) without cloud
dependency.
• Latency Reduction: Shortens the time between data collection
and action.
• Data Filtering: Only useful data is sent to the cloud, reducing
bandwidth and storage needs.
• Offline Capabilities: Store and forward data during outages.
OSI Model

• The open systems interconnection (OSI) model was developed by the International
Organization for Standardization (ISO) as a model for a computer communications
architecture and as a framework for developing protocol standards. It consists of seven layers:
Principles of OSI Model

A layershould be created where a Each layer should perform a well-defined

different abstraction is needed. function.

The function of each layershould be The layer boundaries should be chosen to

chosen with an eye toward defining minimize the information flowacross the
internationally standardized protocols. interfaces.

The number of layers should be large

enough that distinct functions need not
be thrown togetherin the same layer out
of necessity and small enough that the
architecture does not become unwieldy
TCP/IP Reference Model (Transmission Control Protocol)
• The Link Layer describes what links such as serial lines and classic Ethernet must do to
meet the needs of this connectionless internet layer. (Packet Switching)

• The Internet Layer's job is to permit hosts to inject packets into any network and have
them travel independently to the destination (potentially on a different network).
• The internet layer defines an official packet format and protocol called IP (Internet Protocol),
plus a companion protocol called ICMP (Internet Control Message Protocol) that helps it
function.

• The Transport Layer segments the incoming byte stream into discrete messages and
passes each one on to the internet layer.
• TCP (Transmission Control Protocol), is a reliable connection-oriented protocol that allows a byte
stream originating on one machine to be delivered without error on any other machine on the
internet.
• UDP (User Datagram Protocol), is an unreliable, connectionless protocol for applications that do
not want TCP’s sequencing or flow control and wish to provide their own.
TCP/IP

• TCP/IP stands for Transmission Control Protocol / Internet Protocol.

It’s the fundamental communication framework that allows devices to send and receive data reliably across
networks — including local networks (LAN) and the internet.

• Think of TCP/IP as the postal system for data:

• IP = Assigns an address to each “house” (device) so data knows where to go.
• TCP = Ensures the “letters” (data packets) arrive safely, in the right order, without loss.
TCP/IP Layer Role Example Protocols Sending temperature data from a Raspberry Pi to a cloud-
based DT dashboard
Interfaces for user
Application applications to HTTP, MQTT Transport Layer
send/receive data • TCP breaks the message into packets, numbers them, and
End-to-end delivery & ensures they are delivered and reassembled in order.
Transport TCP, UDP
reliability
Internet Layer
Logical addressing & • IP adds the source and destination addresses (e.g.,
Internet IP (IPv4/IPv6), ICMP
routing Raspberry Pi’s IP → Cloud server’s IP).
Physical data
Ethernet, Wi-Fi, Network Access Layer
Network Access (Link) transmission over the
Modbus TCP • Uses Ethernet or Wi-Fi hardware to send packets as
network medium
electrical signals or radio waves.
Modbus
Modbus

• The Modbus protocol was created in 1979 by Modicon* as a means of sharing data between their
PLCs.
• Modicon took the approach of openly publishing its specification and allowing its use by anyone without
asking for royalties.
• These factors, along with the simplicity of the protocol itself, allowed it to become the first widely
accepted de facto standard for industrial communication.
• Modbus-IDA a nonprofit organization oversees the evolution of the protocol and seeks to drive its adoption
by continuing to openly distribute the protocol specifications and providing an
infrastructure for device compatibility certification.
Modbus Interaction a n d Data Models

• The Modbus communication protocol provides a means whereby one device may read and
write data to memory areas located on another remote device.
• The typical example of a remote device is an RTU (remote terminal unit) providing a
physical interface (inputs and outputs) to an industrial process device taking the
initiative to read and write data is usually a PLC.
• Two devices interact over the Modbus protocol using the client–server interaction model.

• It's the server manufacturer to decide the semantics attributed to the data located in these
memory areas.
Modbus Serial

• A Modbus serial network comprises multiple devices connected to a physical medium

(e.g., a serial bus).
• Due to the shared nature of the physical medium, a mechanism is required to regulate
medium access.
• For this reason, Modbus serial follows the master–slave interaction model.
• Client becoming the master, and the Modbus servers taking the role of slaves. The master
is responsible for initiating the communication by sending requests to the slaves, one
request at a time.
• The request/reply exchange can be performed in one of two ways:
• • Unicastmode: The master sends a request to a specific slave. The slave processes this
request and replies to the master.
• • Broadcast mode: The master sends a request to all slaves. The slaves process this
request, but do not reply to the master.
• Each Modbus serial network may only have one master. The number of slaves is limited
to 247.
Frames

• The request/reply exchange is performed using request and reply APDUs, which are transmitted
within frames that always have the same structure :
• Address:This field contains the slave address. In a request frame, it identifies the destination device, while
in a reply frame it identifies the sender. Each slave has a unique address in the range 1–247. Address 0 is used (by the
master) for broadcasting messages. The range 248–255 is reserved.
• Modbus APDU: This field is the application layer.
• CRC: This field is used for error detection purposes. The content depends on the transmission
mode (RTU or ASCII) being used.
Modbus TCP
• A Modbus TCP network consists of multiple devices connected through a TCP/IP
network, interacting following the client–server model.
• Connection establishment and management are handled by the TCP/IP protocol and
occur independently of the Modbus protocol.
• Some Modbus server devices may support multiple connections from distinct clients
simultaneously.
• Similarly, clients may establish multiple connections to distinct servers.
Ethernet

Ethernet is a wired networking technology that defines:

• How devices are physically connected with cables
• How data is formatted into frames
• How devices share the same medium without talking over each other.

• OSI Model → Lives in the Data Link layer (Layer 2) and Physical layer (Layer 1)
• TCP/IP Model → Fits into the Link layer

• It decides how two devices on the same network exchange frames.!

Ethernet

Data Link Layer

Physical Layer Medium Access Control (MAC)
• Data is packaged into Ethernet
• Devices are connected frames • Originally used CSMA/CD
with cables: • Source MAC address (who (Carrier Sense Multiple Access
• Twisted pair (Cat5e, sent it) with Collision Detection)
Cat6, Cat6a…) • Destination MAC address → devices “listen” before
• Fiber optics (for longer (who it’s for) sending, and back off if they
distances/high speed) • Payload (data from higher detect a collision.
• Electrical or optical signals layers, e.g., an IP packet) • Modern Ethernet uses switches,
carry the bits. • CRC checksum (to check so collisions are rare.
for errors)
TCP UDP

TCP (Transmission Control Protocol) UDP (User Datagram Protocol)

• TCP is connection-oriented and focuses on • UDP is connectionless and focuses on speed

reliability. over reliability.
Both parties must connect before talking, and
they keep checking if the other is listening.
• No Handshake
• The sender just sends the data — no prior
Connection Setup – Three-Way Handshake agreement.
• Before sending data, TCP establishes a • Each message is a datagram, sent
connection: independently.
• SYN – Sender says “I want to talk”
• SYN-ACK – Receiver says “Okay, I’m ready”
• ACK – Sender confirms, and the connection is
open.
Data Acquisition through Internet

I2C, UART, Wi-Fi

/Ethernet +
Http

Cloud Platform / Server

Application Layer (HTTP

packet: "Humidity=62%")

Transport Layer (TCP

ensures it arrives intact)

Network Layer (IP address Data Link Layer (MAC Physical Layer (Wi-Fi
locates the cloud server) address of router and Pi) radio waves)
List of Common Device-to-Device Communication Protocols

Wired (Short/Medium Range)

Protocol Year Introduced Key Points Typical Use

UART 1960s Simple point-to-point async Microcontroller PC or gateway

I²C 1982 Two-wire, multi-device Sensors, EEPROM, RTC

SPI 1980s Fast, full-duplex, more wires Displays, high-speed ADC/DAC

1-Wire 1990 Single data wire + ground Temperature sensors (DS18B20)

Robust, multi-device, error- Automotive ECUs, industrial
CAN bus 1986
checking control
RS-232 1960s Legacy serial over longer distance Modems, legacy equipment
Differential signaling, long
RS-485 1983 Industrial Modbus RTU
distances
Modbus RTU/TCP 1979 / 1999 Industry standard PLC SCADA/DT gateway

EtherCAT 2003 Real-time Ethernet Robotics, motion control

Profinet 2003 Industrial Ethernet Factory automation

List of Common Device-to-Device Communication Protocols

Wireless (Short/Medium Range)

Protocol Year Key Points Typical Use

Bluetooth Classic 1999 Higher data rate Audio devices
Bluetooth Low Energy (BLE) 2010 Low power, short bursts Wearables, beacons
ZigBee 2004 Mesh networking, low power Smart home, lighting
Z-Wave 2001 Mesh, sub-GHz Home automation
Wi-Fi 1997 High data rate Cameras, gateways
Thread 2014 IPv6 mesh over 802.15.4 Smart home (Google Nest)
UWB (Ultra Wideband) ~2019 Precise location (<10 cm) Asset tracking, AR/VR
List of Common Device-to-Device Communication Protocols

Wireless (Long Range)

Protocol Year Key Points Typical Use

LoRa/LoRaWAN 2015 Long range, low power Agriculture, smart cities
Sigfox 2010 Ultra-narrowband Remote monitoring
Utility meters, environmental
NB-IoT 2016 Cellular IoT standard
sensing
LTE-M (Cat-M1) 2016 Cellular IoT Asset tracking
Ultra-low latency, high Industrial DT, autonomous
5G NR 2020
bandwidth vehicles
Data Acquisition through Internet

Circuit Switching Packet Switching

Node A Node B
Node B
Node A

• Dedicated communication path

• No Dedicated communication path
• Physical links between nodes
• Data transmitted along dedicated • Data sent in packets
paths • Data stored in nodes are passed
further
ROUTING IN CIRCUIT-SWITCHED (synchronous) NETWORKS

• Resilience & Efficiency

• Static Routing

• Dynamic Routing
1. Alternate Routing

If the preferred path between sender and receiver

is busy, the system tries predefined alternate
paths.
These paths are fixed and chosen in a specific
order.
Routes are dynamically selected based on
Example: If Path A is busy, try Path B, then Path C.
network conditions like traffic, congestion, or
failures.
It optimizes performance but adds overhead
2. Adaptive Routing (!Overhead, @Optimization) (extra processing and updates).
Example: If a link fails or gets congested, the
system finds the best available route in real-time.
 Adaptive Routing

Assume, Each switch i communicates the following

traffic measurements to the central controller
A = Parameter in the range [0,1] that reflects the availability of switch t.
It is 1 if t functions normally, but it is less if 't' is overloaded; its role is to
make alternative routes that transit through overloaded switches less
attractive and, hence, less likely to be chosen by the network controller.

Max {At x Min [Iit, Itj]) t # ij

PA = Protective-allowance parameter for the direct traffic on link x-y; its

role is to divert traffic away from the link when it is nearly fully Adaptive route selection
occupied

Max {At x Min [Iit-PA, Itj-PA]) t # ij

Packet Switching (~Virtualization)
• Shortcoming of Circuit Switching
[Link] Usage
[Link] Data Transfer rate

• Switching Approaches
1. Datagram (Independent Entities, Receiver's Burden, !Overhead)
2. Virtual Circuit (Preplanned, Asynchronous, Special Control Pckt--> !Overhead)

• Advantages
[Link] Initial setup delay time
[Link] for Congestion
3. Reliable
Comparision
Packet Switching Routing Strategies

• Rationale-- > Low cost, Higher Reinforcement

Packet Switching Routing Strategies
• Flooding

The flooding technique has three remarkable

properties:

1. All possible routes between source and destination

are tried. Thus, no matter what link or node
outages have occurred, a packet will always get
through if at least one path between source and
destination exists.
2. Because all routes are tried, at least one copy of the
packet to arrive at the destination will have used a
minimum-hop route.
3. All nodes that are directly or indirectly connected
to the source node are visited.
Packet Switching Routing Strategies
• Random Routing
• Random routing has the simplicity and robustness of flooding with far less traffic load. With random routing, a
node selects only one outgoing path for retransmission of an incoming packet. The outgoing link is chosen at
random, excluding the link on which the packet arrived. If all links are equally likely to be chosen, then a node
may simply utilize outgoing links in a round-robin fashion.

• Adaptive Routing
• Bellman Ford Algorithm
Congestion Control
• The objective here is to maintain the number of packets within the network below the level at
which performance falls off dramatically.
• If the rate at which packets arrive and queue up exceeds the rate at which packets can be
transmitted, the queue size grows without bound and the delay experienced by a packet goes to
infinity.
• Even if the packet arrival rate is less than the packet transmission rate, queue length will grow
dramatically as the arrival rate approaches the transmission rate
• Strategies at the point of Saturation
• Stop and Discard --> Latency
• Love thy Neighbour
Congestion Control
As the load increases, utilization increases for a while.
Then as the queue lengths at the various nodes begin to grow, throughput actually drops because the buffers at
each node are of finite size.
When a node's buffers are full, it must discard packets. Thus, the source stations must retransmit the discarded
packets in addition to the new packets; this only exacerbates the situation: As more and more packets are
retransmitted, the load on the system grows, and more buffers become saturated

Preventive Measures:

1. Alert Signal as control packet

[Link] carry the information while
transit
Traffic Rate Management

• The simplest way to cope with congestion is for the frame relaying network to simply discard
frames arbitrarily, with no regard to the source of a particular frame.

• To provide for a fairer allocation of resources, the frame relaying bearer service includes the
concept of a committed information rate (CIR). This is a rate, in bits per second, that the network
agrees to support for a particular frame-mode connection. Any data transmitted in excess of the
CIR is vulnerable to discard in the event of congestion.

• Committed Burst Size (Bc). The maximum amount of data that the network agrees to transfer,
under normal conditions, over a measurement interval T. These data may or may not be
contiguous (i.e., it may appear in one frame or in several frames).

• Excess Burst Size (Be). The maximum amount of data in excess of Bc that the network will attempt
to transfer, under normal conditions, over a measurement interval T. These data are uncommitted
in the sense that the network does not commit to delivery under normal conditions. Put another
way, the data that represent Be are delivered with lower probability than the data within Bc.
Congestion Avoidance with Explicit Signaling

• Backward explicit congestion notification (BECN). Notifies the user that

congestion avoidance procedures should be initiated where applicable for traffic in
the opposite direction of the received frame. The notification indicates that the
frames transmitted by the user on this logical connection may encounter congested
resources.

• Forward explicit congestion notification (FECN). Notifies the user that congestion
avoidance procedures should be initiated where applicable for traffic in the same
direction as the received frame. The notification indicates that this frame, on this
logical connection, has encountered congested resources.
Latest / Emerging

• Matter Protocol (2022) – For seamless smart home device interoperability (runs over Thread, Wi-Fi, Ethernet).

• Wi-Fi HaLow (802.11ah) – Long-range, low-power Wi-Fi for IoT.

• 6LoWPAN – IPv6 over Low-Power Wireless Personal Area Networks (enabling DT devices to be IP-addressable).

• TSN (Time-Sensitive Networking) – Ethernet extension for deterministic latency in Industry 4.0.

• Li-Fi – High-speed data transfer using light (still niche but promising).
Carrier Sense Multiple Access Protocols

• Required--> To be able to detect what other stations are doing, and thus adapt their
behavior accordingly
• 1-persistent CSMA
• When a station has data to send, it first listens to the channel to see if anyone else is
transmitting at that moment.
• If the channel is idle, the stations sends its data. Otherwise, if the channel is busy, the
station just waits until it becomes idle.
• ! If a collision occurs, the station waits a random amount of time and starts all over again
• The Problem of bandwidth-delay product
 Carrier Sense Multiple Access Protocols

• Non-persistent CSMA
• Non-greedy Algorithm
• When a station senses the channel when it wants to send a frame, and if no one else
is sending, the station begins doing so itself..
• If the channel is already in use, the station does not continually sense it for the purpose of
seizing it immediately upon detecting the end of the previous transmission.
• ! If a collision occurs, the station waits a random amount of time and starts all over again
• Longer delays than 1-persistent CSMA
• The Problem of bandwidth-delay product
Carrier Sense Multiple Access Protocols
• p-persistent CSMA
• When a station becomes ready to send, it senses the channel.
• If it is idle, it transmits with a probability p.
• With a probability q = 1 − p, it defers until the next slot.
• If that slot is also idle, it either transmits or defers again, with probabilities p and q.
• This process is repeated until either the frame has been transmitted or another station has
begun transmitting.
Comparative Analysis of Carrier Sense Multiple Access Protocols
Collision Free Protocol
• A Bit-Map Protocol
• A station j may announce that it has a frame to send by inserting a 1 bit into slot j. After all
N slots have passed by, each station has complete knowledge of which stations wish to
transmit.
• ! Since everyone agrees on who goes next, there will never be any collisions

• The desire to transmit is broadcast before the actual transmission are called reservation
protocols because they reserve channel ownership in advance and prevent collisions
Collision Free Protocol
• Token Passing
• The token represents permission to send.
• If a station has a frame queued for transmission when it receives the token, it can send
that frame before it passes the token to the next station.
• If it has no queued frame, it simply passes the token.

• Fiber Distributed Data Interface

• Resilient Packet Ring
Collision Free Protocol
• Binary Countdown
• Drawbacks==> Overhead bits!
• A station wanting to use the channel now broadcasts its address as a binary bit string,
starting with the high order bit
• To avoid conflicts, an arbitration rule must be applied:
• as soon as a station sees that a high-order bit position that is 0 in its address has been
overwritten with a 1, it gives up.
MQTT
Inception (1999)
• Developed by Andy Stanford-Clark (IBM) and Arlen Nipper (Arcom, now Eurotech).
• Goal: Create a lightweight messaging protocol for oil pipelines — low bandwidth, unreliable networks, and
remote locations.
• Designed for machine-to-machine (M2M) communication.

Open Standard (2010–2013)

• IBM released MQTT as a royalty-free protocol.
• Standardized by OASIS in 2013 (MQTT v3.1).

Modern Era (2014–Present)

• MQTT 3.1.1 became an ISO standard in 2016.
• MQTT 5.0 released in 2019 — more features like better error reporting, shared subscriptions, and user properties.
• Widely used in IoT, Industrial IoT, Smart Homes, Automotive, and Cloud services.
Message Queuing Telemetry Transport MQTT

How MQTT Works (Simple View)

• Architecture: Publish–Subscribe model with a Broker in the middle.

Key Components:
• Publisher → Sends messages (e.g., sensor data).
• Broker → Receives all messages, manages topics, delivers to subscribers.
• Subscriber → Receives messages from topics it has subscribed to.
Three Dimensions of Decoupling

• Space Decoupling: No direct knowledge or

connection (e.g., IP/port) between pub and sub.
• Time Decoupling: Publisher and subscriber
operate independently in time.
• Synchronization Decoupling: Publishing and
receiving do not interrupt component operations.
MQTT - Message Filtering in Pub/Sub Systems

Content-Based Filtering Type-Based Filtering

Subject-Based Filtering
Filters by message content using Filters by message type/class
Filters by topic or subject string.
query language. (common in OOP).
Example: A subscriber to
Example: A subscriber interested Example: A subscriber listens for
weather/india receives only
in "temperature > 30°C" gets only messages of type ErrorEvent or its
weather updates for India.
hot weather alerts subtypes.

Messages stay in Message Messages are lost

the queue until Persistence if no one
consumed. subscribes.
. Each message is One-to-One All subscribers to
delivered to only Delivery a topic receive the
one consumer. message.

Queues must be Explicit Setup Topics are created

named and on the fly, no
created before setup needed.
use.
MQTT - Client, Broker and Connection Establishment

Client Broker
• An MQTT client is any • The broker is responsible
device (from a micro for receiving all messages,
controller up to a full- filtering the messages,
fledged server) that runs determining who is
an MQTT library and subscribed to each
connects to an MQTT message, and sending the
broker over a network. message to these
subscribed clients.
QoS in MQTT

QoS (Quality of Service) defines how reliably a message is delivered between the publisher and subscriber.

QoS 0 – "At Most Once“ QoS 1 – "At Least Once"

QoS 2 – "Exactly Once"
Message is always delivered,
Message is delivered once and only
• No guarantee of delivery. but may be duplicated.
once.
• Message is sent once, no retries.
• Fastest, but may lose messages. Broker sends acknowledgment; if
Most reliable, but slowest due to extra
• Example: Temperature updates not received, message is resent.
steps.
every second—missing one isn’t Example: Sending a payment
Example: Updating a database—
critical. confirmation—better to duplicate
duplicates could cause errors.
than lose.

fire and forget.

a four-part handshake)
No ack
MQTT Publish, Subscribe & Unsubscribe
• Publishing: Clients can send messages to a broker immediately after
connecting.
• Topic-Based Routing: Each message includes a topic (e.g.,
home/livingroom/temp) used by the broker to deliver it to subscribers.
• Payload: The actual data (text, binary, JSON, etc.). MQTT is data-agnostic—
any format is allowed.
• QoS (Quality of Service): (what kind of guarantee a message has for reaching
the intended recipient (client or broker).)
➢ 0: At most once (no guarantee)
➢ 1: At least once (may be duplicated)
➢ 2: Exactly once (no duplicates)
• Retain Flag: Stores the last message for a topic so new subscribers get the
latest value.
• Packet Identifier: Uniquely tags messages (used in QoS 1 & 2).
• DUP Flag: Marks a message as a duplicate if it's being resent due to no
acknowledgment.
MQTT Publish, Subscribe & Unsubscribe
Packet Identifier
• Uniquely tags each subscribe message (used for QoS > 0).

List of Subscriptions
• A single SUBSCRIBE message can include multiple topics with different QoS
levels.
• Example: Subscribe to home/+/temperature and alerts/fire in one request.

Wildcard Support
• Topics can include wildcards to match patterns.
• Example: home/+/temperature matches all rooms' temperature topics.

QoS Conflict Resolution

• If multiple subscriptions overlap, the broker delivers the message using the
highest QoS level.
• Example: If subscribed to home/# with QoS 1 and home/livingroom with QoS 2,
messages to home/livingroom are delivered with QoS 2.
MQTT SUBACK – Acknowledging Subscriptions

• Purpose: Sent by the broker to confirm receipt of a SUBSCRIBE

request.
• Packet Identifier
Matches the one in the original SUBSCRIBE message to track which
request is being acknowledged.
• Return Codes
One code per topic/QoS pair in the SUBSCRIBE message:
• Shows the granted QoS level for each topic.
• If a subscription is rejected (e.g., due to permissions or invalid
topic), a failure code is returned.

• After SUBACK
The client starts receiving messages for all successfully subscribed
topics.
MQTT - Topics

• Topic refers to an UTF-8 string that the broker uses to filter messages for each connected client. The topic
consists of one or more topic levels. Each topic level is separated by a forward slash (topic level separator).

• Wildcards (Subscribe many topics)

subscription to myhome/groundfloor/+/temperature
MQTT - Retained Messages

• A retained message is a normal MQTT message with the retained flag set to true. The broker stores the last
retained message and the corresponding QoS for that topic.
• Retained messages help newly-subscribed clients get a status update immediately after they subscribe to a
topic. The retained message eliminates the wait for the publishing clients to send the next update.

• Client A publishes a retained message to

myhome/livingroom/temperature.
• Sometime later, client B subscribes to myhome/#.
• Client B receives the myhome/livingroom/temperature retained
message directly after subscribing to myhome/#.
• Client B (the subscribing client) can see that the message is a
retained message because
• the broker sends retained messages with the retained flag set to true.
The client can decide
• how it wants to process the retained messages.
MQTT- Last Will and Testament

• Why it's needed: MQTT often runs on unreliable networks. Devices may
disconnect unexpectedly (e.g., power loss, weak signal).
• What is LWT?
A special message set by a client when it connects to the broker.
If the client disconnects ungracefully (without sending a DISCONNECT), the
broker sends the LWT message to a predefined topic.
• LWT Message Includes:
• Topic: Where the message will be published.
• Payload: The actual message (e.g., “Device offline”).
• QoS & Retain Flag: Controls delivery and storage.
• Graceful Disconnect: If the client disconnects properly, the
broker discards the LWT message.
MQTT - Keep Alive and Client Take-Over
• Problem: In unreliable networks, a client may disconnect unexpectedly (e.g., crash, battery drain). This can cause a half-
open connection, where one side thinks the connection is still active.
• Solution – Keep Alive:
• When a client connects, it sets a keep-alive interval (in seconds).
• If no messages are sent during this time, the client must send a PINGREQ to check if the broker is still alive.
• The broker replies with PINGRESP.

Timeout Rule:
If the broker doesn’t hear from the client within 1.5×
keep-alive interval, it disconnects the client.
Similarly, the client disconnects if it doesn’t get a
response from the broker.
Hypertext Transport Protocol (HTTP)

History of HTTP

• Inception (1989–1991)
• Created by Tim Berners-Lee at CERN as the foundation of the World Wide Web.
• The idea: exchange hypertext documents (HTML) between servers and clients.
• Early Versions
• HTTP/0.9 (1991): Extremely simple, only supported GET requests.
• HTTP/1.0 (1996): Added status codes, headers, and more request methods.
• HTTP/1.1 (1997): Persistent connections, caching, chunked transfer — still widely used today.
• Modern Era
• HTTP/2 (2015): Multiplexing (multiple requests in one connection), header compression, faster browsing.
• HTTP/3 (2022): Uses QUIC (UDP-based) for lower latency, better security.
Hypertext Transport Protocol (HTTP)

How HTTP Works

•Client-Server Model
• Client (e.g., browser, IoT device)
sends a request.
• Server responds with a response.

•Stateless Protocol
• Each request is independent; server
doesn’t “remember” previous
requests (cookies/sessions help
maintain state).
HTTP Request Structure

• A request is what the client (browser, IoT device, etc.) sends to a server.

• GET → Method (tells the server what action to perform)

• GET: retrieve resource
• POST: send data GET /[Link] HTTP/1.1
• PUT: update data Host: [Link]
• DELETE: remove data User-Agent: Chrome/120
• /[Link] → Path (what resource you want on the server)
• HTTP/1.1 → Version of HTTP used

Headers (next lines)

• Host: [Link]
• Specifies the target server (important since many sites share the same IP).
• User-Agent: Chrome/120
• Identifies the client (browser, IoT device, custom app).
HTTP Request Structure
• A response is what the server sends back.

• Line 1 → Status Line

• HTTP/1.1 → Version used by server
• 200 → Status code HTTP/1.1 200 OK
• 200 = Success Content-Type: text/html
• 404 = Not Found Content-Length: 1256
• 500 = Server Error
• OK → Reason phrase (textual meaning of status code) <html> ... </html>

Headers
• Content-Type: text/html
• Tells client what’s in the body (HTML, JSON, image, etc.).
• Content-Length: 1256
• Number of bytes in the body → helps client know when message ends.
• Body
• Actual content requested (HTML page, JSON, file, etc.).
Using HTTP Locally

• Run a web server on Raspberry Pi → access it with

[Link] or [Link] on your laptop. Representational State Transfer – Application
Programming Interface
IoT setup
• REST is an architectural style (not a protocol) for
•A sensor gateway exposes a local REST API: designing networked applications.
•Request: GET [Link] • It uses HTTP as the underlying protocol.
•Response: { "temp": 25.4 } • In REST, everything is treated as a resource
• Each resource is identified by a URI (Uniform
Private Digital Twin Resource Identifier).
•A local DT dashboard running on your lab server → all
devices send HTTP data locally → no internet involved.
Applications in IoT & Digital Twins

• IoT Devices → Cloud: Sensors expose REST APIs for

reading data.
• Digital Twin Control: Digital Twin can send a PUT
request to update actuator states (e.g., change motor
speed).
• Dashboards: Web apps fetch data with REST APIs to
display machine status in real-time.
CoAP (Constrained Application Protocol)

• A lightweight web transfer protocol designed for:

• Constrained devices (low CPU, memory)
• Constrained networks (low bandwidth, lossy wireless links)

How CoAP Works

• Runs on UDP (unlike HTTP,

which uses TCP). HTTP for tiny
• Supports Request/Response Message Types IoT devices
model (like HTTP) but in a
much lighter way. CoAP has 4 message types:
• Uses binary, compressed • Confirmable (CON): Must be acknowledged
headers → very small packet (reliable).
size. • Non-confirmable (NON): Fire-and-forget.
• Built for machine-to-machine • Acknowledgement (ACK): Confirms a CON
(M2M) communication message.
• Reset (RST): Tells sender something went
wrong.
Databases
 DATA BASE

A data base can be defined as a collection of data in a single location

designed to be used by different programmers for a variety of
applications.

More specifically it is a collection of logically related data stored

together in a set of files intended to serve one or more applications in an
optimal fashion.

A database must also have a predetermined structure and organization

suitable for access, interpretation, or processing either manually or
automatically.
OBJECTIVES OF DATABASE

1. Reduce or eliminate redundant data

2. Integrate existing data
3. Provide security
4. Share data among users
5. Incorporate changes quickly and effectively
6. Exercise effective control over data
7. Simplify the method of using data
8. Reduce the cost of storage and retrieval of data
9. Improve accuracy and integrity of data
ISSUES OF CONCERN IN DATABASE

Issues to be considered while implementing a database are:

1. High investment in hardware and software

2. Need to use larger and faster hardware

3. Necessity to have highly trained manpower

4. Redundancy to take care of eventualities like crash of the database server.

5. Need to ensure integrity and reliability of data

FEATURES OFA DATABASE MANAGEMENT SYSTEM

A database management system consists of a collection of interrelated data and a set of

programs to access that data. Database management involves:

1. Organize a database.

2. Add new data to the database.

3. Sort the data in some meaningful order.

4. Search the database for types of information.

5. Print the data into formatted reports.

6. Edit the data.

7. Delete the data.

Problems of traditional file systems that can be overcome by DBMS are

1. Data dependence
2. Rigidity
3. Static nature
4. Lack of integration
5. Data duplication
6. Inconsistency
7. Difficulty in sharing information
8. Inefficiency
9. Inability to handle ADHOC requests
 DATABASE MODELS

1. HIERARCHICAL DATABASE
The nodes in level 2 are the children of node at level 1. The nodes at level 2 in turn
become parents of nodes in level 3 and so on.
2. NETWORK DATABASE

The network database is a combination of several hierarchies in which child files can
have more than one parent file, thereby establishing a many-to- many relationship
among data.

A hierarchical model is actually a subset of a network model. Examples of network

database languages are TOTAL and IDMS.

In both hierarchical and network databases data relationships are predefined and
embedded in the structure of the database. Access to data is processed by associated
application programs.
A limitation of both hierarchical and network systems is the restriction they place on
data access.

• They both require that the rules of data access be defined when the data structure is
defined. The access rules are difficult to modify after the database has been
implemented.

• They are suited for batch operations that are highly structured and repetitive
involving high transaction rates.
3. RELATIONAL DATABASE MANAGEMENT SYSTEMS (RDBMS)
Data is organized in the form of a table for a large variety of manufacturing
applications.

In a RDBMS (Relational Database Management System) an entity is an object that is

distinguishable from other objects.

An entity set is a set of entities of the same type and is represented by a set of
attributes.

For each attribute there is a set of permitted values for domains.

An entity relationship model (E-R Model) is based on the perception of the real world
which consists of a set of entities and relationships among them.
• The relational database eliminates the need to follow predefined access paths to
reach target data, and makes data access more flexible.

• The database user gains quicker access to information since the database provides
direct access to all data. The access is independent of the way it is stored.

• Hence relational database facilitates unanticipated queries and makes it well suited
to the manufacturing environment.
Features of RDBMS include:
• ADHOC or unanticipated queries. This is typical in a manufacturing environment.
• Relational database is dynamic. The relationships change and are extended
frequently in a manufacturing database.
• Suitable where enterprise information has to be available to a large number of users
for decision making.
• Desirable where application specifications, development and maintenance costs are
to be kept at the minimum level.

• Compatible with distributed databases.

Some of the important features of RDBMS are given below:

1. Guaranteed logical accessibility.

2. Dynamic on-line catalogue.
3. Views that could be modified depending on need.
4. High-level insert updates and delete.
5. Physical data independence.
6. Logical data independence.
7. Integrity independence.
8. Distribution independence.
RDBMS

• Structured and organized data

• Structured query language (SQL)

• Data and its relationships are stored in separate tables.

• Data Manipulation Language, Data Definition Language

• Tight Consistency
Schema in RDBMS

• The schema helps programmers

accurately understand the size and
complexity of a project before creating a
database and its relationships.
• schemas are used for constructing database
management systems (DBMSs) and
relational database management systems
(RDBMSs).
QUERY LANGUAGE

A query language is one with which a user requests information from a database.
Two categories of query languages are:
• Procedural
• Non-procedural

There are a number of commercial query languages available today. They can be
classified as:
SQL - Structured Query Language
QUEL - Query Language
QBE - Query by Example
 Column-oriented databases

• Most databases have a row as a unit of storage which, in

particular, helps write performance.

• However, there are many scenarios where writes are rare, but we
often need to read a few columns of many rows at once.

• In this situation, it’s better to store groups of columns for all rows
as the basic storage unit—which is why these databases are
called column stores.

• Example of Column-oriented databases : BigTable, Cassandra etc.

 Document Oriented databases

• A collection of documents

• Data in this model is stored inside documents.

• A document is a key value collection where the key allows access to its value.

• Documentsare not typically forced to have a schema and

therefore are flexible and easy to change.

• Documents can contain many different key-value pairs, or key-array pairs, or even

nested documents.

• Example of Document Oriented databases : MongoDB, CouchDB etc.

Graph databases

• A graph database stores data in a graph.

• A graph database is a collection of nodes and edges
• Each node represents an entity (such as a student or business) and each edge
represents a connection or relationship between two nodes.

• Every node and edge are defined by a unique identifier.

• Each node knows its adjacent nodes.
• Example of Graph databases : OrientDB, Neo4J, Titan. etc.
Brewer’s CAP
Theorem
• The theorem states that within a large-scale
distributed data system, there are three requirements
that have a relationship of sliding dependency:
Consistency, Availability, and Partition Tolerance.

• Consistency : All database clients will read the same

value for the same query, even given concurrent updates.

• Availability : All database clients will always be able to

read and write data.

• Partition Tolerance : The database can be split into

multiple machines; it can continue functioning in the
face of network segmentation breaks.
• The system will be allowed to loose arbitrarily
many messages sent from one node to another.
Figure shows the general focus of some of the different databases.

Where different databases appear on the CAP continuum

• In this depiction, relational databases are on the line between Consistency
and Availability.

• Graph databases such as Neo4J and the set of databases derived at least in
part from the design of Google’s Bigtable database (such as MongoDB,
HBase, Hypertable, and Redis) all are focused slightly less on Availability and
more on ensuring Consistency and Partition Tolerance.

• Finally, the databases Cassandra, Project Voldemort, CouchDB, and Riak are
more focused on Availability and Partition-Tolerance.

• However, this does not mean that they dismiss Consistency as unimportant.
CA :
• Single site cluster, therefore all nodes are always in contact, when partition
occurs system block.

CP :
• Some data may not be accessible, but the rest is still consistent/accurate.

AP :
• System is still avaiable under partioning, but some of the data returned
may be inaccurate.
Database Scalability & Reliability: Sharding, Replication, and Partitioning

Modern applications (IoT, Digital Twins, Social Media, E-commerce) deal with massive, real-time data.

A single monolithic RDBMS server is insufficient due to:

•Data volume (billions of rows)

•Throughput needs (thousands of transactions/sec)
•Fault tolerance (systems must stay up 24/7)

Solution: Distributed Database Techniques → Sharding, Replication, Partitioning.

 Sharding
Horizontal partitioning of a database into smaller, independent pieces called shards.
Goal: Handle huge datasets by distributing across multiple servers. (Scalability & Load Distribution.)

Key Ideas:

1. Each shard = subset of rows.

2. Each shard runs on its own server (scale out).

3. Application logic decides which shard a query goes to.

• The load is balanced out nicely between servers—for example, if we have ten servers, each one only
has to handle 10% of the load.
 Sharding - Strategies

Range-based Sharding A hot shard happens

•Divide by value ranges (e.g., user_id 1–1M in Shard when one shard
A, 1M–2M in Shard B). receives
•Simple, but risk of "hot shards.“ disproportionately
more traffic or data
Hash-based Sharding compared to others.
•Apply hash function to key (e.g., hash(user_id) % N).
•Balances load better. Sensor Data in Digital Twins (IoT)

Geographic / Directory-based Sharding Scenario: Digital twin of a smart city →

•Data split by region (e.g., EU customers in one shard, thousands of IoT sensors (traffic, air quality,
energy usage).
US in another). Shard Key: sensor_id.
Hash Sharding: hash(sensor_id) % N.
• Sensors spread across multiple shards, even
if some sensors send more frequent data.
• Prevents one factory’s or one district’s
sensors from overwhelming a single shard.
 Sharding - Strategies

How Hash-Based Sharding Works

Choose a Shard Key

Pick a field that uniquely identifies the data. hash(user_ Shard
user_id %4
Examples: user_id, order_id, sensor_id. id) (Server)
Apply a Hash Function 101 8901 1 Shard 1
Compute a hash value of the key (e.g., hash(user_id)).
202 7318 2 Shard 2
Hash ensures values are spread evenly (good
distribution). 303 11543 3 Shard 3
Modulo Operation (% N) 404 6540 0 Shard 0
Take the hash value modulo N (number of
505 12897 1 Shard 1
shards/servers).
Result = the shard number.
Store Data in the Shard
The system directs the data entry to the correct server
based on that result.
 Replication
Definition: Creating and maintaining copies of the same data on multiple servers.
Goal: Availability, Fault-tolerance, Read Scalability.
• Sharding does little to improve resilience when
used alone. Master-Slave (Primary-Replica)
•Writes go to the master.
•Reads can be distributed across replicas.
• Although the data is on different nodes, a node •Example: MySQL Replication.
failure makes that shard’s data
unavailable just as surely as it does for a single-
Multi-Master Replication
server solution.
•Multiple nodes can accept writes.
•Conflict resolution required.
• The resilience benefit it does provide is that only •Example: CouchDB, some PostgreSQL
the users of the data on that shard will suffer; clusters.
however, it’s not good to have a database with
part of its data missing.
Master-Slave Replication

• With master-slave distribution, we replicate data across

multiple nodes.

• One node is designated as the master, or primary.

• This master is the authoritative source for the data and is

usually responsible for processing any updates to that
data.

• The other nodes are slaves, or secondaries.

• A replication process synchronizes the slaves with the

master
 • Masters can be appointed manually
Master-slave replication is most helpful or automatically.
for scaling when we have a read intensive
dataset.
• Manual appointing typically means
• We can scale horizontally to handle • A second advantage of master- that when we configure our cluster,
more read requests by adding more slave replication is read we configure one node as the
slave nodes and ensuring that all read resilience: Should the master master.
requests are routed to the slaves. fail, the slaves can still handle
• We are still, however, limited by the read requests
• With automatic appointment, we
ability of the master to process • Again, this is useful if most of
create a cluster of nodes and they
updates and its ability to pass those our data access is reads.
• The failure of the master select one of themselves to be the
updates on.
does eliminate the ability to master.
Consequently it isn’t such a good scheme handle writes until
for datasets with heavy write traffic. either the master is restored or a • Apart from simpler configuration,
new master is appointed. automatic appointment means that
the cluster can automatically
appoint a new master when a
master fails, reducing downtime.
Peer-to-Peer Replication

• Master-slave replication helps with read scalability but

doesn’t help with scalability of writes.

• It provides resilience against failure of a slave, but not of a

master.

• Essentially, the master is still a bottleneck and a single point

of failure.

• Peer-to-peer replication attacks these problems by not

having a master.

• All the replicas have equal weight, they can all accept
writes, and the loss of
any of them doesn’t prevent access to the data store.
• With a peer-to-peer replication cluster, we can ride over node failures without losing access to
data.

• Furthermore, we can easily add nodes to improve our performance.

• There’s much to like here—but there are complications.

• The biggest complication is, again, consistency.

• When we can write to two different places, we run the risk that two people will attempt to
update the same record at the same time—a write-write conflict.
• Inconsistencies on read lead to problems but at least they are relatively transient.
Inconsistent writes are forever.
Partitioning (Performance & Manageability. )

• Partitioning = splitting a table into smaller parts for easier management & performance.

• Different from Sharding:

• Partitioning = within one database system.
• Sharding = across multiple servers.

Types of Partitioning:
• Horizontal Partitioning (rows → similar to sharding).
• Vertical Partitioning (columns → e.g., separating large text fields).
• Range, List, Hash Partitioning (built-in RDBMS techniques).
Data Warehouse
Data Warehouse
• The term Data Warehouse was coined by Bill Inmon in 1990, which he defined in the following
way:
• A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of
data in support of management's decision making process

• Subject Oriented: Data that gives information about a particular subject instead of about a
company's ongoing operations.
• Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into
a coherent whole.
• Time-variant: All data in the data warehouse is identified with a particular time period.
• Non-volatile: Data is stable in a data warehouse. More data is added but data is never removed.
This enables management to gain a consistent picture of the business.
Data Warehouse
• A central location where consolidated data from multiple locations (databases) are stored.
• Data Warehouse is maintained separately from the operational database
• End Users access it whenever any information if needed.
• Note: Data Warehouse is not loaded with new data every time.
OLAP vs OLTP
Online Transaction Processing OLTP
• Information technology to help the knowledge worker (executive, manager) make faster and better
decisions
• On-line analytical processing (OLAP) is an element of decision support systems (DSS)
• Decision support places some rather different requirements on database technology compared to
traditional on-line transaction processing applications
• OLTP → Database
• OLTP applications typically automate clerical data processing tasks such as order entry and
banking transactions that are the bread-and-butter day-to-day operations of an organization.
• These tasks are structured and repetitive, and consist of short, atomic, isolated transactions. The
transactions require detailed, up-to-date data, and read or update a few (tens of) records accessed
typically on their primary keys.
• Consistency and recoverability of the database are critical, and maximizing transaction throughput
is the key performance metric.
• Consequently, the database is designed to reflect the operational semantics of known applications,
and, in particular, to minimize concurrency conflicts.
Data Warehouse Online Analytical Processing (OLAP)
• Data warehouses, in contrast, are targeted for decision support.
• Historical, summarized and consolidated data is more important than detailed, individual records.
• The workloads are query intensive with mostly ad hoc, complex queries that can access millions of
records and perform a lot of scans, joins, and aggregates.
• Query throughput and response times are more important than transaction throughput.
• To facilitate complex analyses and visualization, the data in a warehouse is typically modeled
multidimensionally.
• OLAP operations include rollup (increasing the level of aggregation) and drill-down (decreasing
the level of aggregation or increasing detail) along one or more dimension hierarchies,
slice_and_dice (selection and projection), and pivot (re-orienting the multidimensional view of
data).
• Data warehouses might be implemented on standard or extended relational DBMSs, called
Relational OLAP (ROLAP) servers. These servers assume that data is stored in relational databases,
and they support extensions to SQL and special access and implementation methods to efficiently
implement the multidimensional data model and operations
Examples
• OLAP
• Bank Manager wants to know how many customers are utilizing the ATM of this branch. Based on
which he’ll take a call to continue the ATM location
• An insurance company wants to know the number of policies each agent has sold. This will help in
better performance management of agents.

• OLTP
• A supermarket server which records each single product purchased at the market.
• A bank server which record every time a transaction is made for a particular account.
Data Mart
• A smaller version of Data Warehouse which deals with single subject
• Focused on one area.
• Simple and easy way to build.
• Limited source.
Database Design Methodology
• Fact Table
• A ‘fact table’ is a group of associated data items.
• It consists of values of dimensions and measure.
• This means that a fact table can be defined from the given dimension and measure.
• A fact table typically consists of two types of columns such as foreign keys and measure.
• Foreign keys are linked to dimension tables and measures consist of numeric facts as shown in
Figure
Database Design Methodology
• Dimension
• The term ‘dimension’in data warehousing is a
collection of reference information about a
measurable event. The descriptive information that gives
context to facts.
• These events are stored in a fact table and are Examples:
known as facts. • Time Dimension → date,
month, year
• The dimensions are generally the entities for which
• Location Dimension → city,
an organization wants to preserve records. region, country
• The descriptive attributes are organized as • Product Dimension → name,
columns in dimension tables by a data warehouse. category, brand

• For example, a student’s dimension attributes Think of Dimension Table = Labels

could consist of first and last name, roll number, for data.
age, gender, or an address dimension that would
include street name, state, and country attributes.
Database Design Methodology
• In de-normalization, smaller tables are merged
to form larger tables to reduce joining
operations. De-normalization
• De-normalization is particularly performed in • In normal databases (OLTP), we keep data
those cases where retrieval is a major normalized (split into many small tables to reduce
requirement and insert, update, and delete redundancy).
operations are minimal, as in case of historical • In data warehouses, we de-normalize → merge data
data or data warehouse.
into fewer, bigger tables for faster queries.
• Measure • Reason: Analytics values speed > storage efficiency.
• Measure is the term used for the values that
rely on the dimensions. For example, Analogy:
amount sold, quantity sold, etc. Normalized = a super-organized library with books in
many small sections.
De-normalized = fewer, bigger sections, so students
• Entity Relationship diagrams and
can find books faster (even if some books are
normalization techniques are popularly used
duplicated).
for database
design in OLTP environment
Database Design Methodology- Star Schema

• The star schema is one of the simplest of data warehouse

schemas.
• It is known as star because it appears like a star with points
expanding from a center.
• Each dimension in a star schema represents a one-
dimensional table only and the dimension table consists
of a set of attributes.
• Dimension tables comprise of relatively small numbers of
records in comparison to fact tables, but each record may
consist of a large number of attributes to describe the fact
data.
• Fact tables usually consist of numeric facts and foreign
keys to dimensional data.
Database Design Methodology- Star Schema
• The main characteristics of star schema are as follows:
➢ It has high query performance as it requires less join operations due to de-normalization of data.
➢ It has a simple structure which is easy to understand.
➢ It takes a relatively longer time to load the data into dimension tables due to de-normalization, that
also causes data redundancy that may increase the size of table data.
➢ It is the most generally used and simplest structure in the data warehouse and supported by an
enormous number of tools.
➢ Data integrity cannot be applied in star schema
because its database is de-normalized and
➢ It has simple queries as it will not require join
contains redundant data.
operations due to de-normalization of data.
➢ Inserts and updates in star schema can result in
➢ In comparison to highly normalized schemas,
data anomalies due to data redundancy which
star schema has basic business reporting
normalized schemas are designed to avoid.
logic. It has high query performance due to
➢ It is not flexible in terms of analytical needs when
fewer join operations.
compared to a normalized data model.
➢ It has fast aggregations due to its simpler
➢ It is too specific as it is designed for a fixed
queries.
analysis of data and does not allow complex
analytics.
Database Design Methodology- Snowflake Schema

• The major difference between star and snowflake

schema is that the snowflake schema may consist of
normalized dimensions while star schema always
consists of de-normalized dimensions.
• Thus, snowflake schema is a modification of the star
schema supporting normalization of dimension tables.
• Some dimension tables are normalized in snowflake
schema which splits up the data into additional tables.
• Thus, ‘Snowflaking’ is a process of normalizing the
dimension tables in a star schema
Database Design Methodology- Snowflake Schema
• It is important to note that redundancy is reduced in snowflake schema due to normalization.
• the main goal of snowflake schema is to store the normalized data efficiently but it does so at the
significant cost of query performance

➢ The snowflake schema results in saving of

storage space due to normalized attributes
despite additional complexity in source query
joins.

➢ Some OLAP multi-dimensional database

modelling tools are optimized for snowflake
schemas.

➢ It has complex queries due to join operations

because dimensions are normalized.
➢ It has poor query performance due to joins
required for normalized data
Database Design Methodology- Galaxy Schema
• The main difference between star/snowflake and fact
constellation schema is that star/snowflake schema
consist of only one fact table while fact constellation
schema always consists of multiple fact tables.
• Fact constellation is a measure of online analytical
processing, which is a collection of multiple fact tables
sharing dimension tables. This is an improvement
over Star schema.
• The sales fact table in the fact constellation schema is similar
to the one in star schema. The sales fact table here, also
consists of four attributes, namely, time_key, item_code,
branch_code, location_id and two measures such as
rupees_sold and units_sold as shown in Figure
• The major advantage of this schema is that it provides
better support to end user because it has multiple fact
tables.
• The major disadvantage of this schema is its
complicated design because of consideration of
different variants of aggregation.