MODULE 5

IoT and Data Analytics

 IoT analytics
IoT analytics refers to collect , process and analyze data that are generated by IoT devices. As
more devices are connected in the internet , it generate a large amount of data that provides a
valuable insights and provide valuable information from that particular data. IoT can be the
subset of Bigdata and it consist of heterogenous streams that combined and transformed to
correct information.

The Significance of Data Analytics in IoT

 Data analytics is a process of analyzing unstructured data to give meaningful conclusions.
Numerous of the methods and processes of data analytics are automated and algorithms
designed to process raw data for humans to understand.
 IoT devices give large volumes of precious data that are used for multiple applications. The
main goal is to use this data in a comprehensive and precise way so that it's organized, and
structured into a further usable format.
 A Data analytics uses methods to process large data sets of varying sizes and characteristics,
it provides meaningful patterns, and extracts useful outputs from raw data.
 Manually analyzing these large data sets is veritably time consuming, resource intensive, and
expensive. Data analytics is used for saving time, energy, resources and gives precious
information in the form of statistics, patterns ,and trends.
 Organizations use this information to improve their decision making processes, apply further
effective strategies, and achieve desired outcomes.
Roles of Data Analysts in IoT
The roles of data analysts within organizations are contingent on their knowledge, skills, and
expertise. Here are seven prominent roles that data analysts fulfil:
1. Determining Organizational Goals : A data analyst's most crucial part is helping a business
define its primary organizational objectives. This original step is vital for setting a business
apart, outperforming challengers, and attracting the right audience. Data analysts collaborate
with staff and team members to monitor, track, gather, and analyze data, necessitating access
to all available data used within the organization.
2. Data Mining: Data analysts gather and mine data from internet sources and company
databases, conducting analysis and research. This research helps businesses understand
market dynamics, current trends, competitor activities, and consumer preferences.
3. Data Cleaning: Data analysts play an essential part in data cleansing, a critical aspect of data
preparation. Data cleansing involves correcting, identifying, and analyzing raw data,
significantly improving decision making by providing accurate and precise data.
4. Data Analysis: Data analysts offer data entry services that include data analysis. They
employ ways to efficiently explore data, excerpt relevant information, and give accurate
answers to business-specific questions. Data analysts bring statistical and logical tools to the
table, enhancing a business's competitive advantage.
5. Recognizing Patterns and Identifying Trends: Data analysts excel in recognizing trends
within industries and making sense of vast datasets. Their expertise in identifying industry
trends enables businesses to enhance performance, estimate strategy effectiveness, and more.
6. Reporting: Data analysts convert essential insights from raw data into reports that drive
advancements in business operations. Reporting is vital for monitoring online business
performance and safeguarding against data misuse. It serves as the primary means to measure
overall business performance.
 7. Data and System Maintenance: Data analysts also contribute to maintaining data systems
and databases, ensuring data coherence, availability, and storage align with organizational
requirements. Data analysts employ ways to enhance data gathering, structuring, and
evaluation across various datasets.
IoT Analytics - Use case
Here are several IoT analytics use cases across various industries
Smart Agriculture
 Use Case: Monitoring crop conditions
 How IoT Analytics Helps: Analyzing data from detectors measuring soil humidity,
temperature, and thundershower conditions to optimize irrigation and enhance crop yield.
Healthcare
 Use Case: Remote patient monitoring
 How IoT Analytics Helps: using wearable devices to collect and analyze patient health data,
enabling proactive healthcare interventions and reducing hospital admissions.
Manufacturing
 Use Case: Quality control and predictive maintenance
 How IoT Analytics Helps: Monitoring product line data to identify defects, ensure product
quality, and predict equipment maintenance needs to minimize time-out.
Smart metropolises
 Use Case: Traffic management
 How IoT Analytics Helps: Analyzing data from connected traffic cameras, sensors, and GPS
to optimize traffic flow, reduce congestion, and enhance overall urban mobility.

 IoT Analytics challenges

IoT data analytics challenges include managing high-volume, noisy, and heterogeneous data
streams (velocity/variety) and ensuring data security and privacy. Key hurdles involve scaling
infrastructure for millions of devices, achieving interoperability between different vendor systems,
addressing low-latency requirements via edge computing, and mitigating data quality issues like
missing values.
Here are the primary IoT data analytics challenges:
 Data Volume and Velocity: IoT devices generate immense, continuous streams of data, often
overwhelming traditional storage and processing systems. Real-time analysis is required to prevent
latency and extract actionable insights, often requiring specialized tools like Apache Kafka or
Spark.

 Security and Privacy: Protecting sensitive data from unauthorized access, managing device
authentication, and preventing cyberattacks (e.g., malware, default password hacking) are major
concerns.

 Data Quality and Heterogeneity: IoT data is frequently incomplete, inaccurate, or noisy due to
sensor errors, requiring complex data cleaning and validation processes. Data is also multi-modal,
originating from diverse formats and protocols.

 Scalability and Interoperability: As IoT ecosystems grow, systems must scale to handle millions
of devices. Interoperability challenges arise when trying to integrate data from various vendors,
devices, and communication protocols, hindering seamless analysis.
 Data Management and Integration: Integrating data from scattered sensor devices, gateways, and
external sources into a centralized or distributed system is complex, necessitating robust edge
computing approaches.

 Machine Learning and Model Updates: Applying AI/ML requires managing "concept drift,"
where IoT data patterns change over time, requiring consistent updating of models.

Key Solutions & Strategies:

 Edge Computing: Processing data near the source to reduce bandwidth and latency.

 Hybrid Storage: Using a mix of edge and cloud storage to balance speed and capacity.

 AutoML: Utilizing automated machine learning to efficiently update models and reduce the need
for specialized data scientists.

 Data Governance: Implementing strict data management protocols to ensure integrity.

 Cloud-based IoT analytics

Cloud-based IoT analytics involves collecting, storing, and analyzing vast, streaming data from
connected devices using cloud infrastructure (AWS, Azure, Google Cloud) to derive real-time
insights, predict failures, and optimize operations. These platforms leverage scalable cloud
computing for data processing, machine learning for predictive maintenance, and visualization
tools to enable informed, automated decisions.

IoT analytics for the cloud refers to the use of cloud computing platforms to collect, store, process,
and analyze data generated by IoT devices such as sensors, smart appliances, and industrial
machines. With the rapid growth of IoT systems, an enormous amount of data is continuously
produced. Traditional systems are not capable of handling such large-scale, high-velocity data,
which is why cloud-based analytics has become essential. The cloud provides scalable
infrastructure, powerful processing capabilities, and advanced analytics tools that help transform
raw IoT data into meaningful insights.

In a typical IoT cloud analytics system, the process begins with the data generation layer, where
sensors and devices collect real-world data such as temperature, humidity, motion, or pressure. This
data is then transmitted through communication protocols like MQTT or HTTP to an IoT gateway
or directly to the cloud. The gateway may perform preliminary filtering or aggregation to reduce
unnecessary data transmission.

Once the data reaches the cloud, it enters the data ingestion and storage layer. Cloud platforms store
this data in various types of databases such as time-series databases for continuous sensor data or
NoSQL databases for unstructured data. The cloud’s ability to store vast amounts of data efficiently
makes it ideal for IoT applications.

The next stage is data processing, which can be either real-time (stream processing) or batch
processing. Real-time processing is used in applications where immediate action is required, such
as detecting equipment failure or monitoring health parameters. Batch processing, on the other
hand, is used for analyzing historical data to identify trends and patterns. Technologies like
distributed computing frameworks are often used to handle this processing efficiently.
 After processing, the analytics layer applies different techniques such as statistical analysis,
machine learning, and predictive modeling. These techniques help in extracting valuable insights
from the data. For example, predictive analytics can forecast equipment failures, while diagnostic
analytics can identify the root cause of a problem. Prescriptive analytics goes a step further by
suggesting actions based on the analysis.

The final stage is visualization and decision-making, where the analyzed data is presented through
dashboards, graphs, and alerts. This helps users and organizations to monitor systems, make
informed decisions, and take timely actions. Visualization tools provide an intuitive interface to
understand complex data easily.

Cloud-based IoT analytics offers several advantages. It provides scalability, allowing systems to
handle millions of connected devices. It is cost-effective due to its pay-as-you-use model,
eliminating the need for heavy infrastructure investment. It also enables remote access, making it
possible to monitor and control systems from anywhere. Additionally, cloud platforms support
integration with advanced technologies like artificial intelligence and big data analytics.

However, there are also challenges associated with IoT cloud analytics. Data security and privacy
are major concerns, as sensitive information is transmitted and stored over the internet. Latency can
be an issue in applications requiring real-time responses. There is also the challenge of managing
and processing massive volumes of data, as well as ensuring compatibility among different devices
and platforms.

IoT cloud analytics is widely used in various real-world applications. In smart cities, it helps in
traffic management, waste management, and pollution monitoring. In environmental monitoring, it
is used for weather forecasting and disaster prediction. In industrial IoT, it enables predictive
maintenance and improves operational efficiency. In agriculture, it supports smart irrigation
systems and soil monitoring to enhance crop yield.

Key Components & Architecture

 Data Ingestion: IoT Hubs or Event Hubs ingest high-velocity data using protocols like MQTT or
REST.

 Storage: Cloud storage solutions (e.g., Data Lakes, NoSQL databases) manage both structured and
unstructured data

 Processing & Analytics: Stream processing (e.g., Apache Spark) provides real-time analysis,
while batch processing handles historical data, enabling AI-driven insights.

 Visualization & Action: Dashboards and automated systems trigger alerts or actions.

Top Cloud IoT Analytics Platforms

 AWS IoT Analytics: Pre-processes, analyzes, and stores IoT data.

 Microsoft Azure IoT Hub: Enables bidirectional communication and integrates with Azure Data
Explorer for near real-time analytics.

 Google Cloud IoT Core: Connects, manages, and ingests data securely.

 ThingSpeak: Specialized in sensor data visualization and MATLAB analysis.

 Benefits of Cloud IoT Analytics
 Scalability: Effortlessly handle growing volumes of device data.

 Predictive Maintenance: Reduce downtime by forecasting equipment failures.

 Improved Decision Making: Real-time data processing enables immediate, informed actions.

 Cost Optimization: Enhance operational efficiency and reduce energy usage.

Common Use Cases

 Smart Factories: Monitoring production efficiency, optimizing processes, and tracking asset
health.

 Predictive Maintenance: Analyzing vibration, temperature, or sensor data to predict when

machinery needs service.

 Environmental Monitoring: Analyzing environmental sensors for agricultural or urban

management.

Challenges
 Security & Privacy: Ensuring data security and adhering to privacy regulations.

 Data Volume/Velocity: Managing the massive, constant influx of data efficiently.

 Strategies to organize Data for IoT Analytics

Effective IoT data organization strategies include implementing data governance, using time-
series/NoSQL databases (e.g., InfluxDB, Cassandra), and utilizing hybrid storage (edge + cloud) to
manage high-volume, diverse data. Key techniques involve using data lakes for ingestion,
structuring data with Metadata/data catalogs, and pre-aggregating data for faster analytics.

Key Strategies to Organize IoT Data:

 Data Modeling & Structuring: Implement specialized database types.

 Time-Series Databases (TSDB): Ideal for storing timestamped sensor metrics (e.g., InfluxDB,
Prometheus), allowing efficient querying of temporal data.

 NoSQL/Document Databases: Use MongoDB or Couchbase for high throughput and flexible,
schema-less structures, enabling rapid ingestion of varied JSON data.

 Graph Databases: Use these to map relationships between complex IoT systems, assets, and
processes, particularly in industrial IoT.
 Hybrid Storage Architecture (Edge + Cloud): Use edge computing to store/process immediate,
high-frequency data for low latency, while streaming aggregated or historical data to cloud data
lakes (e.g., S3) for long-term analytics.

 Data Preprocessing & Enrichment: Before storage, clean data, normalize units, and enrich it with
metadata (e.g., device location, firmware version).

 Data Tiering & Retention Policies: Create policies based on data age and value. High-value,
granular data might be kept for 30 days, while summarized, historical, or low-priority data is
retained longer to optimize storage costs.

 Semantic Modeling & Cataloging: Implement a metadata catalog that defines what each sensor
measurement means (e.g., "temp_01" = "Boiler 3 Intake Temp") to make disparate data sources
searchable and usable.

 Stream Processing & Aggregation: Use tools like Apache Kafka or Spark Streaming to aggregate
data in real time (e.g., calculating hourly averages) before it lands in the database, reducing the
storage volume.

 Linked Analytics Data Sets

Linked Analytics Data Sets in IoT refer to the integration of massive, heterogeneous, and often
unstructured datasets generated by IoT devices (sensors, gateways, actuators) with internal business
data or external information sources to gain actionable insights. These datasets combine real-time
sensor streams with contextual data—such as operational logs, user profiles, or environmental
information—to enable advanced, predictive, and prescriptive analytics.
Key Components of Linked IoT Data Sets
IoT data is rarely valuable in isolation. Linking it to other data sources enables a complete view of
an entity.
 Raw Sensor Data: Numerical data from IoT sensors, such as temperature, pressure, humidity,
motion, and location coordinates.

 Operational/Business Data: Internal organizational data like product schedules, maintenance logs,
supply chain transactions, and Customer Relationship Management (CRM) records.

 Meta Data: Information obtained from the network connectivity layer that helps identify data
traffic anomalies.

 Contextual Data: External information, such as weather updates, traffic reports, or city data used
to understand the environment in which the IoT devices are operating.

Types of Linked Data Analytics in IoT

Linked IoT data allows for different, compounding levels of analytical insights:
 Descriptive Analytics: Analyzes historical data to understand past events and patterns.
 Diagnostic Analytics: Analyzes the data to determine why a particular incident occurred, such as a
machine malfunction, by drilling into the connected data.

 Predictive Analytics: Uses ML algorithms and historical data to forecast future trends, such as
predictive maintenance to forecast equipment failure.

 Prescriptive Analytics: Provides recommendations and automatically suggests the best actions to
take based on the analyzed data.

 Data Lake

A Data Lake is a centralized storage system that stores structured,

semi-structured, and unstructured data in its raw format for
flexible analysis. Unlike data warehouses, it follows a “store first,
analyze later” approach, making it ideal for big data, machine
learning, and real-time processing.
It provides scalable, low-cost storage where analysts, engineers, and
data scientists can use their own tools to extract insights.

 Stores all data types and uses schema-on-read (structure applied

during analysis).
 Highly scalable with distributed storage like Hadoop HDFS, AWS
S3, and Azure Data Lake Storage.
 Cost-effective for storing massive volumes of raw data.
 Supports advanced analytics, ML, streaming, and multiple user
teams simultaneously.

Data Lake Architecture

A typical data lake architecture consists of the following layers:
 Real-World Example
Consider an e-commerce company, in the company there are multiple data sources so the
complete workflow given below:
Data Sources:
 Website clickstreams
 Payment transactions
 Inventory databases
 Customer reviews
 Warehouse IoT sensors

 Data lake management

Data lake management is the systematic process of ingesting, storing, organizing, securing, and
governing large volumes of raw structured, semi-structured, and unstructured data. It enables
analytics, AI, and machine learning by maintaining data quality and lineage in a centralized
repository, often organized into raw, cleansed, and curated zones.

Key Aspects of Data Lake Management

 Data Ingestion: Transferring data from various sources (databases, IoT logs, social media) in real-
time or batches.

 Storage & Organization: Utilizing raw (landing), cleansed (validated), and curated (analytics-
ready) zones to organize data.

 Governance & Metadata: Implementing data cataloging and metadata tagging to prevent the data
lake from becoming an unnavigable "data swamp".

 Security & Compliance: Ensuring data protection with encryption, access controls, auditing, and
compliance with regulations.
 Data Quality Management: Cleaning and validating data to ensure reliability for analysis, often
using ELT (Extract, Load, Transform) processes.

 Data Retention
Data retention refers to the practice of storing and maintaining data, typically in a digital format,
for a specific period. This concept involves determining how long different types of data should
be preserved and ensuring compliance with legal, regulatory, or organizational requirements.
Data retention is like keeping information for a particular time. People and businesses do this
for different reasons, such as following the law, ensuring things keep running smoothly, and
studying the data. In the age of digitization, data play a significant role in making decisions and
policies inside any industry. It is also used to analyze customer behavior.
Data Retention is important for following reasons that are as follows:
 Legal compliance: Many industries and educational institutions are legally bound to keep the
records for a certain period. Maintaining and organizing data is essential to ensure an
organization follows the rules and laws that apply to it. This helps the organization avoid
getting into trouble with the law.
 Audit and Accountability: Organizations retaining required data go through the audit
process smoothly. It creates a record of information that can be checked to see if the company
is doing what it's supposed to and following the rules.
 Business and Decision-making: Data plays a significant role in learning from past mistakes
and analyzing future market trends; it leverages an organization's decision-making process.
 Customer Relationship Management: Retained customer data enables organizations to
understand their clients better. This, in turn, facilitates personalized services and helps
identify valuable customers, improving customer satisfaction and loyalty.
 Resilience to Data Loss: Retaining essential operational data ensures businesses can recover
quickly in the face of data loss or system failures. It supports business continuity and
minimizes loss.
 Security incident investigation: In a security breach or cyber attack, retained data can be
crucial in identifying potential sources and helping strengthen the system.
 Compliance with Financial Regulations: Industries must retain financial records for audit
purposes, especially in finance. This ensures transparency and compliance with financial
regulations.
Core Elements of Data Retention
A strong data retention policy hinges on three crucial elements: data classification, retention
periods, and storage considerations. Let's explore what are core elements of data retention:

A. Data Classification

The first step in effective data retention is understanding what information you possess. Data
classification involves categorizing data based on its sensitivity and legal requirements. This is
vital for several reasons:
 Prioritization: Sensitive data, such as financial records or personal information (e.g., Social
Security numbers), requires stricter security measures and may have shorter retention periods
due to privacy regulations.
 Compliance: Classifying data helps identify which information falls under specific
regulations, like HIPAA for healthcare data or GDPR for EU user data. This ensures you
comply with mandated retention periods for certain data types.
 Risk Management: Understanding the sensitivity of data allows you to implement
appropriate security protocols and minimize the risks associated with data breaches or
unauthorized access.

B. Retention Periods

Data retention periods define how long different types of data need to be stored. These periods
can vary significantly depending on:
 Legal Requirements: Many regulations specify minimum retention periods for certain data
types. For example, tax laws might dictate how long financial records must be kept.
 Business Needs: Organizations may need to retain data for internal purposes like business
continuity (disaster recovery) or data analysis. Customer purchase history can be valuable for
understanding buying trends, but may not require long-term storage.
 Data Lifecycle: Consider the "natural lifespan" of the data. For instance, social media posts
might only be relevant for a short period, while historical sales data might be valuable for
long-term analysis.
It's crucial to strike a balance between legal compliance, business needs, and data minimization.
Holding onto unnecessary data for extended periods not only increases storage costs but also
creates a larger security footprint.

C. Storage Considerations

Where you store your data is an essential aspect of data retention. The chosen method needs to be
secure, cost-effective, and scalable to accommodate future growth. Here's a breakdown of
common storage options:
 On-premise Storage: Data is physically stored on servers located within your organization's
infrastructure. This offers greater control but can be expensive and require significant IT
expertise for maintenance and security.
 Cloud Storage: Data is stored on remote servers managed by a cloud service provider. This
offers scalability, flexibility, and often lower upfront costs. However, it's crucial to choose a
reputable provider with robust security measures.
 Hybrid Storage: A combination of on-premise and cloud storage provides a balance between
control, security, and scalability. You can store sensitive data on-site and less critical data in
the cloud.

Data Analytics
Traditional data analytics involves processing historical data in batches to uncover patterns and
insights. This approach, often called batch analytics or historical analytics, processes large volumes
of data that have already been collected and stored. Common use cases include:

Monthly business reporting

Customer behavior analysis
Quarterly sales performance
Historical trend analysis
Marketing campaign evaluation

Real-Time Analytics (Streaming Analytics)

Definition: Real-time analytics processes IoT data immediately as it is generated by sensors and
devices.

Key Characteristics

Data is analyzed instantly (milliseconds to seconds)

Uses stream processing systems

Works on continuous data flow

Enables instant decision-making

Technologies Used

Apache Kafka

Apache Spark Streaming

AWS IoT Analytics

IoT Examples

Smart traffic systems: Adjust signals based on live congestion

Healthcare monitoring: Detect abnormal heart rate instantly

Industrial IoT: Detect machine faults in real time

Advantages

Immediate insights and actions

Reduces risk (e.g., accident prevention)

Supports automation and alerts

Limitations

Requires high processing power

Complex system design

Costly infrastructure

Offline Analytics (Batch Analytics)

Definition: Offline analytics processes IoT data after it has been stored, usually in large
batches.

Key Characteristics
Data is analyzed after hours, days, or weeks

Uses batch processing systems

Focuses on historical data analysis

Supports deep insights and trends

Technologies Used

Hadoop

Apache Hive

Google BigQuery

IoT Examples

Smart agriculture: Analyze seasonal crop data

Energy consumption analysis: Monthly usage trends

Predictive maintenance: Analyze past machine failures

Advantages

Handles large volumes of data efficiently

Cost-effective

Suitable for complex analysis and ML models

Limitations

Delayed insights

Not suitable for time-critical decisions

 Real-time analytics vs. offline analytics

Aspect Offline Data Analytics Real-Time Analytics

Data Processing Batch processing of Continuous processing of incoming

historical data data
Latency Minutes to hours Milliseconds to seconds

Data Volume Large batches (GB/TB) Small, continuous streams

Query Complexity Complex queries with Simpler queries focused on recent

multiple joins and data
aggregations
 Use Cases Business reporting, trend Monitoring, alerting, immediate
analysis, forecasting decision making

Storage Requirements Optimized for large-scale Optimized for quick access to recent
storage and compression data

Query Patterns Ad-hoc analysis, scheduled Continuous queries, real-time

reports dashboards
Data Freshness Historical data Current data (seconds/minutes old)
(hours/days/months old)

Resource Usage Periodic high resource Constant moderate resource usage

usage during batch
processing

Cost Considerations Storage costs dominate Compute costs dominate

Database Features Bulk loading, complex High-speed ingestion, time-series

Needed query optimization optimization

Scaling Challenges Storage capacity, query Write throughput, concurrent

performance operations
Typical Tools Data warehouses, OLAP Stream processing, time-series
systems databases
Business Impact Long-term strategic Immediate operational decisions
decisions
Data Quality Thorough validation and Basic validation, handle incomplete
cleaning data

ThingSpeak dashboard
ThingSpeak is an open-source IoT (Internet of Things) analytics platform service that allows users
to aggregate, visualize, and analyze live data streams in the cloud. It is developed by MathWorks
and is frequently used for prototyping IoT systems, including applications like weather stations,
sensor logging, and tracking, without requiring server setup.
The ThingSpeak dashboard acts as the user interface to manage data, visualize it in real-time, and
trigger actions based on data thresholds.
Key Components of the ThingSpeak Dashboard
1. Channels: The core of ThingSpeak, where data is stored. Each channel can hold data from a single
device or service, supporting up to 8 fields of data (e.g., Temperature, Humidity, Pressure).

2. Private View: A dashboard tab that displays charts and widgets for a user's own data, accessible
only via API keys.

3. Public View: A tab used to share channel visualizations with others, ideal for public projects or
dashboards.
4. Channel Settings: Used to configure channel names, descriptions, and field labels.

5. API Keys: Credentials required to write data to or read data from the channel using HTTP or
MQTT protocols.

6. Visualizations (Widgets): The dashboard allows for instant visualization of data using charts (line,
bar, column, spline, step) and other widgets.

7. Add Widgets Button: A feature that allows users to add custom elements like Gauges, Numeric
Displays, Lamp Indicators, and maps to the dashboard.

Core Functionalities
 Data Aggregation: Collects live data from devices like Arduino, ESP8266, Raspberry Pi, or web
services.

 MATLAB Analytics: Integrates MATLAB to perform data analysis, pre-processing, and

visualization directly on the cloud.

Apps & Integrations:

 React: Triggers actions (e.g., send an email via ThingHTTP) when channel data meets specific
criteria.

 TalkBack: Queues commands for devices to execute.

 TimeControl: Schedules actions at specific times.

 Public/Private Sharing: Users can keep data private or publish it to share with the world.

Getting Started Steps

1. Create Account: Sign up for a MathWorks account on [Link].

2. Create Channel: Create a new channel, name it, and define the data fields.

3. Get API Keys: Note the API Keys to authorize your hardware.

4. Send Data: Program your device (Arduino, ESP32, etc.) to post data to the channel's API URL.

5. Visualize: Customize the Private View with widgets to display data.