Abstract

Modern cloud infrastructure generates thousands of metric data points every second across
distributed nodes, services, and regions. Traditional monitoring tools such as Nagios, Grafana, and
Zabbix excel at visualization but fall short in autonomous decision-making — they alert humans, but
cannot reason, remember, or recommend. This project presents an Agentic Reasoning Framework
with Self-Evolving Memory, a multi-agent AIOps platform that autonomously collects real-time
infrastructure telemetry, detects anomalies using rule-based intelligence, retains institutional
memory of recurring patterns, dispatches professional alerts with cooldown control, evaluates its
own detection accuracy, and responds to natural language queries grounded in live infrastructure
context. The system is composed of nine cooperating autonomous agents — Collector, Detection,
Memory, Reasoning, Automation, Cost, Evaluation, Benchmark, and Assistant — each with defined
responsibilities and inter-agent communication. Unlike conventional dashboards, this framework
does not merely display data; it understands it, remembers it, and acts on it.

Introduction

The scale of cloud infrastructure has grown beyond the capacity of human operators to monitor
manually. A mid-sized technology company running on AWS, Google Cloud, or Azure may operate
hundreds of virtual machines, thousands of microservices, and millions of metric events per day.
When a CPU spike occurs at 3:47 AM, a memory leak grows silently over six hours, or a network
anomaly signals a potential intrusion — human operators are either asleep, overwhelmed, or looking
at the wrong dashboard.

The industry response has been AIOps — Artificial Intelligence for IT Operations — a discipline that
applies machine learning, automation, and intelligent reasoning to infrastructure monitoring. Tools
like Dynatrace, Datadog, PagerDuty, and Splunk have commercialized this space at enterprise scale,
commanding licensing costs of $50,000 to $500,000 per year.

This project implements the core architectural principles of enterprise AIOps from the ground up:

 Autonomous detection without human-defined alert rules for every scenario

 Self-evolving memory that tracks which incidents recur and how frequently

 Intelligent alerting that suppresses noise and dispatches only meaningful notifications

 Natural language interface that allows any team member — not just engineers — to query
infrastructure health

 Self-evaluation that continuously measures the system's own accuracy

The platform is built on a real hardware telemetry pipeline using Python psutil, a [Link] multi-agent
backend, SQLite for persistent storage, and OpenRouter-hosted large language models for reasoning
and conversation.
Problem Statement

The Real-World Problem

On July 2, 2021, Fastly — a major cloud CDN provider — suffered a global outage that took down
Amazon, Reddit, The Guardian, BBC, and the New York Times simultaneously for approximately one
hour. The root cause was a single software bug triggered by one customer changing a configuration
setting. The total estimated economic impact exceeded $100 million in lost revenue across affected
businesses.

On October 4, 2021, Facebook (now Meta) experienced a six-hour complete outage caused by a
misconfigured BGP route update. Approximately 3.5 billion users lost access to Facebook, Instagram,
and WhatsApp. Facebook's own internal monitoring tools went offline with the platform, leaving
engineers unable to diagnose the problem remotely. Engineers had to physically drive to the data
center to fix it manually.

These are not isolated incidents. According to the Uptime Institute's 2023 Global Outage Report:

 25% of enterprises suffered a significant outage in the past three years

 The average cost of a critical outage is $300,000 per hour

 85% of outages were caused by human error or process failures

 Only 40% of organizations have automated incident detection

The Specific Gaps This Project Addresses

Gap 1 — Reactive monitoring, not proactive intelligence Current tools alert after a threshold is
crossed. By then the damage is already happening. A system that remembers "CPU spikes every night
at 2 AM on this node" can predict and flag it before it becomes critical.

Gap 2 — Alert fatigue A 2022 survey by Opsgenie found that 70% of on-call engineers regularly
ignore alerts because too many are false positives or duplicates. Engineers stop trusting the system.
This project's cooldown system and severity classification directly address this.

Gap 3 — Siloed knowledge When a senior engineer leaves, institutional knowledge about recurring
incidents leaves with them. This project's Memory Agent solves this — the system itself retains the
history: "this disk warning has occurred 47 times, first seen 3 weeks ago, always resolves within 20
minutes."

Gap 4 — Non-technical stakeholders cannot access infrastructure data A product manager asking "is
our server stable right now?" currently needs to find an engineer, interrupt their work, and wait for
an answer. The AI Assistant in this project lets anyone ask that question in plain English and get a
grounded, accurate answer in seconds.

Gap 5 — No self-measurement Most monitoring tools do not measure their own accuracy. If the
detection engine starts missing incidents, nobody knows until an outage happens. This project's
Evaluation Agent continuously tests detection accuracy against known test cases and exposes it on
the dashboard.
Problem Statement (Formal)

Existing infrastructure monitoring solutions lack autonomous reasoning, persistent incident memory,
and self-evaluation capabilities, resulting in reactive incident response, alert fatigue, loss of
institutional knowledge, and exclusion of non-technical stakeholders from infrastructure visibility.
This project addresses these gaps through a nine-agent AIOps framework that autonomously detects,
remembers, evaluates, and communicates infrastructure health using real-time telemetry and large
language model reasoning.

Real-World Relevance

This Project Feature Real-World Equivalent

Multi-agent architecture Dynatrace Davis AI Engine

Self-evolving memory PagerDuty's Alert Grouping + History

Intent-classified AI assistant Datadog's Bits AI

Alert cooldown + suppression OpsGenie Notification Rules

Detection accuracy evaluation Splunk MLTK Model Evaluation

Cost optimization agent AWS Cost Explorer + Trusted Advisor

9-agent pipeline registry ServiceNow IT Operations Management

Python psutil collector Datadog Agent, Prometheus Node Exporter

Who This Is Built For

Primary audience — Engineering teams at startups and mid-size companies who cannot afford
$200K/year Datadog licenses but need real AIOps capabilities.

Secondary audience — DevOps engineers who spend 30% of their time responding to alerts
manually and need an autonomous first responder that triages issues before human escalation.

Tertiary audience — Engineering managers and CTOs who need infrastructure visibility in plain
English without learning Grafana query language or reading raw logs.

Academic audience — Demonstrates applied implementation of multi-agent systems, LLM

grounding, self-evaluation loops, and memory-augmented reasoning in a production-adjacent
context.
PROJECT DASHBOARD
Subtitle: Autonomous Infrastructure Monitoring Dashboard — clarifies what the system does
operationally: monitors cloud infrastructure without human intervention.

☁ CLOUD SIMULATION MODE badge — tells you this is running on simulated cloud nodes (not real
AWS/GCP). Important for demos and development.

Last updated / ● LIVE — confirms the backend collector is running and pushing real-time data. If this
stops updating, the collector Python script has died.

1. Cloud Instance Overview

Why included: First thing any ops engineer needs — a quick health snapshot of the whole system.

 Status (Healthy/Warning/Critical) — computed from latest CPU/Memory/Disk thresholds.

Tells you at a glance if action is needed.

 Uptime — how long the system has been running without a restart. High uptime = stable.

 API — confirms the backend Express server is reachable. If this is down, nothing works.

 Last Sync — when the Python collector last pushed metrics. If stale → collector is offline.

 Total Incidents / Alerts — cumulative count since startup. Gives context on how busy this
system has been.

2. Cloud Node Status

Why included: Real cloud environments have multiple nodes/VMs. This simulates a 4-node cloud
cluster.

 Cloud-Instance-01 — primary node, real metrics from your machine via psutil

 Cloud-VM-02 — simulated node with high CPU to show what a stressed compute node looks
like

 Cloud-DB-01 — simulated memory-heavy node (databases consume RAM)

 Cloud-Gateway-01 — simulated network spike node (gateways handle traffic bursts)

Why important: In real AIOps, you need per-node visibility. An issue on one node shouldn't be
confused with a cluster-wide outage.

3. Live Cloud Metrics — the 4 big cards

Why included: These are the 4 pillars of infrastructure health. Polled every 2 seconds from real psutil
data.

 Compute Utilization (CPU %) — how hard the processor is working. >80% = performance
risk. Spikes indicate runaway processes.

 Memory Pressure (%) — RAM usage. High memory = swap risk, slow apps, OOM crashes.
  Storage Utilization (%) — disk fullness. At 100% the system stops writing logs, crashes DBs,
fails deployments.

 Network Throughput (MB/s) — data in/out. Spikes indicate DDoS, backup jobs, or data
exfiltration.

Color coding: Green = healthy, Yellow = warning, Red = critical — so you can read status without
thinking.

4. Cloud Resource Trends (Charts)

Why included: Metrics at a single moment lie. Trends tell the truth.

 Shows history for the same 4 metrics (CPU, Memory, Disk, Network) over last 10/20/50/100
readings

 A CPU spike at 3am that's back to normal looks fine now but the chart shows you it
happened

 History range selector — lets you zoom out to see patterns over more data points

Why important: Incident detection needs baselines. You can't know if 70% CPU is abnormal without
seeing what's "normal" for this system.

5. Top Resource-Consuming Services

Why included: Knowing CPU is at 85% is not enough — you need to know which process is causing it.

 Lists top processes by CPU and memory usage (from psutil)

 In a real incident, this is how you identify the culprit (e.g., [Link] consuming 60% CPU)

6. Recent Cloud Telemetry

Why included: Raw metrics table — every data point the collector has pushed.

 Timestamp, CPU, Memory, Disk, Network per row

 Used for audit, debugging, and feeding the incident detection engine

 Engineers use this to verify the collector is working and data quality is clean

7. Recent Infrastructure Incidents

Why included: The incident detection engine ([Link]) analyzes every metric push and
creates incidents when thresholds are crossed.

 Type — what kind of incident (CPU Spike, Memory High, Disk Warning, Network Anomaly)

 Severity — Critical / High / Warning

  Status — Active or Resolved

 Timestamp — when it was detected

Why important: This is the core output of the AIOps system. Every alert, every memory record, every
evaluation result traces back to these incidents.

8. Autonomous Infrastructure Alerts

Why included: Incidents detected → alerts dispatched automatically. No human needed.

 Shows which alerts were sent (Email / WhatsApp)

 Cooldown system — prevents alert spam; same incident type suppressed for 15 minutes

 IST timestamps — localized for the operator's timezone (India)

Why important: An AIOps system that detects but doesn't notify is useless. This closes the loop:
detect → analyze → alert.

9. Infrastructure Memory Context

Why included: This is the "self-evolving memory" part of the title.

 Tracks recurring incident patterns across time

 If CPU Spike has occurred 47 times, the system remembers it and weights it differently than a
one-time event

 Fields: incident_type, occurrence_count, first_seen, last_seen

Why important: A system that forgets is reactive. A system that remembers is proactive. Memory
enables the AI assistant to say "this has happened 47 times before" instead of treating every incident
as new.

10. Cloud Monitoring Evaluation (Evaluation Agent)

Why included: The system evaluates itself — measures how well it's detecting and responding.

 Health Score (0-100) — weighted penalty from critical/high incidents. Lower score = more
problems detected

 CRITICAL / HIGH / WARNING threat grid — breakdown of incident severity distribution

 Detection Accuracy — what % of real incidents the engine correctly identified (tested
against [Link])

 Memory Accuracy — how well the memory system tracks recurring patterns (tracked
occurrences / total incidents)

 Alert Success Rate — what % of alerts were actually dispatched vs errored (excludes
suppressed — suppression is correct behavior)
  Agent Intelligence Feed — AI-generated insights about the system state

Why important: Without evaluation, you're flying blind. This tells you if the AI agents are actually
working correctly.

11. Cloud Performance Benchmark

Why included: Benchmarks the whole pipeline's performance end-to-end.

 Overall Score — weighted formula: Detection Accuracy×0.4 + Alert Success×0.2 + Memory

Accuracy×0.2 + LLM Latency Score×0.2

 Compute Utilization / RAM — real system resource usage during benchmark (from psutil,
not [Link] which returns 0 on Windows)

 LLM Latency — how fast the AI model responds (ms). Affects user experience in the assistant

 Token Usage — how many tokens the LLM consumed (cost tracking)

 Loop Time — how long one full benchmark cycle takes

 Precision / Recall / F1 — standard ML metrics measuring detection quality

Why important: Tells you if the system is getting faster or slower, if the AI model is performing, and
justifies the infrastructure cost.

12. Cloud Cost Optimization

Why included: Cloud costs money. Unoptimized infra bleeds budget.

 Estimated Monthly Spend — projected cost based on current resource consumption

 Potential Savings % — how much you could save by right-sizing

 High Cost Node — which node is most expensive

 Idle Nodes — nodes doing nothing but costing money

 Detected Issues with recommendations — e.g. "over-provisioned memory → downsize

instance"

Why important: In real cloud (AWS/GCP/Azure), unmonitored spend can cause massive surprise bills.
Cost optimization is a first-class concern in AIOps.

13. Multi-Agent System Registry

Why included: The entire system is built as 9 cooperating autonomous agents. This panel shows the
full agent pipeline.

Each agent listed:

Agent Role

Collector Agent Pulls raw metrics from psutil every 2s

Detection Agent Analyzes metrics, fires incidents when thresholds crossed

Memory Agent Tracks recurring patterns, updates occurrence counts

Reasoning Agent LLM-powered analysis of incidents

Automation
Dispatches alerts (email/WhatsApp) with cooldown
Agent

Cost Agent Estimates spend, finds savings

Evaluation Agent Measures detection accuracy and system health

Benchmark Agent Runs full performance benchmark

Assistant Agent Answers natural language questions using live infra context

Why important: Shows the architecture. Each agent has defined inputs, outputs, capabilities, and
pipeline order. This is what makes it an "Agentic Framework" — not one monolithic script, but 9
specialized autonomous agents working together.

Overall System Purpose

This is a full-stack AIOps (AI for IT Operations) platform that:

1. Collects real infrastructure data (Python psutil)

2. Detects anomalies automatically (rule-based engine)

3. Remembers patterns over time (memory agent)

4. Sends professional alerts instantly (automation agent)

5. Evaluates its own accuracy (evaluation agent)

6. Optimizes costs proactively (cost agent)

7. Benchmarks performance continuously (benchmark agent)

8. Answers natural language questions about the infrastructure (AI assistant)

Every section exists because each one solves a specific gap in traditional monitoring — which is
reactive, manual, and forgetful. This system is proactive, autonomous, and memory-driven.

how the backend is working..where the datas are stored

How the Backend Works — Complete Flow

Tech Stack
  Runtime: [Link] + Express (port 3001)

 Database: SQLite (single file, no server needed)

 AI: OpenRouter API (free LLM models)

 Alerts: Nodemailer (email) + Twilio (WhatsApp)

 Data Collector: Python + psutil (separate process)

Where Data Is Stored

Everything is stored in a single SQLite file:

d:\rockit\PROJECT_9\backend\db\[Link]

SQLite = a .db file on disk. No separate database server. No MySQL, no MongoDB. Just one file that
the backend reads/writes directly.

Tables inside that file:

Table What it stores

metrics Every CPU/Memory/Disk/Network reading

incidents Every detected anomaly (type, severity, status)

memory_record
Recurring incident patterns + occurrence counts
s

benchmark_logs Performance benchmark results

processes Top CPU/RAM consuming processes

The Full Data Pipeline

Python Collector

↓ POST /api/metrics (every 2 seconds)

Express Backend

↓ saves to SQLite → metrics table

↓ runs incidentEngine → saves to incidents table

↓ runs memoryAgent → updates memory_records table

↓ runs automationAgent → sends email/WhatsApp

Frontend polls every 2-15s via REST API

↓
React Dashboard renders live data

Step 1 — Python Collector (collector/[Link])

Runs as a separate process on your machine. Every 2 seconds it:

 Reads real system stats using psutil (CPU, RAM, Disk, Network, Processes)

 POSTs JSON to [Link]

Your Machine Hardware

→ psutil reads CPU/RAM/Disk/Network

→ POST { cpu_usage: 14.1, memory_usage: 63.2, ... }

→ Express receives it

Step 2 — Express Receives Metric (POST /api/metrics)

Located in backend/src/routes/[Link]

When data arrives:

1. Saves row to metrics table in SQLite

2. Calls [Link]() — checks thresholds:

o CPU > 80% → High, CPU > 90% → Critical

o Memory > 85% → High, Memory > 90% → Critical

o Disk > 80% → High, Disk > 90% → Critical

3. If incident detected → saves to incidents table

4. Calls memoryAgent → checks if this incident type was seen before:

o First time → creates new row in memory_records

o Seen before → increments occurrence_count, updates last_seen

5. Calls automationAgent → checks cooldown (15 min):

o Not in cooldown → sends email + WhatsApp alert

o In cooldown → suppressed (not a failure, intentional)

Step 3 — SQLite Tables in Detail

metrics table — raw telemetry

id | cpu_usage | memory_usage | disk_usage | network_usage | timestamp

1 | 14.1 | 63.2 | 85.9 | 0.04 | 2025-03-26 14:52:01

2 | 15.3 | 63.8 | 85.9 | 0.06 | 2025-03-26 14:52:03

incidents table — detected anomalies

id | type | severity | status | message | timestamp

1 | CPU Spike | High | Active | CPU usage at 85.2% | 2025-03-26 14:52:01

2 | Memory High | Critical | Active | Memory at 91.4% | 2025-03-26 14:53:10

memory_records table — pattern memory

id | incident_type | source_metric | occurrence_count | first_seen | last_seen

1 | CPU Spike | cpu_usage | 47 | 2025-03-25 | 2025-03-26

2 | Memory High | memory_usage | 12 | 2025-03-25 | 2025-03-26

benchmark_logs table — benchmark history

id | score | latency | cpu | ram | errors | timestamp

1 | 78 | 1842 | 14.1 | 63.2 | 0 | 2025-03-26 14:55:00

Step 4 — Frontend Polling

The React frontend never waits — it polls the backend constantly:

Hook Endpoint Interval

useLatestMetric GET /api/metrics/latest every 2s

useMetrics GET /api/metrics?limit=20 every 2s

useIncidents GET /api/incidents every 5s

useMemory GET /api/memory every 10s

useEvaluation GET /api/evaluation every 15s

useCostOptimization GET /api/cost on-demand

useSystemBenchmark GET /api/benchmark/run on-demand

useAgentRegistry GET /api/agents once + retry

Step 5 — AI Assistant Flow

When you type a question:

User types question

→ POST /api/assistant/query

→ classifyIntent() → is it greeting / general / infra?

 → buildContext() → reads latest from ALL tables simultaneously

→ formatContext() → builds a text summary of live data

→ buildPrompt() → combines system prompt + question + context

→ askLLM() → POST to OpenRouter API → free LLM model responds

→ stripMarkdown() → cleans ** bold ** formatting

→ returns { answer, intent, sources, model_used }

→ frontend renders it

Where Each Config Lives

What File

Database file path backend/db/[Link]

Schema (table
backend/database/[Link]
definitions)

API routes backend/src/routes/

Agent logic backend/services/

OpenRouter API key backend/.env → OPENROUTER_API_KEY

backend/.env → EMAIL_USER, EMAIL_PAS

Email credentials
S

Twilio credentials backend/.env → TWILIO_*

Alert cooldown (15 min) backend/config/[Link]

Incident thresholds backend/services/[Link]

LLM model selection backend/services/[Link]

Summary in One Line

The Python collector reads your real hardware every 2 seconds → pushes to Express → Express
detects incidents, updates memory, sends alerts, and saves everything to a single SQLite file → React
frontend polls that data and renders it live → AI assistant queries the same SQLite data to answer
your questions.

now rate this project

Project Rating
Overall: 8.2 / 10

Category Breakdown

Category Score Reason

Clean multi-agent design, clear separation of concerns, each agent has

Architecture 9/10
single responsibility

Real-time Pipeline 8.5/10 2s polling, live SQLite writes, psutil integration — genuinely real-time

Intent classification, context-grounded LLM answers, retry logic,

AI Integration 8/10
markdown stripping

Professional dark theme, evaluation agent panel, live charts, responsive

Frontend UI 8/10
layout

Alerting System 7.5/10 Email + WhatsApp, IST timezone, cooldown logic, HTML email — solid

Self-Evaluation 9/10 System evaluates its own accuracy — rare feature, very impressive

Pattern tracking, occurrence counting, first/last seen — genuine self-

Memory System 8.5/10
learning

Cost Optimization 7/10 Rule-based estimates, not real cloud API billing data

Documentation 7/10 Code comments are good, no external README/API docs

Production
6/10 SQLite limits scale, no auth, no HTTPS, simulated nodes
Readiness

What Makes It Strong

 Self-evaluation loop — the system measures its own detection accuracy against test cases.
Most student/portfolio projects never do this

 Memory agent — incidents aren't forgotten. Pattern recurrence tracking is a real AIOps
concept used in tools like PagerDuty and Dynatrace

 9-agent architecture — not a monolith. Each agent is independently responsible and

communicates through defined inputs/outputs

 Intent classification — the AI assistant understands context (greeting vs general vs infra)

before answering. Shows architectural thinking

 Alert cooldown — the bug where markAlertSent() ran after the send (causing spam) was
properly fixed by moving it before. Shows attention to real-world edge cases

What Holds It Back

  SQLite — works great for one machine, but cannot scale to multiple collectors or high write
frequency in production

 Simulated nodes — Cloud-VM-02, Cloud-DB-01, Cloud-Gateway-01 are hardcoded fake data.

Real value needs real nodes

 No authentication — anyone on the network can hit /api/metrics and inject fake data

 Free LLM models — response quality and availability are inconsistent. Production needs a
paid model

 [Link]() on Windows — needed a workaround (read CPU from DB instead). Shows

platform dependency

 No persistent benchmark history chart — benchmark runs but trends aren't visualized over
time

Compared To What?

Context Rating

Student/college project 10/10 — exceptional

Portfolio/job application 9/10 — stands out clearly

Indie/side project 8/10 — very solid

Production enterprise tool 5/10 — needs auth, real DB, real nodes

What Would Push It To 10/10

1. Replace SQLite with PostgreSQL

2. Add JWT authentication on all API routes

3. Connect real cloud APIs (AWS CloudWatch / GCP Monitoring)

4. Replace free LLMs with GPT-4o or Claude for consistent quality

5. Add benchmark history chart (score over time)

6. Write a proper README with architecture diagram

Bottom line: This is a genuinely impressive system with real AIOps concepts implemented correctly
— memory, evaluation, multi-agent orchestration, LLM grounding. It's not a tutorial clone. It
demonstrates original architectural thinking.