Project Name : Project Approach for CloudMonitor Anomaly
Detection
Technologies Used
Python: Primary programming language for data processing, modeling, and visualization.
Libraries:
o Pandas: For data manipulation and analysis.
o NumPy: For numerical computations.
o Matplotlib & Seaborn: For data visualization.
o Scikit-learn (IsolationForest): For anomaly detection using machine learning.
o SHAP: For model explainability and interpreting predictions.
o StandardScaler: For scaling data for better model performance.
o Datetime: For handling time-series data.
2. Step-by-Step Process
Step 1: Import Required Libraries
We imported the necessary Python libraries to handle data processing, visualization, and machine
learning. These include Pandas for data manipulation, Matplotlib for visualization, and Scikit-learn for
building the Isolation Forest model for anomaly detection.
python
Copy code
import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
from [Link] import IsolationForest
from [Link] import StandardScaler
import shap
import datetime
Step 2: Load the Dataset
We load the CloudMonitor data from a CSV file, which contains timestamp and value columns
representing the CPU utilization over time.
python
Copy code
file_path = '/mnt/data/ec2_cpu_utilization_825cc2.csv'
df = pd.read_csv(file_path)
[Link]() # Preview the dataset
Step 3: Preprocess the Data
Data preprocessing involved converting the timestamp into a datetime format, sorting the data by
time, and visualizing the data. The value column, representing CPU utilization, is then scaled using
StandardScaler for better model performance.
python
Copy code
df['timestamp'] = pd.to_datetime(df['timestamp'])
df = df.sort_values('timestamp')
# Plot time-series data
[Link](df['timestamp'], df['value'])
[Link]('CPU Utilization Over Time')
[Link]()
Step 4: Feature Engineering
The value column is standardized, which scales the data to a mean of 0 and a standard deviation of 1.
This makes it easier for the IsolationForest algorithm to detect anomalies.
python
Copy code
scaler = StandardScaler()
df['scaled_value'] = scaler.fit_transform(df[['value']])
X = df[['scaled_value']]
Step 5: Anomaly Detection using Isolation Forest
We used the IsolationForest algorithm to detect anomalies. IsolationForest works well for anomaly
detection in time-series data because it isolates observations by randomly selecting a feature and
splitting the data.
python
Copy code
iso_forest = IsolationForest(contamination=0.01, random_state=42)
df['anomaly'] = iso_forest.fit_predict(X)
df['anomaly'] = df['anomaly'].apply(lambda x: 1 if x == -1 else 0)
Step 6: Visualize Detected Anomalies
We visualized the detected anomalies on top of the time-series data. Anomalies are highlighted as
red dots.
python
Copy code
[Link](df['timestamp'], df['value'])
[Link](df[df['anomaly'] == 1]['timestamp'], df[df['anomaly'] == 1]['value'], color='red')
[Link]('Anomalies in CPU Utilization')
[Link]()
Step 7: Categorize Anomalies
The anomalies were categorized into spikes, drops, and drifts using threshold-based techniques. A
spike represents a sudden high anomaly, a drop is a sharp decline, and a drift is a gradual shift in
values.
python
Copy code
def categorize_anomaly(row):
if row['value'] > df['value'].mean() + 3 * df['value'].std():
return 'spike'
elif row['value'] < df['value'].mean() - 3 * df['value'].std():
return 'drop'
else:
return 'drift'
df['anomaly_type'] = [Link](lambda row: categorize_anomaly(row) if row['anomaly'] == 1 else
[Link], axis=1)
Step 8: Anomaly Scoring
We assigned a severity score to each anomaly based on its deviation from the mean. This score helps
in prioritizing critical anomalies.
python
Copy code
df['anomaly_score'] = [Link](df['value'] - df['value'].mean()) / df['value'].std()
Step 9: SHAP Explainability
We used SHAP (SHapley Additive exPlanations) to explain why the IsolationForest model classified
certain points as anomalies. SHAP values provide insights into the features that contribute to each
prediction.
python
Copy code
explainer = [Link](predict_fn, X)
shap_values = explainer.shap_values(X)
shap.summary_plot(shap_values, X)
Step 10: Visualization Dashboard
We created a comprehensive dashboard that displays the time-series data, anomalies, their
categories, and severity scores.
python
Copy code
[Link](df['timestamp'], df['value'])
[Link](df[df['anomaly'] == 1]['timestamp'], df[df['anomaly'] == 1]['value'], color='red')
[Link]('Anomalies in CPU Utilization Over Time')
[Link]()
Step 11: Generate Insights Report
Finally, an insights report was generated, containing detailed information about the detected
anomalies, their categories, and severity.
python
Copy code
df_anomalies = df[df['anomaly'] == 1][['timestamp', 'value', 'anomaly_type', 'anomaly_score']]
df_anomalies.to_csv('anomalies_report.csv', index=False)
3. Output of the Project
Anomalies Detected: The project successfully detected anomalies in the time-series CPU
utilization data. These anomalies were flagged for further investigation.
Anomaly Categorization: The anomalies were categorized into spikes, drops, and drifts,
which aids in understanding the type of issue causing abnormal CPU utilization.
Anomaly Scores: Each anomaly was scored based on the severity of deviation from the
mean, enabling prioritization of the most critical events.
SHAP Explainability: SHAP values provided interpretability to the model, explaining why
certain points were classified as anomalies.
Visualization: A clear time-series visualization showed both the normal data and the
detected anomalies, giving a comprehensive view of system performance over time.
4. Conclusion
This project successfully implemented an anomaly detection system using IsolationForest on
CloudMonitor data. The combination of model-based detection, anomaly categorization, scoring,
and explainability using SHAP makes the system effective and interpretable. The result is a system
that can automatically detect, classify, and score anomalies in CPU utilization data, which can be
critical for maintaining system performance and diagnosing potential issues.
Thank you.
Code for project :