9/30/24, 6:05 PM model

Skillbanc App Model Integration Report

Satya Pratheek TATA, [Link] Inc.
Data Science Team
Overview
The aim of this document is to outline the process of integrating data science models
into the Skillbanc app, focusing on user retention, frustration, and session duration
prediction. The models are created to track user behavior and enhance math learning
experiences by providing personalized feedback based on user engagement data.
Purpose of the Models
1. User Retention Prediction: Determines if a user is likely to return to the app
within 2 days, which helps Skillbanc maintain and increase user engagement.
2. Frustration Detection: Identifies when a user may be getting frustrated,
allowing the app to adjust the difficulty or provide additional support to prevent
dropout.
3. Session Duration Prediction: Predicts how long a user’s session will last,
helping optimize content delivery and engagement.

Dataset Description
File Used: event_log4.csv
Number of Rows Used: 2056
Session Threshold: 15 minutes (used to define when a new session starts)
The data captures user events, such as the actions they take, timestamps, and
results of those actions (whether correct or incorrect).

Data Preprocessing and Cleaning

Key Cleaning Steps:
1. Session Identification: New sessions are defined based on a threshold of 15
minutes between actions.
2. Feature Extraction:
Session duration: Time spent by a user in a session.
Correct and Incorrect Streaks: Tracks consecutive correct and incorrect
answers to detect frustration.
[Link] 1/10
9/30/24, 6:05 PM model

Inactivity Periods: Measures gaps between user interactions within a

session.
App Usage Frequency: Average time between user sessions, which helps in
predicting retention.
Why It Matters: Clean, well-structured data enables more accurate predictions,
ensuring that the app can make reliable adjustments based on user behavior.

In [ ]: import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, GridSearchCV
from [Link] import StandardScaler
from [Link] import RandomForestRegressor, RandomForestClassifie
from [Link] import mean_squared_error, accuracy_score
import xgboost as xgb
import [Link] as plt
import seaborn as sns
import joblib
from datetime import timedelta

def load_and_clean_data(file_path, nrows=None, session_threshold=15):

"""
Load and clean the data, returning a cleaned dataframe.
"""
# Load the data
try:
data = pd.read_csv(file_path, nrows=nrows)
data = [Link]()
except Exception as e:
raise ValueError(f"Error loading data from {file_path}: {e}")

# Extract relevant columns

df = [Link]()
df['device_id'] = data['Event Log'].[Link]('-').str[0]
df['created_on'] = pd.to_datetime(data['Created On'], errors='coerce'
df['event_type'] = data['Event Type']
df['action'] = data['Action']

# Filter rows with valid datetimes

df = [Link](subset=['created_on'])

# Sort by device and timestamp

df = df.sort_values(by=['device_id', 'created_on'])

# Calculate time difference between consecutive actions for each devi

df['time_diff'] = [Link]('device_id')['created_on'].diff()
df['time_diff'] = df['time_diff'].apply(lambda x: x.total_seconds() i

# Define new session if time difference exceeds the session threshold

df['new_session'] = df['time_diff'] > session_threshold * 60

# Generate session ID based on new sessions

df['session_id'] = [Link]('device_id')['new_session'].cumsum() +

# Calculate inactivity period (same as time_diff for now)

df['inactivity_period'] = df['time_diff']

[Link] 2/10
9/30/24, 6:05 PM model

# Add binary column to indicate whether the action was correct

df['is_correct'] = (df['event_type'] == 'correct').astype(int)

# Add total actions per session

df['total_actions'] = [Link]('session_id')['action'].transform('c

# Calculate average time between actions per session

df['average_time_between_actions'] = [Link]('session_id')['time_d

# Add cumulative count of incorrect actions (streak drops)

df['streak_drops'] = [Link]('session_id')['is_correct'].transform

# Add correct streaks (reset when incorrect)

df['correct_streak'] = [Link]('session_id')['is_correct'].transfo

# Count of correct actions per session

df['correct_count'] = [Link]('session_id')['is_correct'].transfor

# Count of incorrect actions per session

df['incorrect_count'] = [Link]('session_id')['is_correct'].transf

# Ratio of incorrect to correct actions

df['incorrect_to_correct_ratio'] = df['incorrect_count'] / df['correc

# Track session start and calculate time since last session

df['session_start_time'] = [Link]('session_id')['created_on'].tra
df['previous_session_end'] = [Link]('device_id')['session_start_t
df['time_since_last_session'] = (df['session_start_time'] - df['previ
df['time_since_last_session'] = df['time_since_last_session'].fillna(

# Calculate app usage frequency (mean time between sessions for each
df['app_usage_frequency'] = [Link]('device_id')['time_since_last_

# Calculate session duration in minutes

df['session_duration'] = ([Link]('session_id')['created_on'].tran

# Aggregate unique actions per session

df['event_context'] = [Link]('session_id')['action'].transform(la

# Extract hour of the day and categorize into time of day

df['hour_of_day'] = df['created_on'].[Link]
df['time_of_day'] = [Link](df['hour_of_day'], bins=[0, 6, 12, 18, 24]

# Ensure one-hot encoding is consistent

df = pd.get_dummies(df, columns=['time_of_day'], prefix='tod')

# Add session count and average session duration per device

df['previous_session_count'] = [Link]('device_id')['session_id'].
df['average_session_duration'] = [Link]('device_id')['session_dur

# Track total device usage duration

df['device_usage_duration'] = [Link]('device_id')['session_durati

# Extract the day of the week

df['day_of_week'] = df['created_on'].[Link]

return df

[Link] 3/10
9/30/24, 6:05 PM model

def train_models(df):
# Example for Session Length Prediction
X_reg = df[['total_actions', 'average_time_between_actions', 'streak_
y_reg = df['session_duration']

# Example for User Frustration Classification

X_clf_frustration = df[['total_actions', 'average_time_between_action
y_clf_frustration = (df['streak_drops'] > 2).astype(int) # Binary ta

# Example for User Retention Prediction

X_clf_retention = df[['previous_session_count', 'average_session_dura
y_clf_retention = (df['time_since_last_session'] < 2).astype(int) #

# Split data into training, validation, and test sets (80/10/10 split
X_train_reg, X_temp_reg, y_train_reg, y_temp_reg = train_test_split(X
X_val_reg, X_test_reg, y_val_reg, y_test_reg = train_test_split(X_tem

X_train_clf_frustration, X_temp_clf_frustration, y_train_clf_frustrat

X_val_clf_frustration, X_test_clf_frustration, y_val_clf_frustration,

X_train_clf_retention, X_temp_clf_retention, y_train_clf_retention, y

X_val_clf_retention, X_test_clf_retention, y_val_clf_retention, y_tes

# Scaling the data

reg_scaler = StandardScaler()
X_train_reg_scaled = reg_scaler.fit_transform(X_train_reg)
X_val_reg_scaled = reg_scaler.transform(X_val_reg)
X_test_reg_scaled = reg_scaler.transform(X_test_reg)
print('train shape {}, test shape {}'.format(X_train_reg_scaled.shape

frust_scaler = StandardScaler()
X_train_clf_frustration_scaled = frust_scaler.fit_transform(X_train_c
X_val_clf_frustration_scaled = frust_scaler.transform(X_val_clf_frust
X_test_clf_frustration_scaled = frust_scaler.transform(X_test_clf_fru
print('train shape {}, test shape {}'.format(X_train_clf_frustration_

ret_scaler = StandardScaler()
X_train_clf_retention_scaled = ret_scaler.fit_transform(X_train_clf_r
X_val_clf_retention_scaled = ret_scaler.transform(X_val_clf_retention
X_test_clf_retention_scaled = ret_scaler.transform(X_test_clf_retenti
print('train shape {}, test shape {}'.format(X_train_clf_retention_sc

# Model training with RandomForest and XGBoost follows as in your ori

# Return trained models and scalers for further use.
return (X_reg, X_train_reg_scaled, X_val_reg_scaled, X_test_reg_scale

# Save the cleaned data and models

def save_models_and_data(model_clf_frustration, model_regression, model_x
[Link](model_clf_frustration, 'random_forest_clf_model.pkl')
[Link](model_regression, 'random_forest_reg_model.pkl')
[Link](model_xgb, 'xgboost_clf_model.pkl')
df.to_csv('cleaned_event_data.csv', index=False)

# Call the functions

df_cleaned = load_and_clean_data('event_log4.csv', nrows=2056, session_th
reg, clf_frustration, clf_retention = train_models(df_cleaned)

[Link] 4/10
9/30/24, 6:05 PM model

X_reg, X_train_reg_scaled, X_val_reg_scaled, X_test_reg_scaled, y_reg, y_

X_clf_frustration, X_train_clf_frustration_scaled, X_val_clf_frustration_
X_clf_retention, X_train_clf_retention_scaled, X_val_clf_retention_scaled
print('train shape {}, test shape {}'.format(X_train_clf_retention_scaled
print('train shape {}, test shape {}'.format(X_train_clf_frustration_scal
print('train shape {}, test shape {}'.format(X_train_reg_scaled.shape, X_

Modeling Approach
1. Session Duration Prediction
Goal: Predict how long a user will stay engaged in a session.
Model: Random Forest Regressor
Features: Number of actions, time between actions, streak drops, usage
frequency, and more.
Target Variable: session_duration (in minutes)

In [ ]: from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

from [Link] import RandomForestRegressor
from [Link] import mean_squared_error

# Random Forest Regressor

rf_reg = RandomForestRegressor(random_state=42)

# Hyperparameter grid
param_grid_rf = {
'n_estimators': [100, 200, 300],
'max_depth': [10, 20, 30, None],
'min_samples_split': [2, 5, 10],
'min_samples_leaf': [1, 2, 4],
'bootstrap': [True, False]
}

# Grid search
grid_search_rf = GridSearchCV(estimator=rf_reg, param_grid=param_grid_rf,
grid_search_rf.fit(X_train_reg_scaled, y_train_reg)

# Best parameters and performance

print("Best parameters found: ", grid_search_rf.best_params_)
y_pred_rf = grid_search_rf.predict(X_val_reg_scaled)
mse_rf = mean_squared_error(y_val_reg, y_pred_rf)
print("Validation MSE: ", mse_rf)

2. User Frustration Detection

Goal: Detect when users become frustrated to adjust learning strategies.
Model: Random Forest Classifier
Features: Number of actions, average time between actions, streak drops, and
more.
[Link] 5/10
9/30/24, 6:05 PM model

Target Variable: Binary indicator ( streak_drops > 2 ) signaling frustration.

In [ ]: from [Link] import RandomForestClassifier

from [Link] import accuracy_score

# Random Forest Classifier

rf_clf = RandomForestClassifier(random_state=42)

# Hyperparameter grid
param_grid_rf_clf = {
'n_estimators': [100, 200, 300],
'max_depth': [10, 20, 30, None],
'min_samples_split': [2, 5, 10],
'min_samples_leaf': [1, 2, 4],
'bootstrap': [True, False]
}

# Grid search
grid_search_rf_clf = GridSearchCV(estimator=rf_clf, param_grid=param_grid
grid_search_rf_clf.fit(X_train_clf_frustration_scaled, y_train_clf_frustr

# Best parameters and performance

print("Best parameters found: ", grid_search_rf_clf.best_params_)
y_pred_rf_clf = grid_search_rf_clf.predict(X_val_clf_frustration_scaled)
accuracy_rf_clf = accuracy_score(y_val_clf_frustration, y_pred_rf_clf)
print("Validation Accuracy: ", accuracy_rf_clf)

3. User Retention Prediction

Goal: Predict if a user will return to the app within 2 days.
Model: XGBoost Classifier
Features: Session counts, average session duration, time since the last session,
and more.
Target Variable: Binary indicator of whether the user returned within 2 days.

In [ ]: import xgboost as xgb

from [Link] import accuracy_score

# XGBoost Classifier
xgb_clf = [Link](random_state=42)

# Hyperparameter grid
param_grid_xgb = {
'learning_rate': [0.01, 0.1],
'n_estimators': [100],
'max_depth': [3, 5],
'colsample_bytree': [0.8],
'subsample': [0.8]
}

# Grid search
grid_search_xgb = GridSearchCV(estimator=xgb_clf, param_grid=param_grid_x

[Link] 6/10
9/30/24, 6:05 PM model

grid_search_xgb.fit(X_train_clf_retention_scaled, y_train_clf_retention)

# Best parameters and performance

print("Best parameters found: ", grid_search_xgb.best_params_)
y_pred_xgb = grid_search_xgb.predict(X_val_clf_retention_scaled)
accuracy_xgb = accuracy_score(y_val_clf_retention, y_pred_xgb)
print("Validation Accuracy: ", accuracy_xgb)

Model Evaluation
Session Duration:
Best MSE: 1581635.2032971184
Recommendations: Fine-tune the model to further reduce errors by
adjusting min_samples_split or increasing n_estimators .
Frustration Detection:
Best Accuracy: 1.0
Insights: Early frustration detection helps retain users by adjusting difficulty
or offering tutorials, crucial for long-term engagement.
User Retention:
Best Accuracy: 0.9782608695652174
Insights: Retention is a key metric for growth. Improving this model’s
accuracy will help Skillbanc effectively re-engage users who may otherwise
abandon the app.

In [5]: # Evaluate Random Forest Regressor on the test set

y_test_pred_rf = grid_search_rf.predict(X_test_reg_scaled)
test_mse_rf = mean_squared_error(y_test_reg, y_test_pred_rf)
print("Test MSE (Random Forest Regressor): ", test_mse_rf)

# Ensure X_test_clf_frustration_scaled is 2D (even if it's a single sampl

if isinstance(X_test_clf_frustration_scaled, [Link]):
X_test_clf_frustration_scaled = X_test_clf_frustration_scaled.values.
elif X_test_clf_frustration_scaled.ndim == 1:
X_test_clf_frustration_scaled = X_test_clf_frustration_scaled.reshape

# Evaluate Random Forest Classifier on the test set

y_test_pred_rf_clf = grid_search_rf_clf.predict(X_test_clf_frustration_sc
test_accuracy_rf_clf = accuracy_score(y_test_clf_frustration, y_test_pred
print("Test Accuracy (Random Forest Classifier): ", test_accuracy_rf_clf)

# Ensure X_test_clf_retention_scaled is 2D (even if it's a single sample)

if isinstance(X_test_clf_retention_scaled, [Link]):
X_test_clf_retention_scaled = X_test_clf_retention_scaled.[Link]
elif X_test_clf_retention_scaled.ndim == 1:
X_test_clf_retention_scaled = X_test_clf_retention_scaled.reshape(1,

# Evaluate XGBoost Classifier on the test set

y_test_pred_xgb = grid_search_xgb.predict(X_test_clf_retention_scaled)

[Link] 7/10
9/30/24, 6:05 PM model

test_accuracy_xgb = accuracy_score(y_test_clf_retention, y_test_pred_xgb)

print("Test Accuracy (XGBoost Classifier): ", test_accuracy_xgb)

Test MSE (Random Forest Regressor): 1581635.2032971184

Test Accuracy (Random Forest Classifier): 1.0
Test Accuracy (XGBoost Classifier): 0.9782608695652174

In [6]: # Feature importance for Random Forest Regressor

import [Link] as plt
import seaborn as sns

rf_importances = grid_search_rf.best_estimator_.feature_importances_
features = X_reg.columns
indices = [Link](rf_importances)

[Link](figsize=(10, 6))
[Link]("Feature Importances (Session length prediction)")
[Link](x=rf_importances[indices], y=features[indices])
[Link]()

# Feature importance for Random Forest Classifier

rf_clf_importances = grid_search_rf_clf.best_estimator_.feature_importanc
features = X_clf_frustration.columns
indices = [Link](rf_clf_importances)

[Link](figsize=(10, 6))
[Link]("Feature Importances (User frustration classification)")
[Link](x=rf_clf_importances[indices], y=features[indices])
[Link]()

# Feature importance for XGBoost

xgb_importances = grid_search_xgb.best_estimator_.feature_importances_
features = X_clf_retention.columns
indices = [Link](xgb_importances)

[Link](figsize=(10, 6))
[Link]("Feature Importances (User retention prediction)")
[Link](x=xgb_importances[indices], y=features[indices])
[Link]()

[Link] 8/10
9/30/24, 6:05 PM model

In [7]: import joblib

[Link](grid_search_rf_clf.best_estimator_, 'random_forest_clf_model.
[Link](grid_search_rf.best_estimator_, 'random_forest_reg_model.pkl'
[Link](grid_search_xgb.best_estimator_, 'xgboost_clf_model.pkl')

Out[7]: ['xgboost_clf_model.pkl']

Business Insights and Recommendations

1. Personalized Learning: Based on user behavior, you can dynamically adjust
difficulty levels, pacing, and content delivery, significantly improving user
engagement and learning outcomes.
2. Frustration Management: The frustration model allows real-time intervention
when users struggle. By lowering difficulty levels or offering encouragement,
users are less likely to abandon the app.
3. Retention Strategies: With predictions of when a user is likely to return, the app
can strategically send notifications or offer incentives at the right time to boost
[Link] 9/10
9/30/24, 6:05 PM model

return rates.
4. Optimization of Session Length: Predicting session duration helps in delivering
optimal content in digestible chunks, keeping users engaged without
overwhelming them.

Next Steps for Flutter Integration

1. Export Trained Models: Use [Link]() to export the models, which can
be loaded into the Skillbanc backend for inference.
[Link](model_clf_frustration,
'random_forest_clf_model.pkl')
[Link](model_regression, 'random_forest_reg_model.pkl')
[Link](model_xgb, 'xgboost_clf_model.pkl')
2. Connect Models to the Flutter Backend:
Flutter developers can integrate the exported models into a Python API (e.g.,
using Flask or FastAPI) that interfaces with the Flutter app.
Ensure that data passed from the app matches the input format expected by
the models, such as pre-processing with the same scaling techniques used
in the notebook.

[Link] 10/10