Project Report: Song Popularity Prediction
1. Introduction
The aim of this project is to build an end-to-end machine learning pipeline that predicts the
popularity of songs based on their audio features and platform-related metrics. The project
demonstrates how data-driven insights can guide marketing, playlist strategies, and A&R
(Artists & Repertoire) decisions for streaming platforms and record labels.
2. Dataset
• Primary Dataset:
Most Streamed Spotify Songs 2023 (Kaggle)
o Features included:
§ Track Name
§ Artist Name
§ Release Year & Month
§ Playlist presence
§ Chart appearances
§ Audio features: BPM, Key, Mode, Danceability %, Energy %, etc.
§ Target: Streams (in millions)
• Notes on Dataset:
o Streams are only given in millions (not ideal for real-world variety).
o Highly skewed distribution of streams → log transformation is required.
o Additional datasets could be incorporated for better results, e.g.:
§ Spotify Web API (real-time audio features, popularity score)
§ Billboard/YouTube trending datasets (cross-platform insights)
3. Tools & Libraries
Core Libraries
# Data handling
import numpy as np
import pandas as pd
# Visualization
import [Link] as plt
import seaborn as sns
# Machine Learning
from sklearn.model_selection import train_test_split, cross_val_score
from [Link] import LabelEncoder
from sklearn.linear_model import LinearRegression
from [Link] import RandomForestRegressor
from xgboost import XGBRegressor
# Evaluation Metrics
from [Link] import mean_absolute_error, mean_squared_error,
r2_score
# Saving Models
import joblib
Deployment
# Streamlit for dashboard
import streamlit as st
4. Methodology
Step 1: Data Cleaning
• Remove duplicates & null values
• Drop irrelevant columns (e.g., "key")
• Convert categorical features (artist, mode, etc.) to numeric (Label Encoding / One-Hot
Encoding)
Step 2: Exploratory Data Analysis (EDA)
• Distribution of streams (highly right-skewed)
• Correlation heatmap (identify strong feature relationships)
• Feature vs. target plots (e.g., BPM vs. Streams, Release Month vs. Streams)
• Key finding: Playlist presence is more important than raw audio features
Step 3: Feature Engineering
• Encode categorical values (major/minor → 0/1)
• Log transform the target (streams) to reduce skewness
• Normalize/scale features if required
Step 4: Model Building
• Frame as a regression problem (predict numeric streams).
• Split dataset: 80% training, 20% testing
• Train and compare models:
o Linear Regression (baseline, interpretable but weak)
o Random Forest (best performance, interpretable via feature importance)
o XGBoost (competitive, slightly better than Random Forest in some tests)
Step 5: Model Evaluation
• Metrics: MAE, RMSE, R² Score
• With log transformation:
o Random Forest gave best balance of accuracy + interpretability
o Linear Regression underperformed
Step 6: Feature Importance
• Random Forest and XGBoost showed:
o Spotify Playlist presence → strongest predictor of popularity
o Audio features (danceability, BPM, energy) are secondary
Step 7: Deployment
• Use Streamlit to build an interactive dashboard:
o Upload/input song features
o Predict popularity (streams in millions)
o Show EDA insights and model explanations
• Export model as .pkl using joblib and load into Streamlit app
5. Results
• Best Model: Random Forest with log-transformed targets
• Key Insights:
o Song popularity is strongly tied to playlist inclusion rather than only audio
features.
o Distribution of streams shows “winner-takes-most” behavior.
o Log transformation improves stability and interpretability.
6. Business Applications
• Playlist Strategy: Prioritize getting songs into major Spotify playlists.
• Promotion Planning: Use feature importance to guide marketing campaigns.
• Artist Development: Identify rising artists early using data-driven predictions.
• Decision Support: Provide A&R teams with interpretable dashboards and visual
insights.
7. Future Improvements
• Use larger and more varied datasets (include songs with lower stream counts).
• Integrate NLP (lyrics analysis) for deeper insights.
• Add social media metrics (TikTok trends, Instagram reels) as features.
• Monitor model drift and retrain with updated datasets.
• Explore deep learning models if dataset size increases.
8. Deployment Demo
• Streamlit Dashboard Includes:
o Project Overview & Problem Statement
o EDA Insights with graphs
o Prediction Tool (input features → predicted popularity)
o Business Insights (actionable recommendations for marketing & A&R)