Scalable Anime Recommendation &
Search System Using MLOps and Big-Data
Hosted live Link : [Link]
Presented By: Ninad Kulkarni BT22CSE104
Akshat Manihar BT22CSE116
Manan Jain BT22CSE117
1. Introduction
This project delivers a Big Data–driven, end-to-end anime recommendation and search
platform, powered by scalable MLOps practices and real-time APIs. Leveraging the
distributed computing power of Apache Spark, the system ensures low-latency responses,
fault tolerance, and high throughput, making it production-ready for large-scale deployments.
The architecture is modular and cloud-native, built to handle massive datasets and frequent
retraining needs, suitable for platforms with millions of users and terabytes of user behavior
logs.
2. Method to Solve the Problem
a. Dataset Details
● Source: MyAnimeList Anime Recommendation Dataset
● Scale: Designed to scale to millions of interactions and thousands of anime titles.
● Data Augmentation:
○ Jikan API: Provides anime metadata at scale.
○ Alphacoders: Scrapes and delivers image content for UI richness.
b. System Architecture and Workflow
1. Data Ingestion Stage
● Anime datasets are stored in Google Cloud Storage (GCS).
● Files are pulled and loaded into MongoDB, an ideal NoSQL solution for handling both
structured and semi-structured big data formats like JSON.
2. Big Data Processing Stage (Apache Stack)
This stage is powered by Apache Spark, the industry-standard Big Data processing engine,
providing:
● Parallelized data processing
● In-memory computation for faster transformation
● Distributed scalability for heavy workloads
Key steps include:
● Load Raw Data from MongoDB using Spark connectors.
● Filter Active Users to focus on high-signal data.
● Scale Ratings using MinMaxScaler for normalized scoring.
● Encode IDs for both anime and users.
● Split Processed Data for training, validation, and inference.
● Save Feature Artifacts versioned with DVC for reproducibility.
⚡ Spark’s distributed architecture ensures that even growing user data and
content metadata are processed efficiently in a horizontally scalable fashion.
c. Model Development (Hybrid Recommender on Big Data)
● Content-Based Filtering: Implemented using TF-IDF vectorization on anime genres,
themes, and descriptions.
● Collaborative Filtering: Applied using Cosine Similarity on user-item interaction
matrices.
● Hybrid Strategy: Merged scores via weighted strategy, optimized via experimentation.
This hybrid system is ideal for cold start and sparse data conditions — typical of large-scale,
dynamic platforms.
d. Experiment Tracking and Model Versioning
● Comet-ML:
○ Tracked over 20+ experiments with hyperparameters, metrics, and graphs.
○ Logged performance metrics like Precision@10 and similarity score thresholds.
● DVC (Data Version Control):
○ Versioned raw and processed datasets, model artifacts, and evaluation results.
○ Connected to Google Cloud Storage for remote state management.
e. ⚙ CI/CD & Scalable Deployment (MLOps + Big Data Ready)
● Jenkins CI/CD Pipeline:
○ Auto-triggered on GitHub push.
○ Handles code pull, environment creation, DVC pull, model (re)training, Docker
build, and deployment to GKE.
● Docker & Kubernetes on GCP (GKE):
○ Deployed as microservices.
○ Uses LoadBalancer with autoscaling config.
○ Ensures high availability and dynamic scaling with zero downtime.
3. Conclusion
The project successfully delivers a Big Data–ready anime platform combining:
● Real-time search
● Scalable recommendations
● Seamless UI/UX
● Fully automated ML pipelines
The use of Apache Spark brings parallel processing, fault tolerance, and resilience, making
this platform a reliable backbone for recommendation-based web services.
It proves how Spark + MLOps tools can power a production-grade recommendation engine
using Big Data principles.
4. Future Work
To further scale the system into a real-time data streaming architecture, we propose:
● Apache Kafka Integration:
○ Record live user interactions (search clicks, watchlists).
○ Stream these into MongoDB.
○ Use Spark Structured Streaming for incremental processing.
● Apache Airflow DAGs:
○ Automate model retraining and deployment.
○ Track tasks with better observability and recovery.
5. References
1. Github: [Link]
2. Dataset: Kaggle - MyAnimeList
3. Jikan API: [Link]
4. Alphacoders: [Link]
5. Apache Spark: [Link]
6. Apache Kafka: [Link]
7. DVC: [Link]
8. Comet-ML: [Link]
9. Google Cloud Platform: [Link]