Department of Computer Science & Engineering
SUMMER TRAINING PROJECT REPORT FILE
NEWS CATEGORY CLASSIFIER
Faculty Guide: Submitted by:
Ms. Anjali Bakshi Name: Arnav Mahajan
Roll no: 2336768
1
TABLE OF CONTENT
Content Page no
Objective 3
Technology Used 4
Design and Development 5-7
Process Flow 8-9
Screenshots of input and output 10-11
Code Snippets 12-17
Key Challenges and Solutions 18-19
Conclusion and Learning Outcome 20
2
OBJECTIVE
The primary objective of this project is to develop a machine learning-based text classification system
that can automatically categorize news articles into their respective topics, such as sports, politics,
technology, business, entertainment, and others, based on the content of the article.
The specific goals of the project include:
• Automate the classification process of textual news data using Natural Language Processing
(NLP), eliminating the need for manual sorting and labeling.
• Build a reliable and scalable ML model, specifically using the Random Forest Classifier, trained
on TF-IDF-transformed news content for multi-class prediction.
• Clean and preprocess raw text data using steps like lowercasing, stopword removal,
punctuation handling, and lemmatization to improve model accuracy.
• Visualize class distribution, accuracy, and model performance using metrics such as confusion
matrix, precision, recall, and F1-score to evaluate effectiveness.
• Design and deploy an intuitive Streamlit-based web interface, where users can:
o Upload datasets for training
o Visualize categories and statistics
o Perform real-time predictions on custom news inputs
• Enhance accessibility and usability of the model by providing a web-based app suitable for
journalists, media houses, and educators.
• Support balanced classification by considering techniques for handling skewed or imbalanced
categories, ensuring fair predictions across all labels.
This project not only demonstrates the application of NLP and ML in real-world scenarios but also
emphasizes usability by integrating the solution into a responsive web environment for seamless user
interaction.
3
Technology used
Programming Language
• Python – The primary language used for data preprocessing, feature extraction, machine learning
model training, and web app development.
Libraries & Frameworks
• Pandas – For loading and manipulating the dataset (news text + category).
• NumPy – For numerical operations and data transformation.
• NLTK (Natural Language Toolkit) – For text preprocessing: tokenization, stopword removal,
lemmatization.
• Scikit-learn (sklearn) – For:
o TF-IDF vectorization
o Train-test split
o Random Forest Classifier
o Model evaluation (accuracy, F1-score, confusion matrix)
• Matplotlib & Seaborn – For plotting class distributions and visualizing model performance metrics like
confusion matrix and bar charts.
Machine Learning Algorithm
• Random Forest Classifier – A robust and interpretable ensemble learning model used for accurate
multi-class classification of news articles.
Web Application Framework
• Streamlit – For building a responsive, interactive web interface that allows users to:
o Upload news datasets
o Train and evaluate the model
o Make real-time predictions from custom news input
o Visualize outputs such as confusion matrix and accuracy scores
4
️ File Handling & Data
• CSV Format – Used for dataset storage ([Link], [Link]), including article text and its category.
DESIGN AND DEVELOPMENT
The News Category Classifier was designed using a modular and layered architecture, ensuring that
each component—from data ingestion to model prediction and web deployment—was developed
with clarity, scalability, and usability in mind. The system follows a typical machine learning pipeline,
enhanced with Natural Language Processing (NLP) techniques and a Streamlit-based GUI for
seamless user interaction.
🔹 1. Dataset Loading and Exploration
• The project begins with loading the news dataset ([Link]) which contains text content
and their associated categories.
• Pandas was used to explore the data, analyze class distribution, detect null values, and inspect
text samples.
🔹 2. Text Preprocessing (NLP Pipeline)
To ensure effective model training, the raw news text underwent comprehensive cleaning:
• Removal of punctuation, numbers, and special characters using regex.
• Conversion of text to lowercase for normalization.
• Tokenization of sentences and words using [Link].
• Removal of stopwords using [Link].
• Lemmatization using WordNetLemmatizer to reduce words to their base form.
• The cleaned text was stored in a new column called clean_text.
5
🔹 3. Feature Extraction
• The cleaned text was converted into numerical vectors using TF-IDF (Term Frequency–Inverse
Document Frequency) from sklearn.feature_extraction.text.
• This method captures the importance of words relative to documents, reducing noise and
emphasizing meaningful terms.
🔹 4. Label Encoding and Splitting
• The target categories were encoded into numerical labels using LabelEncoder.
• The dataset was split into training and test sets (80:20 ratio) using train_test_split() to
evaluate generalization.
🔹 5. Model Training
• The primary classifier used was Random Forest Classifier from [Link].
• The model was trained on the TF-IDF-transformed data.
• It was chosen for its robustness in handling high-dimensional data and multi-class classification.
🔹 6. Model Evaluation
• Model performance was assessed using:
o Accuracy score
o Confusion matrix
o Precision, recall, and F1-score (via classification report)
• Visualization tools like Seaborn and Matplotlib were used to display results graphically.
🔹 7. Streamlit Web Interface
To enhance usability:
• A clean and responsive web interface was built using Streamlit.
• Key features include:
o File uploader for new datasets
o Dropdown to choose classifier
6
o Text box for manual news prediction
o Visual output for metrics and predictions
• This made the app suitable for users without any programming background.
🔹 8. Screenshots and Testing
• Final screenshots were taken of the dataset preview, model output, confusion matrix, and live
predictions.
• The application was tested for various categories and input cases to validate accuracy and
responsiveness.
This design ensures the system is modular, extensible, and user-friendly, making it suitable for real-
world use in journalism, content filtering, and educational applications.
PROCESS FLOW
The end-to-end flow of the project is structured into the following major steps:
1. Dataset Import
a. Load the news dataset ([Link]) using Pandas.
b. Inspect text data and categories.
2. Text Preprocessing
a. Clean the text: remove special characters, numbers, and punctuation.
b. Convert text to lowercase.
c. Tokenize, remove stopwords, and apply lemmatization.
3. Feature Extraction
a. Apply TF-IDF Vectorization to convert text into numerical feature vectors.
4. Label Encoding
a. Encode news category labels into numeric values for ML compatibility.
5. Train-Test Split
a. Split the dataset into training and testing sets (e.g., 80:20 split).
6. Model Training
a. Use Random Forest Classifier to train the model on TF-IDF features.
7. Model Evaluation
7
a. Measure accuracy, precision, recall, and F1-score.
b. Plot confusion matrix to visualize model performance.
8. Web App Deployment
a. Build a Streamlit app to:
i. Upload CSV files
ii. Visualize data and results
iii. Input custom news and display predicted categories
FLOW CHART
Raw News Data
Text Cleaning & Lemmatization
TF-IDF Vectorization
Train/Test Split
Random Forest Classifier
Model Evaluation (Confusion Matrix, Accuracy)
Streamlit Web App (Input → Prediction)
8
Screenshots of Demo Input and Output
Input Screen (Homepage)
The homepage of the News Category Classifier is developed using Flask (Python backend) and a
custom-designed HTML & CSS frontend. It serves as the main entry point for users to interact with the
classifier in a user-friendly way.
• In this area we enter our news highlight we want to categorize
• then click on “Predict Category” option to find out the category the news belongs to
• Once we have entered the news (eg:- technological news), it will show us the category of
technology after the prediction.
9
Code Snippets
10
This block imports all essential libraries required for:
• Data handling (pandas, numpy)
• Text preprocessing (nltk)
• Machine learning algorithms (RandomForest, Logistic Regression, etc.)
• Evaluation metrics and plotting tools (matplotlib, seaborn)
• Model saving (pickle)
• Downloads necessary NLTK data files used for tokenization and lemmatization.
• Loads the [Link] and [Link] datasets into Pandas DataFrames for processing and
training.
11
• Removes rows from the training and testing data that have null values in the text or label
columns, ensuring data cleanliness.
• Defines a function to lemmatize words (convert them to their base form), remove punctuation,
and lowercase the text. Applies it to both datasets.
• Converts the cleaned text data into numerical form using TF-IDF. This technique captures word
importance and ignores stopwords. It helps the model focus on meaningful terms in the news
content.
• Transforms categorical text labels (e.g., ‘Politics’, ‘Sports’) into numeric values that can be used by
machine learning models.
12
Trains five different classifiers and predicts on the test data:
• Logistic Regression
• Random Forest
• Decision Tree
• Naive Bayes
• Gradient Boosting
This helps compare their performance and choose the best one.
13
• Generates a comparison table showing the performance (accuracy, precision, recall, F1-score) of
each model. This helps determine the most effective classifier.
• Uses bar plots and heatmaps to visually compare how well each model performs. These
visualizations are useful for presentations and interpretation.
• From the bar graph mentioned below it is clear that we are getting best accuracy of 91.3% or 0.913
from logistic regression model among all the other models
14
• After we have done the backend work, we have created a website using Flask, HTML and CSS in
[Link].
The functions of that [Link] performs are: -
• Loads the trained model and tools using pickle.
• Defines a Flask route to:
• Accept user input
• Preprocess the text
• Predict the category
• Display the result on the same page
15
16
Key Challenges and Solutions
Challenge 1: Textual Noise and Irrelevant Words
Description:
Raw news articles often contain HTML tags, punctuation, numbers, and stopwords that reduce model
accuracy.
Solution:
Implemented comprehensive text preprocessing, including:
• Lowercasing
• Removing punctuation and digits
• Stopword removal using NLTK
• Lemmatization using WordNetLemmatizer
️Challenge 2: High Dimensionality of Text Data
Description:
News articles contain thousands of unique words, leading to sparse and high-dimensional vectors.
Solution:
Used TF-IDF Vectorization with parameters like max_df=0.7 and stop_words='english' to reduce noise
and dimensionality while retaining important terms.
️Challenge 3: Imbalanced Dataset
Description:
Certain categories like “Politics” or “Sports” may dominate, while others like “Technology” or “Business”
are underrepresented.
Solution:
Used weighted performance metrics (like weighted F1-score) during model evaluation to fairly assess
performance across all classes.
Challenge 4: Real-time Prediction Integration
17
Description:
Bridging the gap between the trained model and user-friendly interface was initially difficult.
Solution:
Used Flask web framework to build a lightweight app that:
• Accepts user input
• Processes and predicts category in real time
• Returns readable output on a clean HTML page
Challenge 5: Converting Model Output to Readable Labels
Description:
The model’s output was numeric (e.g., 0, 1, 2), not user-friendly.
Solution:
Used LabelEncoder.inverse_transform() to convert numeric labels back into their original category
names (e.g., “Sports”, “Business”).
18
Conclusion and Learning Outcome
Conclusion
• The News Category Classifier project successfully demonstrates the use of Natural Language
Processing (NLP) and Machine Learning (ML) to automatically classify news articles into
predefined categories such as Politics, Sports, Business, and Technology. By combining efficient
text preprocessing, TF-IDF feature extraction, and the Random Forest classification model, the
system achieved reliable and interpretable results.
• Additionally, the integration of the trained model into a user-friendly Flask-based web application
enabled real-time predictions, making the solution accessible and interactive for end users.
Despite challenges like class imbalance and text variability, the project was able to deliver
accurate and scalable results with minimal latency.
Learning Outcomes
Through this project, the following technical and practical skills were developed:
• Understanding of text preprocessing techniques such as tokenization, stopword removal, and
lemmatization using NLTK.
• Proficiency with TF-IDF as a feature engineering method for converting text into meaningful
numeric representations.
• Hands-on experience with multiple ML models like Random Forest, Logistic Regression, Naive
Bayes, and Gradient Boosting.
• Ability to evaluate models using classification metrics like accuracy, precision, recall, F1-score,
and confusion matrix.
• Integration of ML models into web applications using Flask, along with HTML/CSS for frontend
design.
19
• Deployment-ready workflow, including model serialization using Pickle and user interaction via
forms and route handling.
20