Data Science Assignment Report
Dataset Source:
The dataset was prepared manually (not downloaded).
It consists of 20 short posts, written in a style similar to tweets or Facebook posts.
Each post is a short sentence, and the dataset includes hashtags, emoji, and general text expressions.
Dataset Creation Date:
2025-09-18
Why this dataset?
Short posts mimic real social media data.
It contains enough variety (coding, data science, daily life and motivation) to demonstrate text
analysis.
The dataset is small but sufficient to show preprocessing, similarity, and feature extraction
techniques.
1. Text Preprocessing
Concept:
Raw text is messy (contains hashtags, emoji, punctuation, common filler words). To make it usable for
analysis, we:
1. Tokenize means split sentences into words.
2. Remove Stopwords means remove meaningless common words (e.g., “the”, “and”).
3. Stemming means reduce words to their base form (e.g., running changes to run).
Code Snippet:
Result Example:
Original: "Python is awesome for text processing. #python #nlp"
After preprocessing: "python awesom text process python nlp"
2. TF-IDF (Term Frequency – Inverse Document Frequency)
Concept:
TF-IDF measures how important a word is in one document compared to all documents.
High score means the word appears often in a document but not in others.
Useful for keyword extraction and document representation.
Code Snippet:
Results (Top Words):
python: 0.22
project: 0.19
learn: 0.18
report: 0.17
similari: 0.16
Visualization:
3. Vector Space Model (Cosine Similarity)
Concept:
Each document is represented as a vector using TF-IDF.
Cosine similarity measures how close two vectors are (1 = identical, 0 = no similarity).
Code Snippet:
Results:
Most similar pairs (examples):
o Doc 14 & Doc 18 has 0.21 similarity
o Doc 16 & Doc 18 has 0.19 similarity
4. N-Grams (Bag of Words + Phrases)
Concept:
Bag of Words: Count word frequency.
N-Grams: Capture common multi-word phrases (e.g., “machine learning”).
Code Snippet:
Results (Top N-Grams):
project: 4
learn: 3
data scienc: 2
machin learn: 2
text process: 2
Acknowledgment
I acknowledge the guidance provided by Dr. Khalid Iqbal Khattak and external help in understanding text
preprocessing, TF-IDF, and similarity measures.