Assignment 1: Zipf's Law and Stemming Analysis: Roll Number: 23075031 Name: Harsh Kumar Jha
Uploaded by
p1x3lph4nt0mAssignment 1: Zipf's Law and Stemming Analysis: Roll Number: 23075031 Name: Harsh Kumar Jha
Uploaded by
p1x3lph4nt0mCommon questions
Powered by AIThe study aimed to verify Zipf’s Law across English and Indian language collections. For the English dataset, where Zipf’s Law was strongly observed, the outcome aligned well with expectations, indicating natural language consistency. However, for the Hindi dataset, where the Law was only approximately followed with minor deviations in the tail region, the results still supported the law’s underlying principle but highlighted the variance due to different linguistic features. This demonstrates the law's overall applicability across languages while acknowledging morphological impacts .
The hierarchical arrangement by year and newspaper section for English, and mixed XML/UTF-8 formats for Hindi, influenced preprocessing to accommodate varied data presentation. Strategies included handling diverse file structures systematically, removing format-specific tags, and ensuring uniform tokenization. Such measures ensure consistent text input formats, necessary for cross-language analysis of Zipf’s Law and stemming applications .
Preprocessing steps included recursive directory traversal, text conversion to lowercase, removal of XML tags and punctuations, and tokenization based on whitespace. Importantly, stopword removal was intentionally not performed to maintain the natural frequency distributions required for accurate Zipf's Law analysis .
Rule-based stemming can lead to over-stemming because such methods apply general rules to remove suffixes without considering the context of a word, potentially reducing more than necessary. This is problematic in languages like Hindi with complex morphology, where inappropriate suffix removal can lead to loss of meaning or incorrect word forms. The study notes that while stemming reduces vocabulary size, it can be less effective in languages with rich inflectional structures .
Stemming aids text normalization by reducing inflected words to common bases, thereby standardizing lexical forms. English, with fewer morphological variations, benefits more, as evidenced by the significant reduction in unique forms. Hindi's complex morphology means less reduction, suggesting that stems do not always unify variations comprehensively. Thus, stemming efficiency is influenced by linguistic structure, affecting normalization levels achieved across languages .
The English dataset had 125,586 documents and a 580 MB corpus size, while the Hindi dataset had 149,481 documents with a 1.3 GB corpus size. Larger and diverse datasets allow more reliable frequency-based analysis, critical for Zipf’s Law, which depends on extensive lexical data to establish the regularities in word distributions across ranks. These comprehensive datasets ensure robust validation of theoretical assertions like Zipf’s Law .
Stemming reduces the number of unique words by converting them to their root forms. In the study, English stemming resulted in a reduction from 235,138 to 179,440 unique words, while Hindi stemming reduced unique words from 285,108 to 249,661. This suggests that English, with its simpler morphological structure, shows greater vocabulary reduction compared to Hindi, which exhibits smaller reduction due to its richer inflectional morphology .
Stemming reduces computational redundancy by decreasing unique word forms, enhancing efficiency in NLP tasks like indexing or machine learning training. Zipf's Law helps optimize data storage by highlighting key vocabulary distribution, informing scalable model designs that can prioritize higher-frequency terms. Together, these analyses enhance language model robustness and processing speed, crucial for effective, resource-efficient NLP applications .
Zipf's Law is a principle that suggests a statistical regularity in word distributions within natural language texts, indicating that the frequency of a word is inversely proportional to its rank. This study validated Zipf's Law by analyzing real-world English and Indian language document collections. For the English dataset, a strong linear trend in a log-log plot confirmed the law, whereas the Hindi dataset showed minor deviations in the tail region, suggesting the law holds approximately .
The study shows Zipf’s Law as a language-independent property by demonstrating its applicability to both English and Hindi datasets. While the law was strongly adhered to in English, and approximately in Hindi with manageable deviations, both languages followed its core principle. This indicates that Zipf's Law is not constrained by the language's native structures, reinforcing its universality across different linguistic contexts .