CS4063 - Natural Language Processing
Due Date: Friday, September 27st by 11:55pm on Google Classroom.
Assignments are to be done individually. No late assignments will be accepted.
Submissions that do not comply with the specifications given in this document will not be marked
and a zero grade will be assigned.
Write your name and e-mail id in a comment line in on top of each source file. You are required to submit a single
zip file containing an archive of your documentation and ipython notebook on Google Classroom. You should
name your notebook as [Link] where i21-XXXX represents your student id. You must document your
algorithm/technique within the notebook using text cells.
Sentence Segmentation in Urdu and SentencePiece Tokenizer
1 Introduction
In this assignment, the goal is to implement and practice some basic text processing techniques in NLP for the
Urdu language. The Urdu language is written in a different style as compared to English, making segmentation
a challenging task. There can be several reasons but Space Insertion Problem and Space Omission Problems
are the major ones. In this assignment, your task is to perform Urdu Sentence Segmentation and implement
SentencePiece on sentences. This assignment is designed to be completed from scratch. You are free to use
basic libraries if you are comfortable doing so and you can improve existing libraries like urduhack (https :
//[Link]/), but the functions available in these libraries do not use perform up to the mark.
You are provided with the starter file ([Link]) which contains some initial code that is written in python and
will help you load the dataset and shows how the available function in UrduHack performs. A trial Urdu corpus
is provided as [Link].
You must also write a function to evaluate the performance of your segmentation technique. This requires you
to utilize your problem solving skills!
2 Background
Sentence segmentation is the process of determining longer processing units consisting of more than one word.
This task involves identifying sentence boundaries between words in different sentences. Here is an example of
text combination of two sentences:
بے چاری عوام چونکہ ہمیشہ سے دھوکہ کھانے کی عادی رہی ہے اس لئے ’’تبدیلی سرکار‘‘ کی چکنی چپڑی باتوں میں آگئی اور اپنے بہتر مستقبل
کے لئے نئی حکومت کو اقتدار کے ایوانوں تک پہنچا دیا
You have to develop a technique that will perform segmentation of sentences and remove extra white spaces in
sentences for example by passing the above statement, your model should generate output:
بے چاری عوام چونکہ ہمیشہ سے دھوکہ کھانے کی عادی رہی ہے۔ اس لئے “تبدیلی سرکار” کی چکنی چپڑی باتوں میں آگئی اور اپنے بہتر
مستقبل کے لئے نئی حکومت کو اقتدار کے ایوانوں تک پہنچا دیا۔
3 Implementation Challenges
You need to make several decisions in implementing your sentence segmentation and encoding techniqiues and
their evaluation:
1. How can you detect patterns from the text?
2. Can you identify end-words of a sentence in Urdu?
3. How can you evaluate your approach?
Note: Simply counting the number of segments may not be enough!
4. Writing the SentencePiece endode() and decode() functions!
Honor Policy
This assignment is a learning opportunity that will be evaluated based on your ability to think in a group setting, work
through a problem in a logical manner and write a research report on your own. You may however discuss verbally or
via email the assignment with your classmates or the course instructor, but you are to write the actual report for this
assignment without copying or plagiarizing the work of others. You may use the Internet to do your research, but the
written work should be your own. Plagiarized reports or code will get a zero. If in doubt, ask the course instructor.