School of Information Technology
Course : Specialist Diploma in Business and Big Data Analytics
Module : Text Mining and Analytics (ITD253)
Practical : Text Preprocessing using Python
Objectives : At the end of this session, students should be able to:
• Appreciate the objectives of text preprocessing
• Use RapidMiner to do text preprocessing steps like tokenization, transformation,
filtering, stop words handling, stemming/lemmatization
• Differentiate between stemming and lemmatization
Software(s) : PyCharm IDE and Python or Google Colab
Introduction
Python language is commonly used for text preprocessing purpose. It has suitable libraries built that we can use
on the python script, such as, nltk.
We will go through a sample program – [Link] to illustrate the various processing steps that can be
done using a Python program.
Essentially the python program has implemented the following:
1. Read from a file
2. Sentence tokenizer
3. Word tokenizer using split
4. Get the first few words
TMA Practical 1 Page 1 of 8 2018S2
SDBBDA
5. Basic pre-processing like format to lowercase and remove punctuation
6. Remove stop words
7. Handle stemming and show stemming effect
8. Get top word frequency and count
9. Create bigram and extract the bigrams for the top 5 frequency words
You may execute the python program either using Google Colab or PyCharm IDE.
Part A: Using PyCharm and Python
Installing Python
1. Go to [Link] Click on the “Download Python 3.7.1” button for the
latest Python 3 version for Windows. Save the executable file.
2. Open the file and accept the default settings to install Python into your machine.
TMA Practical 1 Page 2 of 8 2018S2
SDBBDA
3. Open up Command Prompt to see if Python has been successfully installed.
4. Type the command “python” into the command prompt. If Python has been successfully installed
and added to the path, this command should work anywhere.
5. Try typing some simple commands to test it out! For example, typing “10**4” (i.e. 10 raised to the
4th power) will yield “10000”.
TMA Practical 1 Page 3 of 8 2018S2
SDBBDA
6. You can close Command Prompt now. We shall next proceed to install an IDE (Integrated
Development Environment) to facilitate our coding journey.
Installing PyCharm IDE
1. Go to [Link] Click on the “Download”
button under Community. Save the executable file.
2. Open the file and run the wizard. You can accept the default settings.
TMA Practical 1 Page 4 of 8 2018S2
SDBBDA
3. When prompted, you do not need to create Desktop Shortcuts. But it would be good to create
associations for .py files.
4. Finally, click on “Install”.
TMA Practical 1 Page 5 of 8 2018S2
SDBBDA
Creating a new project and installing modules
1. Click on File >> New Project.
2. In the “Create Project” window, replace Untitled with TextPreprocessing. Make sure your
interpreter is pointing to the [Link] in the correct path you have just installed Python to. If not,
click on the gear icon on the right and choose the correct path or you can select to use existing
interpreter.
3. Click on File >> Settings. In the following popup you should see the project on the left (and the
project name). Click on the project and you see two subgroups, Project Interpreter and Project
Structure. Click on "Interpreter" and the control panel for the interpreter should open and the
installed packages. Click on the small plus-sign on the right.
TMA Practical 1 Page 6 of 8 2018S2
SDBBDA
4. In the popup-window search for nltk and click on Install Package. If the installation is successful you
can close the window and PyCharm informs you that nltk is installed successfully.
5. Click OK to close the settings window.
6. In the project panel, right-click on the TextPreprocessing folder, select Show in Explorer. Copy
[Link] and [Link] into the TextPreprocessing folder.
TMA Practical 1 Page 7 of 8 2018S2
SDBBDA
7. Now, you should see the two files in your PyCharm project folder. Run the [Link]
python file. You should see the different output generate for each step of the text processing
process.
Part B: Using Google Colab
1. Follow ITD251 Lab01 to setup your Google Colab environment.
2. Create a new Jupyter Notebook and named it as Text [Link].
3. Add the following code to upload [Link].
from [Link] import files
uploaded = [Link]()
for fn in [Link]():
print('User uploaded file "{name}" with length {length}
bytes'.format( name=fn, length=len(uploaded[fn])))
4. Download the [Link] program.
5. Copy the python codes into your Jupyter notebook.
6. Execute the program. You should see the different output generate for each step of the text
processing process.
Click here to view a sample working code in Google Colab. Remember to run the first code snippet to upload
[Link].
---- The End ----
TMA Practical 1 Page 8 of 8 2018S2