Data Pre-processing exercise
Data scientist perform very critical tasks of extracting meaning from data, however, during the
discussion on Data mining processes/ methodology, it was evident that both CRISP-DM and
SEMMA put a critical emphasis on data cleaning. This is so because many real-world datasets
are noisy and thus a data scientist needs to invest considerable amount of time, skill, creativity,
and effort in cleaning the data. This practical task aims at exposing you to these very critical
skills of data pre-processing and problem solving. Attempt the practical exercise in groups and
feel free to check documentations such as sckitlearn, pandas, numpy, matplotlib.
You are provided with two datasets namely [Link] and [Link].
The datasets are noisy and therefore as per the third stage of CRISP-DM you are required to
clean the data. Using python programming language and its libraries perform data pre-
processing on the two sets. Save your pre-processed data and notebooks for evaluation and
feedback.
Key tasks in achieving this objective
1. Load your data to python environment such as jupyter notebook using pandas
2. Print the first 10 rows of the data to have a look at how the data looks like
3. Determine the shape of your dataset in terms of number of rows and column
4. Using the relevant functions determine if any column has null values
5. Compute the measures of central tendency in the numeric columns of the dataset
6. Deal with null values by filling them with values that make business sense such as the
measures of central tendency
7. Determine any outliers in your dataset either through visualization or using the numpy
Z-score
8. Drop the rows with outliers
9. Check for duplicates using an ideal method/function and remove the duplicates if any.
10. Print/check the final shape of your dataset after performing this pre-processing
11. Using pandas .to_csv ( ) save the pre-processed data sets
Specifically, for the ActivityData. csv
On top of the above tasks also perform the following
1. The column names are not provided but the client has informed you that the column
names are ('PhoneNumber-SubscriberIdentityNumber', 'Duration', 'Activity',
'ErrorCode' ). Note that the first column has two values in the same cell which are
separated with a hyphen (-). You are required to split this into two distinct columns
named ( ‘PhoneNumber', 'Subscriber Identity Number' ) .
2. Note that the second last column contains an activity and error code so split this into
separate columns named (‘Activity’, ‘Error Code’).
3. Now you will notice that after the split the old combined columns still exist so drop
them.
4. Print the first 10 rows of the new formatted dataset to confirm everything works okay.
5. Check for null values in all columns.
6. Now the last column with Error code will contain very many null values since if there
was no error it is indicated as Nan so to solve this replace the null values with 0 to mean
no error.
7. Optional- you can use matplotlib to visualize the necessary columns such as call
duration column to help in better understanding of the dataset.
8. Save the new pre-processed dataset.
As part of contributing to the community and building your online profile as a data scientist
you can create a Gitihub profile and upload your code there for others to comment.