Scribd
Upload
14 views11 pages

Hive and PySpark Data Processing Steps

The document summarizes the steps taken to analyze a dataset using Hive and PySpark. Key steps include: 1) Loading data into Hive and dropping the ID column as it was not important for classification. 2) Loading data into PySpark, preprocessing including one-hot encoding categorical variables, and splitting data into train and test sets. 3) Defining a logistic regression model and evaluating model performance using AUC and F1 score on the test data.

Uploaded by

HPot PotTech
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views11 pages

Hive and PySpark Data Processing Steps

The document summarizes the steps taken to analyze a dataset using Hive and PySpark. Key steps include: 1) Loading data into Hive and dropping the ID column as it was not important for classification. 2) Loading data into PySpark, preprocessing including one-hot encoding categorical variables, and splitting data into train and test sets. 3) Defining a logistic regression model and evaluating model performance using AUC and F1 score on the test data.

Uploaded by

HPot PotTech
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Assignment 2

Hive:
Loaded the data into hive. Dropped the ‘ID’ column and saved the data for further processing in spark.
Dropped ‘ID’ column as it was not an important feature for classification.

a) Create and load table

b) Drop ID column and dump table to local for further processing


PySpark:

Steps:

1) Load Data
a) Load the dataset and infer schema

b) Change the column names for better interpretation

2) Preprocessing and Understanding Data


a) None of the columns have Null value. I dropped the ‘ID’ column using hive as it was not useful for
classification.

b) Get the number of examples which belong to each class.

We can see that there are more examples belonging to class 0 are more, so the dataset is not balanced.

c) As the dataset is not balanced, add a column "weight" which tells how important an example is during
training. We give more weight of 2.7 to examples that belong to class 1 and give weight 1 to examples
that belong to class 0.

d) Look at the number of Defaulters/Non-Defaulters for each sex.


e) Distribution of values for Pay_2 column for the dataset. Value of 0 and -2 is not defined for this
dataset, still a lot of examples have these values.

f) Distribution of values for the Age column for this dataset. We can see most of the data is of people of
age group 24-40.
g) Distribution of values for the Marriage column for this dataset. Value of 0 is not defined for this
dataset, but some examples have this value.

h) Distribution of values for the Balance_limit column for this dataset.


i) Randomly split the dataset into training and testing in 60:40 ratio.

j) One-hot encode the categorical variables Marriage and Education.

k) Assemble all the features into one column using Vector assembler.
3) Define Model and Pipeline
a) We use Logistic Regression model and we provide input and output columns to model for training. We
also provide weight that the model should give each example during training.

b) Define Pipeline to chain multiple transformations to specify machine learning workflow.

4) Get Model Predictions


a) Fit the model and get predictions on the test data.

b) Get the number of examples where prediction is the same as true label.
c) Print number of examples which have true label 0/1. Print number of examples which has predicted
label 0/1. Print output for some test examples.

5) Evaluate Model
a) Evaluate the model using AUC metric which is in the BinaryClassificationEvaluator package. Give
probabilities and labels as input to the evaluator.
b) Using F1 score for evaluating the performance of the classifier. Metrics used for evaluation depend on
what is the end goal we want to achieve, but in most scenarios F1 score is a good metric to evaluate
performance of the classifier if the dataset is unbalanced.

Spark WebUI

Most time is taken by [Link]() paragraph. This task started many jobs from Job id 1564 to 1636 on my
laptop. They all took around 3 seconds combined. This is the total duration. For calculating the Executor
Computing Time, we have to go inside each job and sum the Executor Computing Time of each stage of
the job. We can see from the following figure that job id 1564 to job id 1636 have the same job group.
Most time taken by a single job was 0.3s. There were few jobs which had a duration of 0.3s. One such
job is job id 1543 in figure below.

Common questions

Powered by AI

The age distribution showed that most data were for people aged between 24-40. This information is important as it highlights the demographic characteristics of the dataset, which may influence the behaviors and features being analyzed, and potentially affect the generalizability of the model .

The task that took the most time during model training was model.fit(), which initiated many jobs from Job id 1564 to 1636. Analyzing this involves going into each job to sum the Executor Computing Time of each stage, and it was noted that most of these jobs took around 3 seconds combined .

The pipeline is significant as it enables chaining multiple transformations to define the machine learning workflow clearly. It helps automate the data processing and training tasks, ensuring they are executed in a specific sequence, thus reducing errors and improving reproducibility and maintenance of the code .

Evaluating the distribution in 'Pay_2' was necessary to identify data quality issues. It was found that values of 0 and -2 were not defined but still appeared in many examples, indicating potential data entry errors or processing issues that could impact the model's performance if not addressed .

Class imbalance was addressed by adding a 'weight' column to the dataset, which assigned a weight of 2.7 to examples belonging to class 1 and a weight of 1 to examples belonging to class 0. This approach gave more importance to the less frequent class to ensure that the model adequately learns from these instances during training .

The dataset was split into training and testing in a 60:40 ratio, likely because this allocation ensures that there is sufficient data for training while keeping a meaningful portion of data for testing to assess the model's performance. This balance helps in getting reliable performance metrics by testing on unseen data .

Logistic Regression was chosen for the classification task. Key parameters during its setup included specifying the input and output columns for training and incorporating the 'weight' column, which accounts for the importance of each example during training to mitigate class imbalance .

The model was evaluated using the AUC metric and the F1 score. The F1 score might be preferred, especially when the dataset is unbalanced, because it considers both precision and recall, providing a more balanced measure of the classifier's performance than accuracy alone .

The 'ID' column was dropped because it was not an important feature for classification. It was deemed unnecessary for the analysis and modeling tasks that were to follow, as it did not contribute to the predictive capabilities of the model .

The method used to preprocess categorical variables was one-hot encoding, applied to the 'Marriage' and 'Education' columns. One-hot encoding is essential because it converts categorical variables into a format that can be provided to machine learning algorithms to work more efficiently, as most algorithms expect numerical input .

You might also like