0% found this document useful (0 votes)

7 views11 pages

Linear & Logistic Regression Models Guide

Q: What are the implications of the R2 score and mean squared error in evaluating a linear regression model's performance, and what do they indicate in the models from the document?

The R2 score indicates the proportion of variance in the dependent variable that can be explained by the independent variable, with values closer to 1 indicating a better fit. The mean squared error provides an average squared difference between actual and predicted values, with lower values indicating more accurate predictions. In Source 1, the models use these metrics to evaluate performance: a high R2 score suggests a good model fit to data, while low mean error suggests high prediction accuracy. Together, they provide a comprehensive assessment of model effectiveness and its precision in predictions .

Q: How does converting categorical data into numeric format influence the application of the Apriori algorithm, and how is this implemented in the document?

Converting categorical data into numeric format is crucial for applying the Apriori algorithm, which relies on mathematical computations on numerically-represented data to identify frequent itemsets. This conversion allows the algorithm to process non-numeric data by associating distinct numerical codes with categorical values, enabling calculations of support, confidence, and lift metrics. In the document, this is achieved using the TransactionEncoder, which transforms datasets of categorical transactions into a Boolean format that the Apriori algorithm can process to generate frequent itemsets and rules .

Q: How does logistic regression differ from linear regression in terms of predicting outcomes, and what is a practical example from the document?

Logistic regression differs from linear regression in that it is used for binary classification tasks rather than predicting continuous outcomes. While linear regression predicts values across a continuum, logistic regression predicts the probability of a binary event occurring, returning a value between 0 and 1. A practical example from the document is predicting whether someone will purchase a car based on parameters like age and estimated salary using logistic regression . This contrasts with linear regression used for predicting sales or house prices, which are continuous values .

Q: In the context of the assignment, how does frequent itemset mining using the Apriori algorithm help in understanding transactional data?

Frequent itemset mining using the Apriori algorithm helps in discovering regularities or patterns among sets of items across large datasets. It enables the identification of groups of items commonly purchased together, which can be used for market basket analysis, customer recommendations, and inventory management. By generating association rules, businesses can gain actionable insights on product placements and marketing strategies. The document shows the application of Apriori to generate itemsets and association rules from sales transactions, revealing useful patterns in consumer behavior .

Q: How does the process of importing and splitting the dataset into training and testing sets facilitate the creation of a Linear Regression model?

Importing the dataset is a crucial first step in Linear Regression as it allows access to the data necessary for model building. Splitting the dataset into training and testing sets helps in evaluating the model's performance on unseen data, ensuring that it can generalize well. The training set is used to train the model, allowing it to learn patterns, while the test set is used to verify the model's accuracy. In Source 1, the data is divided into a 7:3 ratio, ensuring that there's enough data to train and test effectively. Evaluating with the test set, as shown with measures like R2 score and mean error, presents the model's predictive accuracy and robustness on new data .

Q: Why is tokenization considered a fundamental step in text preprocessing, particularly in sentiment analysis? Provide an example from the document.

Tokenization is a fundamental step in text preprocessing because it breaks down text into individual components such as words or sentences, which are necessary for further analysis like sentiment detection. It allows algorithms to understand and process text data more effectively. In the document, for sentiment analysis, text must be tokenized to assess each word's sentiment individually. For example, in sentiment analysis of review messages, the text is tokenized into words before analyzing their sentiment scores using tools like VADER .

Q: Why is it important to visualize data before constructing a Linear Regression model, and how is this achieved in the datasets discussed?

Visualizing data before constructing a Linear Regression model is vital as it helps identify potential relationships, patterns, and outliers. Understanding these elements can influence model selection and parameter tuning. In the datasets discussed, data visualization is achieved through scatter plots which show the relationship between variables such as 'TV' and 'Sales'. This initial step allows for the assessment of the linear relationship, informing whether Linear Regression is appropriate. Visualization can highlight data trends that linear models can exploit .

Q: Evaluate the significance of preprocessing steps, such as removing stopwords and lemmatization, in the context of text analytics, as discussed in the document.

Preprocessing steps like removing stopwords and lemmatization are crucial in text analytics as they help in refining the text data to a form suitable for analysis. Removing stopwords eliminates common words that carry little meaningful information, thus reducing dataset noise, while lemmatization reduces words to their base form, ensuring consistency in text representation. These processes lead to a cleaner, more effective dataset better suited for techniques like sentiment analysis. In the document, preprocessing these steps facilitate clearer analysis and visualization of textual data by focusing on the substantive content .

Q: What role does data visualization, like word clouds, play in understanding the patterns within text data, as applied in the document's examples?

Data visualization, such as word clouds, plays a significant role in text analytics by providing a graphical representation of the frequency of words, making it easier to identify patterns and trends within text data. By visually highlighting the most common words based on their size, word clouds can quickly convey the salient topics and themes within a dataset. In the document, word clouds are used to visualize WhatsApp data, allowing for an easy grasp of common words and sentiments expressed in messages, facilitating better understanding of communication patterns .

Q: What are the challenges and solutions associated with sentiment analysis as displayed by the various sentiment analysis methods in the document?

Sentiment analysis involves challenges such as dealing with context, sarcasm, and varying sentiment expressions that affect accuracy. The methods used must process nuances in language, requiring sophisticated algorithms. The document demonstrates the use of VADER, which is suited for sentiment analysis tasks in social media contexts due to its dictionary-based approach that captures polarity and sentiment intensity effectively in short texts. To mitigate challenges, the analysis may combine multiple approaches and leverage large sentiment datasets for improving model training and reducing misinterpretations .

Da practicals

Uploaded by

rokadeprajesh

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

7 views11 pages

Linear & Logistic Regression Models Guide

Da practicals

Uploaded by

rokadeprajesh

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

Assignment No: 1 - Linear and Logistic Regression

1. Set A1 - Create ‘sales’ Data set having 5 columns namely: ID, TV, Radio, Newspaper and
Sales.(random 500 entries) Build a linear regression model by identifying independent
and target variable. Split the variables into training and testing sets. then divide the
training and testing sets into a 7:3 ratio, respectively and print them. Build a simple linear
regression model.

import pandas as pd
import numpy as np
import [Link] as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import r2_score,mean_squared_error
df = pd.read_csv('/home/ctbora/Downloads/[Link]') # Importing the data set
new_df = df[['TV','Sales']]
X = [Link](new_df[['TV']]) # Storing into X the 'Engine HP' as [Link]
y = [Link](new_df[['Sales']]) # Storing into y the 'MSRP' as [Link]
print([Link]) # Vewing the shape of X
print([Link]) # Vewing the shape of y
[Link](X,y,color="red") # Plot a graph X vs y
[Link]('TV vs. Sales')
[Link]('TV')
[Link]('Sales')
[Link]()
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.333,random_state=15)
regressor = LinearRegression() # Creating a regressior
print("Training Set :")
print(X_train)
print("Test Set :")
print(X_test)

[Link](X_train,y_train) # Fiting the dataset into the model

[Link](X_test,y_test,color="green") # Plot a graph with X_test vs y_test
[Link](X_train,[Link](X_train),color="red",linewidth=3) # Regressior line showing
[Link]('Regression(Test Set)')
[Link]('TV')
[Link]('Sales')
[Link]()
y_pred = [Link](X_test)
print('R2 score: %.2f' % r2_score(y_test,y_pred)) # Priniting R 2 Score
print('Mean Error :',mean_squared_error(y_test,y_pred)) # Priniting the mean error
2. Set A2 - Create ‘realestate’ Data set having 4 columns namely: ID,flat, houses and
purchases (random 500 entries). Build a linear regression model by identifying
independent and target variable. Split the variables into training and testing sets and print
them. Build a simple linear regression model for predicting purchases.

import pandas as pd
import numpy as np
import [Link] as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import r2_score,mean_squared_error
df = pd.read_csv('/home/ctbora/Downloads/[Link]') # Importing the data set
new_df = df[['X4 number of convenience stores','Y house price of unit area']]
X = [Link](new_df[['X4 number of convenience stores']]) # Storing into X the 'Engine HP' as [Link]
y = [Link](new_df[['Y house price of unit area']]) # Storing into y the 'MSRP' as [Link]
print([Link]) # Vewing the shape of X
print([Link]) # Vewing the shape of y
[Link](X,y,color="red") # Plot a graph X vs y
[Link]('X4 number of convenience stores vs. Y house price of unit area')
[Link]('X1 transaction date')
[Link]('Y house price of unit area')
[Link]()
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.333,random_state=15)
regressor = LinearRegression() # Creating a regressior
print("Training Set :")
print(X_train)
print("Test Set :")
print(X_test)

[Link](X_train,y_train) # Fiting the dataset into the model

[Link](X_test,y_test,color="green") # Plot a graph with X_test vs y_test
[Link](X_train,[Link](X_train),color="red",linewidth=3) # Regressior line showing
[Link]('Regression(Test Set)')
[Link]('X4 number of convenience stores')
[Link]('Y house price of unit area')
[Link]()
y_pred = [Link](X_test)
print('R2 score: %.2f' % r2_score(y_test,y_pred)) # Priniting R 2 Score
print('Mean Error :',mean_squared_error(y_test,y_pred)) # Priniting the mean error
3. Set A3 - Create ‘User’ Data set having 5 columns namely: User ID, Gender, Age,
EstimatedSalary and Purchased. Build a logistic regression model that can predict whether
on the given parameter a person will buy a car or not.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
import seaborn as sn
import [Link] as plt
data = pd.read_csv("/home/ctbora/Downloads/[Link]")
x = [Link][:, [2,3]].values
y = [Link][:, 4].values
print("x")
print(x)
print("y")
print(y)
x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.25,random_state=0)
logistic_regression= LogisticRegression()
logistic_regression.fit(x_train,y_train)
y_pred=logistic_regression.predict(x_test)
confusion_matrix = [Link](y_test, y_pred, rownames=['Actual'], colnames=['Predicted'])
[Link](confusion_matrix, annot=True)
print('Accuracy: ',metrics.accuracy_score(y_test, y_pred))
[Link]()
print (x_test)
print (y_pred)
4. Set B1 - Build a simple linear regression model for Fish Species Weight Prediction.
(download dataset [Link] )
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
import seaborn as sn
import [Link] as plt
data = pd.read_csv("/home/ctbora/Downloads/[Link]")
x = [Link][:, [1,3]].values
y = [Link][:,0 ].values
print("x")
print(x)
print("y")
print(y)
x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.5,random_state=0)
logistic_regression= LogisticRegression()
logistic_regression.fit(x_train,y_train)
y_pred=logistic_regression.predict(x_test)
confusion_matrix = [Link](y_test, y_pred, rownames=['Actual'], colnames=['Predicted'])
[Link](confusion_matrix, annot=True)
print('Accuracy: ',metrics.accuracy_score(y_test, y_pred))
[Link]()
print (x_test)
print (y_pred)
Assignment No: 2 - Frequent itemset and Association rule mining

1. Set A1 - Create the following dataset in python. Convert the categorical values into numeric
format. Apply the apriori algorithm on the above dataset to generate the frequent itemsets and
association rules. Repeat the process with different min_sup values

import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules
data= [['Bread','Milk'],
['Bread','Diaper','Beer','Eggs'],
['Milk','Diaper','Beer','Coke'],
['Bread','Milk','Diaper','Beer'],
['Bread','Milk','Diaper','Coke']
]

df = [Link](data)
print(df)
from [Link] import TransactionEncoder
te=TransactionEncoder()
te_array=[Link](data).transform(data)
df=[Link](te_array, columns=te.columns_)
print(df)
freq_items = apriori(df, min_support = 0.5, use_colnames = True)
print(freq_items)
rules = association_rules(freq_items, metric ='support', min_threshold=0.05)
rules = rules.sort_values(['support', 'confidence'], ascending =[False,False])
print(rules

2. Set A2 - Create your own transactions dataset and apply the above process on your dataset.

import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules
data = [['Sheldon', 'Penny', 'Amy', 'Penny', 'Raj', 'Sheldon'],
['male', 'female', 'female', 'female', 'male', 'male']]
print(data)
df = [Link](data)
print(df)
from [Link] import TransactionEncoder
te=TransactionEncoder()
te_array=[Link](data).transform(data)
df=[Link](te_array, columns=te.columns_)
print(df)
freq_items = apriori(df, min_support = 0.5, use_colnames = True)
print(freq_items)
rules = association_rules(freq_items, metric ='support', min_threshold=0.05)
rules = rules.sort_values(['support', 'confidence'], ascending =[False,False])
print(rules)
3. Set B1 - Download the Market basket dataset. Write a python program to read the dataset
and display its information. Preprocess the data (drop null values etc.) Convert the
categorical values into numeric format. Apply the apriori algorithm on the above dataset
to generate the frequent itemsets and association rules.

import pandas as pd
import io
from mlxtend.frequent_patterns import apriori, association_rules
from [Link] import TransactionEncoder

d = pd.read_csv('/home/ctbora/DataAnalytics/[Link]')
print(d)
d=[Link]("Hi");
print(d)

te=TransactionEncoder()
te_array=[Link](d).transform(d)
df=[Link](te_array, columns=te.columns_)
print(df)

freq_items = apriori(df, min_support = 0.00000001, use_colnames = True)

print(freq_items)

rules = association_rules(freq_items, metric ='support', min_threshold=0.05)

rules = rules.sort_values(['support', 'confidence'], ascending =[False,False])
print(rules)
Assignment No: 3 - Text and Social Media Analytics

1. Set A1 - Consider any text paragraph. Preprocess the text to remove any special characters
and digits. Generate the summary using extractive summarization process.

import warnings
[Link](action='ignore', category=FutureWarning)
import nltk
#[Link]('all')
#Preprocessing
import re
text="""
If98745345b43b54b bkb6jkb36k36b3j6h called with no arguments, ``download()`` will display an interactive
interface which can be used to download and install new packages.
If Tkinter is available, then a graphical interface will be shown,
otherwise a simple text interface will be provided.

Individual packages can be downloaded by calling the ``download()``

function with a single argument, giving the package identifier for the
package that should be downloaded:
"""
text = [Link](r'[[0-9]*]', ' ', text)
text = [Link](r's+', ' ', text)
formatted_text = [Link]('[^a-zA-Z]', ' ', text)
print(formatted_text)
from [Link] import stopwords
from [Link] import word_tokenize, sent_tokenize
stopWords = set([Link]("english"))
words = word_tokenize(formatted_text)
# Creating a frequency table of words
wordfreq = {}
for word in words:
if word in stopWords:
continue
if word in wordfreq:
wordfreq[word] += 1
else: wordfreq[word] = 1
#Compute the weighted frequencies
maximum_frequency = max([Link]())
for word in [Link]():
wordfreq[word] = (wordfreq[word]/maximum_frequency)
# Creating a dictionary to keep the score # of each sentence
sentences = sent_tokenize(text)
sentenceValue = {}
for sentence in sentences:
for word, freq in [Link]():
if word in [Link]():
if sentence in sentenceValue:
sentenceValue[sentence] += freq
else:sentenceValue[sentence] = freq
import heapq
summary = ''
summary_sentences = [Link](4, sentenceValue, key=[Link])
summary = ' '.join(summary_sentences)
print(summary)
2. Set A2 - Consider any text paragraph. Remove the stopwords. Tokenize the paragraph to
extract words and sentences. Calculate the word frequency distribution and plot the
frequencies. Plot the wordcloud of the text.

from [Link] import word_tokenize

from [Link] import stopwords
# Textual data to remove stopwords
paragraph_text="""Hello all, Welcome to Python Programming Academy. Python
Programming Academy is a nice platform to learn new programming skills. It is
difficult to get enrolled in this Academy."""
# Word Tokenization
tokenized_words=word_tokenize(paragraph_text)
# It will find the stowords in English language.
stop_words_data=set([Link]("english"))
# Create a stopwords list to filter it from original text
filtered_words_list=[]
for words in tokenized_words:
if words not in stop_words_data:
filtered_words_list.append(words)
print("Tokenized Words : \n",tokenized_words,"\n")
print("Filtered Words : \n",filtered_words_list,"\n")

3. Set A3 - Consider the following review messages. Perform sentiment analysis on the messages.
i. I purchased headphones online. I am very happy with the product.
ii. I saw the movie yesterday. The animation was really good but the script was ok.
iii. I enjoy listening to music
iv. I take a walk in the park everyday

import nltk
from [Link] import SentimentIntensityAnalyzer
vader_analyzer=SentimentIntensityAnalyzer()
text1="I purchased headphones online. I am very happy with the product."# The text is positive.
print(vader_analyzer.polarity_scores(text1))
text1="I saw the movie yesterday. The animation was really good but the script was ok."
print(vader_analyzer.polarity_scores(text1))
text1="I enjoy listening to music"
print(vader_analyzer.polarity_scores(text1))
text1="I take a walk in the park everyday"
print(vader_analyzer.polarity_scores(text1))
text1="Very Bad day"
print(vader_analyzer.polarity_scores(text1))
4. Set A4 - Perform text analytics on WhatsApp data :
Write a Python script for the following :
i. First Export the WhatsApp chat of any group. Read the exported “.txt” file using open()
and read() functions.
ii. Tokenize the read data into sentences and print it.
iii. Remove the stopwords from data and perform lemmatization.
iv. Plot the wordcloud for the given data.

import regex
import pandas as pd
import numpy as np
import emoji
from collections import Counter
import [Link] as plt
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
def date_time(s):
pattern = '^([0-9]+)(\/)([0-9]+)(\/)([0-9]+), ([0-9]+):([0-9]+)[ ]?(AM|PM|am|pm)? -'
result = [Link](pattern, s)
if result:
return True
return False

def find_author(s):
s = [Link](":")
if len(s)==2:
return True
else:
return False

def getDatapoint(line):
splitline = [Link](' - ')
dateTime = splitline[0]
date, time = [Link](", ")
message = " ".join(splitline[1:])
if find_author(message):
splitmessage = [Link](": ")
author = splitmessage[0]
message = " ".join(splitmessage[1:])
else:
author= None
return date, time, author, message
data = []
conversation = 'WhatsApp Chat with [Link](Comp)[Link]'
with open(conversation, encoding="utf-8") as fp:
[Link]()
messageBuffer = []
date, time, author = None, None, None
while True:
line = [Link]()
if not line:
break
line = [Link]()
if date_time(line):
if len(messageBuffer) > 0:
[Link]([date, time, author, ' '.join(messageBuffer)])
[Link]()
date, time, author, message = getDatapoint(line)
[Link](message)
else:
[Link](line)
df = [Link](data, columns=["Date", 'Time', 'Author', 'Message'])
df['Date'] = pd.to_datetime(df['Date'])
print(df)
print([Link]())
print([Link]())
import nltk
[Link]('wordnet')
# Lemmatization
from [Link] import WordNetLemmatizer
lemmatizer=WordNetLemmatizer()
word_text=message
print("Lemmatized Word : ",[Link](word_text,"v"))
from [Link] import stopwords
from [Link] import SentimentIntensityAnalyzer
from wordcloud import WordCloud, get_single_color_func
import [Link] as plt
import pandas as pd
import numpy as np
movies_reviews=[Link](str)
movies_reviews1=np.array_str(movies_reviews)
# It will find the stowords in English language.
stop_words_data=set([Link]("english"))
words=movies_reviews1.split()
final_data=[]
for w in words:
if w not in final_data:
final_data.append(w)
# Create dictionaries to store positive and negative words with polarity.
positive_words=dict()
negative_words=dict()
# Create lists to store positive and negative words without polarity.
positive=[]
negative=[]
# Sentiment Analysis
sentiment_analyzer=SentimentIntensityAnalyzer()
for i in words:
if not [Link]() in stop_words_data: # It will remove stopwords.
polarity=sentiment_analyzer.polarity_scores(i)
if polarity['compound']>=0.05: # Positive Sentiment
positive_words[i]=polarity['compound']
if polarity['compound']<=-0.05: # Negative Sentiment
negative_words[i]=polarity['compound']
# Append the positive and negative words from dictionaries to lists [Link][] and negative[]
for key,value in positive_words.items():
[Link](key)
for key,value in negative_words.items():
[Link](key)
# Create a dictionary to mention the colors : green for positive and red for negative
coloured_words={"green":positive,"red":negative}
# Implement separate colour assignments
class ColourAssignment(object):
# Functions to give different colours on the basis of sentiments.
def __init__(self,coloured_words,default):
self.coloured_words=[
(get_single_color_func(colour),set(words))
for (colour,words) in coloured_words.items()]
[Link]=get_single_color_func(default)

def get_colour(self,word):
try:
colour=next(
colour for (colour,words) in self.coloured_words
if word in words)
except StopIteration:
colour=[Link]
return colour
def __call__(self,word, **kwargs):
return self.get_colour(word) (word, **kwargs)
word_cloud=WordCloud(collocations=False,background_color='black').generate(movies_reviews1)
# Neutral words will be visible as black
group_color=ColourAssignment(coloured_words, 'white')
#word_cloud.recolor(color_func=group_color)
[Link]()
[Link](word_cloud, interpolation="bilinear")
[Link]("off")
[Link]()

Common questions