Scribd
Upload
5 views1 page

DBSCAN Clustering with PySpark

The document outlines a process for clustering customer data using K-Means in PySpark. It includes steps for reading data, assembling features, scaling the data, applying K-Means clustering, and evaluating the model using silhouette scores. Finally, it visualizes the silhouette scores to determine the optimal number of clusters.

Uploaded by

namyachawla8
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views1 page

DBSCAN Clustering with PySpark

The document outlines a process for clustering customer data using K-Means in PySpark. It includes steps for reading data, assembling features, scaling the data, applying K-Means clustering, and evaluating the model using silhouette scores. Finally, it visualizes the silhouette scores to determine the optimal number of clusters.

Uploaded by

namyachawla8
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

In [ ]:

from [Link] import SparkSession


spark = [Link](‘Clustering using K-Means’).getOrCreate()
data_customer=[Link]('[Link]', header=True, inferSchema=True)
data_customer.printSchema()

In [ ]:
from [Link] import VectorAssembler
data_customer.columns
assemble=VectorAssembler(inputCols=['PDPcountperday','CheckoutHistory','Booked Revnue','B
randname','Styletype'], outputCol='features')
assembled_data=[Link](data_customer)
assembled_data.show(2)

In [ ]:

from [Link] import StandardScaler


scale=StandardScaler(inputCol='features',outputCol='standardized')
data_scale=[Link](assembled_data)
data_scale_output=data_scale.transform(assembled_data)
data_scale_output.show(2)

In [ ]:
from [Link] import KMeans
from [Link] import ClusteringEvaluator
silhouette_score=[]
evaluator = ClusteringEvaluator(predictionCol='prediction', featuresCol='standardized', \
metricName='silhouette', distanceMeasure='squaredEuclide
an')
for i in range(2,10):

KMeans_algo=KMeans(featuresCol='standardized', k=i)

KMeans_fit=KMeans_algo.fit(data_scale_output)

output=KMeans_fit.transform(data_scale_output)

score=[Link](output)

silhouette_score.append(score)

print("Silhouette Score:",score)

In [ ]:

#Visualizing the silhouette scores in a plot


import [Link] as plt
fig, ax = [Link](1,1, figsize =(8,6))
[Link](range(2,10),silhouette_score)
ax.set_xlabel(‘k’)
ax.set_ylabel(‘cost’)

You might also like