Dominant colors in
images
C L U S TE R AN ALYS I S I N P YTH ON
Shaumik Daityari
Business Analyst
Dominant colors in images
All images consist of pixels
Each pixel has three values: Red, Green
and Blue
Pixel color: combination of these RGB
values
Perform k-means on standardized RGB Source
values to nd cluster centers
Uses: Identifying features in satellite images
CLUSTER ANALYSIS IN PYTHON
Feature identification in satellite images
Source
CLUSTER ANALYSIS IN PYTHON
Tools to find dominant colors
Convert image to pixels: [Link]
Display colors of cluster centers: [Link]
CLUSTER ANALYSIS IN PYTHON
CLUSTER ANALYSIS IN PYTHON
Convert image to RGB matrix
import [Link] as img
image = [Link]('[Link]')
[Link]
(475, 764, 3)
r = []
g = []
b = []
for row in image:
for pixel in row:
# A pixel contains RGB values
temp_r, temp_g, temp_b = pixel
[Link](temp_r)
[Link](temp_g)
[Link](temp_b)
CLUSTER ANALYSIS IN PYTHON
Data frame with RGB values
pixels = [Link]({'red': r,
'blue': b,
'green': g})
[Link]()
red blue green
252 255 252
75 103 81
... ... ...
CLUSTER ANALYSIS IN PYTHON
Create an elbow plot
distortions = []
num_clusters = range(1, 11)
# Create a list of distortions from the kmeans method
for i in num_clusters:
cluster_centers, _ = kmeans(pixels[['scaled_red', 'scaled_blue',
'scaled_green']], i)
[Link](distortion)
# Create a data frame with two lists - number of clusters and distortions
elbow_plot = [Link]({'num_clusters': num_clusters,
'distortions': distortions})
# Creat a line plot of num_clusters and distortions
[Link](x='num_clusters', y='distortions', data = elbow_plot)
[Link](num_clusters)
[Link]()
CLUSTER ANALYSIS IN PYTHON
Elbow plot
CLUSTER ANALYSIS IN PYTHON
Find dominant colors
cluster_centers, _ = kmeans(pixels[['scaled_red', 'scaled_blue',
'scaled_green']], 2)
colors = []
# Find Standard Deviations
r_std, g_std, b_std = pixels[['red', 'blue', 'green']].std()
# Scale actual RGB values in range of 0-1
for cluster_center in cluster_centers:
scaled_r, scaled_g, scaled_b = cluster_center
[Link]((
scaled_r * r_std/255,
scaled_g * g_std/255,
scaled_b * b_std/255
))
CLUSTER ANALYSIS IN PYTHON
Display dominant colors
#Dimensions: 2 x 3 (N X 3 matrix)
print(colors)
[(0.08192923122023911, 0.34205845943857993, 0.2824002984155429),
(0.893281510956742, 0.899818770315129, 0.8979114272960784)]
#Dimensions: 1 x 2 x 3 (1 X N x 3 matrix)
[Link]([colors])
[Link]()
CLUSTER ANALYSIS IN PYTHON
Next up: exercises
C L U S TE R AN ALYS I S I N P YTH ON
Document
clustering
C L U S TE R AN ALYS I S I N P YTH ON
Shaumik Daityari
Business Analyst
Document clustering: concepts
1. Clean data before processing
2. Determine the importance of the terms in a document (in TF-IDF matrix)
3. Cluster the TF-IDF matrix
4. Find top terms, documents in each cluster
CLUSTER ANALYSIS IN PYTHON
Clean and tokenize data
Convert text into smaller parts called tokens, clean data for processing
from [Link] import word_tokenize
import re
def remove_noise(text, stop_words = []):
tokens = word_tokenize(text)
cleaned_tokens = []
for token in tokens:
token = [Link]('[^A-Za-z0-9]+', '', token)
if len(token) > 1 and [Link]() not in stop_words:
# Get lowercase
cleaned_tokens.append([Link]())
return cleaned_tokens
remove_noise("It is lovely weather we are having.
I hope the weather continues.")
['lovely', 'weather', 'hope', 'weather', 'continues']
CLUSTER ANALYSIS IN PYTHON
Document term matrix and sparse matrices
Document term matrix formed Sparse matrix is created
Most elements in matrix are zeros
Source
Source
CLUSTER ANALYSIS IN PYTHON
TF-IDF (Term Frequency - Inverse Document
Frequency)
A weighted measure: evaluate how important a word is to a document in a collection
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=50,
min_df=0.2, tokenizer=remove_noise)
tfidf_matrix = tfidf_vectorizer.fit_transform(data)
CLUSTER ANALYSIS IN PYTHON
Clustering with sparse matrix
kmeans() in SciPy does not support sparse matrices
Use .todense() to convert to a matrix
cluster_centers, distortion = kmeans(tfidf_matrix.todense(), num_clusters)
CLUSTER ANALYSIS IN PYTHON
Top terms per cluster
Cluster centers: lists with a size equal to the number of terms
Each value in the cluster center is its importance
Create a dictionary and print top terms
terms = tfidf_vectorizer.get_feature_names()
for i in range(num_clusters):
center_terms = dict(zip(terms, list(cluster_centers[i])))
sorted_terms = sorted(center_terms, key=center_terms.get, reverse=True)
print(sorted_terms[:3])
['room', 'hotel', 'staff']
['bad', 'location', 'breakfast']
CLUSTER ANALYSIS IN PYTHON
More considerations
Work with hyperlinks, emoticons etc.
Normalize words (run, ran, running -> run)
.todense() may not work with large datasets
CLUSTER ANALYSIS IN PYTHON
Next up: exercises!
C L U S TE R AN ALYS I S I N P YTH ON
Clustering with
multiple features
C L U S TE R AN ALYS I S I N P YTH ON
Shaumik Daityari
Business Analyst
Basic checks
# Cluster centers
print([Link]('cluster_labels')[['scaled_heading_accuracy',
'scaled_volleys', 'scaled_finishing']].mean())
cluster_labels scaled_heading_accuracy scaled_volleys scaled_ nishing
0 3.21 2.83 2.76
1 0.71 0.64 0.58
# Cluster sizes
print([Link]('cluster_labels')['ID'].count())
cluster_labels count
0 886
CLUSTER ANALYSIS IN PYTHON
Visualizations
Visualize cluster centers
Visualize other variables for each cluster
# Plot cluster centers
[Link]('cluster_labels') \
[scaled_features].mean()
.plot(kind='bar')
[Link]()
CLUSTER ANALYSIS IN PYTHON
Top items in clusters
# Get the name column of top 5 players in each cluster
for cluster in fifa['cluster_labels'].unique():
print(cluster, fifa[fifa['cluster_labels'] == cluster]['name'].values[:5])
Cluster Label Top Players
0 ['Cristiano Ronaldo' 'L. Messi' 'Neymar' 'L. Suárez' 'R. Lewandowski']
1 ['M. Neuer' 'De Gea' 'G. Bu on' 'T. Courtois' 'H. Lloris']
CLUSTER ANALYSIS IN PYTHON
Feature reduction
Factor analysis
Multidimensional scaling
CLUSTER ANALYSIS IN PYTHON
Final exercises!
# Create centroids with kmeans for 2 clusters
cluster_centers,_C L U S T E R A N A L Y S I2)S
= kmeans(fifa[scaled_features], IN P YTH ON
# Assign cluster labels and print cluster centers
fifa['cluster_labels'], _ = vq(fifa[scaled_features], cluster_centers)
print([Link]('cluster_labels')[scaled_features].mean())
# Plot cluster centers to visualize clusters
[Link]('cluster_labels')
[scaled_features].mean().plot(legend=True, kind='bar')
[Link]()
# Get the name column of first 5 players in each cluster
for cluster in fifa['cluster_labels'].unique():
print(cluster, fifa[fifa['cluster_labels'] == cluster]['name'].values[:5])
Farewell!
C L U S TE R AN ALYS I S I N P YTH ON
Shaumik Daityari
Business Analyst
What comes next?
Clustering is one of the exploratory steps
More courses on DataCamp
Practice, practice, practice!
CLUSTER ANALYSIS IN PYTHON
Until next time
C L U S TE R AN ALYS I S I N P YTH ON