Importation et Configuration

Nous allons importer les bibliothèques nécessaires et créer une session Spark.

from [Link] import SparkSession

from [Link] import StructType, StructField, IntegerType,
StringType, DoubleType, FloatType, ArrayType
from [Link] import col, regexp_replace, avg, max, min,
sum, when, count, desc, isnan
spark = [Link] \
.appName("orange pfe") \
.getOrCreate()

spark

<[Link] at 0x6b4e888>

schema = StructType([
StructField("num_crypted", StringType(), True),
StructField("class_client", IntegerType(), True),
StructField("subclass_client", IntegerType(), True),
StructField("class_service", IntegerType(), True),
StructField("life_time", IntegerType(), True),
StructField("type_sim", IntegerType(), True),
StructField("nb_transaction", IntegerType(), True),
StructField("sum_achat", StringType(), True)
])

# Read the CSV file into a DataFrame

df = [Link]("Desktop/PFE/Services/[Link]",
schema=schema, header=False, sep=',')

# Remplir les valeurs manquantes dans les colonnes de type string

string_columns = [[Link] for field in [Link] if
isinstance([Link], StringType)]
for col_name in string_columns:
df = [Link]({col_name: 'unknown'}) # Remplacer les valeurs
manquantes par 'unknown'

# Remplacer les zéros et NaN par la moyenne ou une valeur spécifique

dans les colonnes numériques
numeric_columns = [[Link] for field in [Link] if
isinstance([Link], (IntegerType, DoubleType, FloatType))]
for col_name in numeric_columns:
mean_value = [Link](col_name).[Link]().agg({col_name:
'mean'}).collect()[0][0]
df = [Link](col_name, when((col(col_name) == 0) |
isnan(col(col_name)), mean_value).otherwise(col(col_name)))

# Remplir les valeurs anormales dans les colonnes de type array

array_columns = [[Link] for field in [Link] if
isinstance([Link], ArrayType)]
for col_name in array_columns:
# Remplacer les zéros et NaN dans les arrays par None
df = [Link](col_name, when(array_contains(col(col_name), 0)
| array_contains(col(col_name), float('nan')),
lit(None)).otherwise(col(col_name)))

# Suppression des doublons:

df = [Link]()
[Link]()

+--------------------+------------+---------------+-------------
+---------+--------+--------------+-----------------+
| num_crypted|class_client|subclass_client|class_service|
life_time|type_sim|nb_transaction| sum_achat|
+--------------------+------------+---------------+-------------
+---------+--------+--------------+-----------------+
|sMpp-SPJ3ntR6gwVx...| 70.0| 124.0| 1.0|
47.0| 2.0| 14.0| 0,01860905175236|
|s3BXUpS8y13Ndw-4l...| 70.0| 124.0| 1.0|
98.0| 1.0| 3.0|0,007096672450206|
|sPRMtjQIwjc9gaEyo...| 70.0| 107.0| 2.0|
97.0| 1.0| 42.0|0,033117804767628|
|s4dty8t1voRaK26tS...| 70.0| 108.0| 1.0|
85.0| 2.0| 3.0|0,004258003357523|
|sHYErfG0V5Mz2C4Gd...| 70.0| 106.0| 1.0|
2.0| 2.0| 2.0| 0,00378489193591|
|sJcB3ghy-_rXyw3kU...| 70.0| 149.0| 1.0|
27.0| 2.0| 5.0|0,006465856971165|
|sN95eGT40RNqPSIDM...| 70.0| 108.0| 1.0|
90.0| 2.0| 2.0| 0,06308153289072|
|s_lLRC-rom-lpQHoR...| 95.0| 106.0| 1.0|
83.0| 2.0| 2.0|0,002838668905015|
|scShyyISHjAZi8-j1...| 70.0| 106.0| 1.0|
34.0| 2.0| 5.0|0,006150449419312|
|s4ncCIj4xmG6kHCM8...| 70.0| 108.0| 1.0|
102.0| 2.0| 17.0| 0,03926825418694|
|ssHO6SlHyi0fsttq6...| 95.0| 106.0| 1.0|
27.0| 2.0| 4.0|0,008516006902714|
|sYHzC8oJeyQiUxUbr...| 70.0| 124.0| 1.0|
68.0| 2.0| 2.0| 0,00378489193591|
|s8P2ZU86X8S3UwPUb...| 70.0| 106.0| 1.0|
15.0| 2.0| 31.0|0,053461598149014|
|sl_BWYrdjdafuZ6EV...| 70.0| 20.0| 1.0|
107.0| 2.0| 1.0|0,001419334452508|
|sJ6AjDgT6EEsUXZG2...| 81.0| 124.0| 1.0|
65.0| 2.0| 1.0|0,001419334452508|
|szOqfiSwSonBdhhTA...| 70.0| 124.0| 1.0|
70.0| 2.0| 24.0|0,150449455906833|
|sZxtoMoqoJ1zk65oj...| 70.0| 106.0| 1.0|
62.0| 2.0| 3.0| 0,00513325973157|
|sG556qd0kNgoC2e5E...| 70.0| 106.0| 1.0|
10.0| 2.0| 3.0|0,006150449419312|
|s5fPNQ8t-bw12vBWW...| 70.0| 108.0| 1.0|
99.0| 1.0| 8.0|0,044551332604071|
|s0yu8uU19_LQW7pA5...| 70.0| 106.0| 1.0|
1.0| 2.0| 3.0|0,005992745643385|
+--------------------+------------+---------------+-------------
+---------+--------+--------------+-----------------+
only showing top 20 rows

# Traitement des valeurs manquantes:

# Supprimer les lignes avec des valeurs manquantes dans des colonnes
spécifiques
df = [Link](subset=['num_crypted'])

# Filtrage : Sélectionner les lignes où class_client est supérieur à

100
df_filtered = [Link](col("class_client") > 100)

# Remplacement de valeurs : Remplacer les valeurs nulles dans

class_service par la moyenne de la colonne
average_class_service =
df_filtered.select(avg(col("class_service"))).collect()[0][0]
df_filtered = df_filtered.fillna({"class_service":
average_class_service})

# Création de nouvelles colonnes : Ajouter une colonne calculée pour

la moyenne de life_time et type_sim
df_filtered = df_filtered.withColumn("avg_life_time_type_sim",
(col("life_time") + col("type_sim")) / 2)

# Détection des anomalies : Marquer les valeurs extrêmes dans

sum_achat
threshold = 0.02 # Seuil arbitraire pour les anomalies
df_filtered = df_filtered.withColumn("anomaly", when(col("sum_achat")
> threshold, 1).otherwise(0))

# Agrégation : Calculer des statistiques sur sum_achat

df_agg = df_filtered.agg(
sum("sum_achat").alias("sum_sum_achat"),
avg("sum_achat").alias("avg_sum_achat"),
max("sum_achat").alias("max_sum_achat"),
min("sum_achat").alias("min_sum_achat")
)

# Afficher les résultats de l'agrégation

print("Aggregated results:")
df_agg.show()

Aggregated results:
+-------------+-------------+-------------+-------------+
|sum_sum_achat|avg_sum_achat|max_sum_achat|min_sum_achat|
+-------------+-------------+-------------+-------------+
| 0.0| 0.0| unknown| 0|
+-------------+-------------+-------------+-------------+

# Segmentation des utilisateurs par subclass_client

df_segmented = df_filtered.groupBy("subclass_client").agg(
count("num_crypted").alias("user_count"),
avg("sum_achat").alias("avg_sum_achat")
)

# Trier par le nombre d'utilisateurs dans chaque segment

df_segmented = df_segmented.orderBy(desc("user_count"))

# Afficher les segments d'utilisateurs

print("User segments:")
df_segmented.show()

User segments:
+---------------+----------+-------------+
|subclass_client|user_count|avg_sum_achat|
+---------------+----------+-------------+
| 150.0| 801| null|
| 137.0| 683| 0.0|
| 235.0| 313| 0.0|
| 113.0| 115| null|
| 124.0| 37| null|
| 106.0| 32| null|
| 36.0| 22| null|
| 34.0| 16| null|
| null| 4| null|
| 126.0| 2| null|
| 102.0| 1| null|
| 33.0| 1| null|
| 93.0| 1| null|
| 105.0| 1| null|
| 61.0| 1| null|
| 57.0| 1| null|
| 11.0| 1| null|
| 341.0| 1| null|
| 119.0| 1| null|
| 127.0| 1| null|
+---------------+----------+-------------+
only showing top 20 rows

EDA
[Link]()

+--------------------+------------+---------------+-------------
+---------+--------+--------------+-----------------+
| num_crypted|class_client|subclass_client|class_service|
life_time|type_sim|nb_transaction| sum_achat|
+--------------------+------------+---------------+-------------
+---------+--------+--------------+-----------------+
|sMpp-SPJ3ntR6gwVx...| 70.0| 124.0| 1.0|
47.0| 2.0| 14.0| 0,01860905175236|
|s3BXUpS8y13Ndw-4l...| 70.0| 124.0| 1.0|
98.0| 1.0| 3.0|0,007096672450206|
|sPRMtjQIwjc9gaEyo...| 70.0| 107.0| 2.0|
97.0| 1.0| 42.0|0,033117804767628|
|s4dty8t1voRaK26tS...| 70.0| 108.0| 1.0|
85.0| 2.0| 3.0|0,004258003357523|
|sHYErfG0V5Mz2C4Gd...| 70.0| 106.0| 1.0|
2.0| 2.0| 2.0| 0,00378489193591|
|sJcB3ghy-_rXyw3kU...| 70.0| 149.0| 1.0|
27.0| 2.0| 5.0|0,006465856971165|
|sN95eGT40RNqPSIDM...| 70.0| 108.0| 1.0|
90.0| 2.0| 2.0| 0,06308153289072|
|s_lLRC-rom-lpQHoR...| 95.0| 106.0| 1.0|
83.0| 2.0| 2.0|0,002838668905015|
|scShyyISHjAZi8-j1...| 70.0| 106.0| 1.0|
34.0| 2.0| 5.0|0,006150449419312|
|s4ncCIj4xmG6kHCM8...| 70.0| 108.0| 1.0|
102.0| 2.0| 17.0| 0,03926825418694|
|ssHO6SlHyi0fsttq6...| 95.0| 106.0| 1.0|
27.0| 2.0| 4.0|0,008516006902714|
|sYHzC8oJeyQiUxUbr...| 70.0| 124.0| 1.0|
68.0| 2.0| 2.0| 0,00378489193591|
|s8P2ZU86X8S3UwPUb...| 70.0| 106.0| 1.0|
15.0| 2.0| 31.0|0,053461598149014|
|sl_BWYrdjdafuZ6EV...| 70.0| 20.0| 1.0|
107.0| 2.0| 1.0|0,001419334452508|
|sJ6AjDgT6EEsUXZG2...| 81.0| 124.0| 1.0|
65.0| 2.0| 1.0|0,001419334452508|
|szOqfiSwSonBdhhTA...| 70.0| 124.0| 1.0|
70.0| 2.0| 24.0|0,150449455906833|
|sZxtoMoqoJ1zk65oj...| 70.0| 106.0| 1.0|
62.0| 2.0| 3.0| 0,00513325973157|
|sG556qd0kNgoC2e5E...| 70.0| 106.0| 1.0|
10.0| 2.0| 3.0|0,006150449419312|
|s5fPNQ8t-bw12vBWW...| 70.0| 108.0| 1.0|
99.0| 1.0| 8.0|0,044551332604071|
|s0yu8uU19_LQW7pA5...| 70.0| 106.0| 1.0|
1.0| 2.0| 3.0|0,005992745643385|
+--------------------+------------+---------------+-------------
+---------+--------+--------------+-----------------+
only showing top 20 rows

# drop title row

df = [Link](df.num_crypted != "Achat Service OTN")
#Number of rows
total_count = [Link]()
total_count

149700

from [Link] import countDistinct

unique_counts = [Link]([countDistinct(col(c)).alias(c) for c in

[Link]])
unique_counts.show()

+-----------+------------+---------------+-------------+---------
+--------+--------------+---------+
|num_crypted|class_client|subclass_client|class_service|life_time|
type_sim|nb_transaction|sum_achat|
+-----------+------------+---------------+-------------+---------
+--------+--------------+---------+
| 147151| 12| 58| 5| 154|
2| 145| 16251|
+-----------+------------+---------------+-------------+---------
+--------+--------------+---------+

[Link]().show() # Statistiques descriptives

+-------+--------------------+-----------------+------------------
+------------------+-----------------+------------------
+------------------+--------------------+
|summary| num_crypted| class_client| subclass_client|
class_service| life_time| type_sim| nb_transaction|
sum_achat|
+-------+--------------------+-----------------+------------------
+------------------+-----------------+------------------
+------------------+--------------------+
| count| 149700| 149699| 149549|
149699| 149698| 149699| 148199|
149700|
| mean| null|72.80275753345046|121.27822319106113|
1.0619443015651406|42.95799543080068| 1.927360904214457|
5.6130068354037475|0.021739130434782608|
| stddev| null|7.741232839293571| 41.03937782912098|
0.2961201559040936|38.06135589803621|0.2595440377650119|
9.128476387018598| 0.14744195615489714|
| min|---ZgNYsEeuL_qp5s...| 70.0| 11.0|
1.0| 1.0| 1.0| 1.0| -
0,13625610743506|
| max|zzyZmxQGvcbFndkp-...| 103.0| 341.0|
5.0| 154.0| 2.0| 1344.0|
unknown|
+-------+--------------------+-----------------+------------------
+------------------+-----------------+------------------
+------------------+--------------------+

Columns classification :
We concluded that :

• Categorical columns : class_client, subclass_client, class_service, type_sim

• Numerical columns : life_time, nb_transaction, sum_achat
• ID columns : num_crypted

EDA Categorical columns

class_client column
# Classes distribution
[Link]('class_client').count().orderBy('count',
ascending=False).show()

+------------+------+
|class_client| count|
+------------+------+
| 70.0|128579|
| 95.0| 10575|
| 81.0| 4732|
| 102.0| 2011|
| 96.0| 1270|
| 72.0| 1133|
| 71.0| 1032|
| 73.0| 282|
| 82.0| 45|
| 103.0| 27|
| 83.0| 10|
| 92.0| 3|
| null| 1|
+------------+------+

# Classes distribution pourcentage

[Link]('class_client').count().withColumn('percentage',
(col('count') / total_count) * 100).orderBy('percentage',
ascending=False).show()

+------------+------+--------------------+
|class_client| count| percentage|
+------------+------+--------------------+
| 70.0|128579| 85.89111556446227|
| 95.0| 10575| 7.064128256513026|
| 81.0| 4732| 3.1609886439545756|
| 102.0| 2011| 1.3433533734134937|
| 96.0| 1270| 0.8483633934535738|
| 72.0| 1133| 0.7568470273881096|
| 71.0| 1032| 0.6893787575150301|
| 73.0| 282| 0.18837675350701402|
| 82.0| 45| 0.03006012024048096|
| 103.0| 27|0.018036072144288578|
| 83.0| 10|0.006680026720106881|
| 92.0| 3|0.002004008016032064|
| null| 1| 6.68002672010688E-4|
+------------+------+--------------------+

We concluded that class 70 is dominant amongst the class_client classes distribution with an
occurance of 85%

[Link](df['class_client'].isNull()).count()

subclass_client cloumn
# Classes distribution
[Link]('subclass_client').count().orderBy('count',
ascending=False).show()

+---------------+-----+
|subclass_client|count|
+---------------+-----+
| 106.0|59954|
| 124.0|34510|
| 108.0| 9994|
| 137.0| 7936|
| 109.0| 7884|
| 116.0| 3863|
| 127.0| 3746|
| 341.0| 3047|
| 126.0| 2674|
| 149.0| 2531|
| 111.0| 2053|
| 113.0| 1657|
| 147.0| 1517|
| 107.0| 1104|
| 105.0| 1085|
| 338.0| 956|
| 150.0| 801|
| 85.0| 593|
| 20.0| 488|
| 98.0| 402|
+---------------+-----+
only showing top 20 rows

# Classes distribution pourcentage

[Link]('subclass_client').count().withColumn('percentage',
(col('count') / total_count) * 100).orderBy('percentage',
ascending=False).show()

+---------------+-----+------------------+
|subclass_client|count| percentage|
+---------------+-----+------------------+
| 106.0|59954| 40.04943219772879|
| 124.0|34510|23.052772211088847|
| 108.0| 9994| 6.676018704074817|
| 137.0| 7936| 5.30126920507682|
| 109.0| 7884| 5.266533066132265|
| 116.0| 3863|2.5804943219772882|
| 127.0| 3746|2.5023380093520373|
| 341.0| 3047|2.0354041416165662|
| 126.0| 2674| 1.78623914495658|
| 149.0| 2531|1.6907147628590513|
| 111.0| 2053|1.3714094856379426|
| 113.0| 1657|1.1068804275217101|
| 147.0| 1517|1.0133600534402138|
| 107.0| 1104|0.7374749498997996|
| 105.0| 1085|0.7247828991315965|
| 338.0| 956|0.6386105544422178|
| 150.0| 801|0.5350701402805611|
| 85.0| 593| 0.396125584502338|
| 20.0| 488|0.3259853039412158|
| 98.0| 402|0.2685370741482966|
+---------------+-----+------------------+
only showing top 20 rows

We concluded that subclass 106 and 124 are dominant amongst the subclass_client classes
distribution with an occurance of 40% and 22% respectively.

[Link](df['subclass_client'].isNull()).count()

151

subclass_client distribution by class_client

[Link]('class_client').agg(countDistinct('subclass_client').alias(
'unique_subclass_client_count')).show()
+------------+----------------------------+
|class_client|unique_subclass_client_count|
+------------+----------------------------+
| 70.0| 42|
| 96.0| 29|
| null| 0|
| 83.0| 1|
| 71.0| 14|
| 103.0| 1|
| 72.0| 10|
| 82.0| 13|
| 81.0| 20|
| 73.0| 28|
| 95.0| 21|
| 102.0| 22|
| 92.0| 3|
+------------+----------------------------+

We noticed that the dominant (over 85% of rows) class in class_client 70 has 42 subclasses / 57

#subclass_client class distribution in the dominant class_client class

(70)
class_70_df = [Link](df['class_client'] == 70)
total_class_70_count = class_70_df.count()
distinct_counts = class_70_df.groupBy('subclass_client').count()
distinct_counts_with_percentage =
distinct_counts.withColumn('percentage', (col('count') /
total_class_70_count) * 100)
distinct_counts_with_percentage_sorted =
distinct_counts_with_percentage.orderBy('percentage', ascending=False)
distinct_counts_with_percentage_sorted.show()

+---------------+-----+-------------------+
|subclass_client|count| percentage|
+---------------+-----+-------------------+
| 106.0|55501| 43.16490251129655|
| 124.0|33045| 25.700153213199666|
| 108.0| 9733| 7.569665341929864|
| 109.0| 7643| 5.944205507897869|
| 116.0| 3755| 2.920383577411552|
| 127.0| 3491| 2.715062335218037|
| 126.0| 2519| 1.9591068525964581|
| 149.0| 2503| 1.9466631409483663|
| 341.0| 2158| 1.6783456085363861|
| 111.0| 1909| 1.484690346012957|
| 107.0| 1048| 0.8150631129500152|
| 105.0| 931| 0.7240684715233436|
| 338.0| 919| 0.7147356877872747|
| 85.0| 583| 0.4534177431773462|
| 20.0| 481| 0.3740890814207608|
| 98.0| 379| 0.2947604196641753|
| 339.0| 324|0.25198516087385964|
| 119.0| 293| 0.2278754695556817|
| 11.0| 232|0.18043381889733162|
| 143.0| 167|0.12988124032695852|
+---------------+-----+-------------------+
only showing top 20 rows

We concluded that there is no clear dominant subclass_client class to use as a replacement for
the missing subclass_client values. Nor is there any strong correlation between class_client and
subclass_client that can help us fill those missing values. Seeing that 146 (number of rows of
missing subclass_client) is negligeable compared with the total number of rows (approxiamtely
150 000) we decided that these rows will be simply dropped in the data cleaning phase.

class_service column
# Classes distribution
[Link]('class_service').count().orderBy('count',
ascending=False).show()

+-------------+------+
|class_service| count|
+-------------+------+
| 1.0|141940|
| 2.0| 6934|
| 4.0| 667|
| 3.0| 147|
| 5.0| 11|
| null| 1|
+-------------+------+

# Classes distribution pourcentage

[Link]('class_service').count().withColumn('percentage',
(col('count') / total_count) * 100).orderBy('percentage',
ascending=False).show()

+-------------+------+--------------------+
|class_service| count| percentage|
+-------------+------+--------------------+
| 1.0|141940| 94.81629926519706|
| 2.0| 6934| 4.631930527722111|
| 4.0| 667| 0.4455577822311289|
| 3.0| 147| 0.09819639278557116|
| 5.0| 11|0.007348029392117568|
| null| 1| 6.68002672010688E-4|
+-------------+------+--------------------+

We concluded that class 1 is dominant amongst the class_service classes distribution with an
occurance of 94%

[Link](df['class_service'].isNull()).count()

type_sim column
# Classes distribution
[Link]('type_sim').count().orderBy('count',
ascending=False).show()

+--------+------+
|type_sim| count|
+--------+------+
| 2.0|138825|
| 1.0| 10874|
| null| 1|
+--------+------+

# Classes distribution pourcentage

[Link]('type_sim').count().withColumn('percentage', (col('count')
/ total_count) * 100).orderBy('percentage', ascending=False).show()

+--------+------+-------------------+
|type_sim| count| percentage|
+--------+------+-------------------+
| 2.0|138825| 92.73547094188376|
| 1.0| 10874| 7.263861055444222|
| null| 1|6.68002672010688E-4|
+--------+------+-------------------+

We concluded that class 2 is dominant amongst the type_sim classes distribution with an
occurance of 92%

[Link](df['type_sim'].isNull()).count()

1
EDA Numerical Columns
life_time column
[Link]('life_time').describe().show()

+-------+-----------------+
|summary| life_time|
+-------+-----------------+
| count| 149698|
| mean|42.95799543080068|
| stddev|38.06135589803621|
| min| 1.0|
| max| 154.0|
+-------+-----------------+

The average life_time of a given sim is 42 months.

The standard deviation is approximately 38.06 months.

This is considered quite high since the mean life-time is 42 months.

This high value indicates a significant spread in the life_time values, meaning there is
considerable variability around the mean.

Outliers analysis
pdf = [Link]()

import [Link] as plt

pdf = [Link](subset=['life_time'])
[Link](figsize=(10, 6))
[Link](pdf['life_time'], vert=False)
[Link]('Life Time (days)')
[Link]('Box Plot of Life Time')
[Link]()
The distribution of life_time shows significant variability.

The presence of high values (e.g., 154 days) suggests potential outliers or a long-tail
distribution.n

threshold = pdf['life_time'].quantile(0.90)

# Filter the data to include only the tail end (values greater than
the threshold)
tail_data = pdf[pdf['life_time'] > threshold]

# Plot the tail distribution

[Link](figsize=(10, 6))
[Link](tail_data['life_time'], bins=10, edgecolor='k', alpha=0.7)
[Link]('Life Time (days)')
[Link]('Frequency')
[Link]('Tail Plot of Life Time')
[Link]()
The tail plot shows a continuous distribution without significant gaps, suggesting no distinct
outliers in the life_time data.

Missing values
from [Link] import isnan, col
[Link](col('life_time').isNull() | isnan(col('life_time'))).count()

There seems to be 1 only row with a missing life time, we'll be dropping it

nb_transaction Column
[Link]('nb_transaction').describe().show()

+-------+------------------+
|summary| nb_transaction|
+-------+------------------+
| count| 148199|
| mean|5.6130068354037475|
| stddev| 9.128476387018598|
| min| 1.0|
| max| 1344.0|
+-------+------------------+

The average nb_transaction is 5.6 transactions.

The standard deviation is approximately 9.1 transaction.

This is considered quite high since the mean nb_transaction is 5.6.

This high value indicates a significant spread in the nb_transaction values, meaning there is
considerable variability around the mean.

Outliers analysis
import [Link] as plt
pdf = [Link](subset=['nb_transaction'])
[Link](figsize=(10, 6))
[Link](pdf['nb_transaction'], vert=False)
[Link]('Number of transactions')
[Link]('Box Plot of Number of transactions')
[Link]()

The box plot indicates that some of the transaction counts display as a discontinuous tail due to
extreme outliers. The Q1-Q3 box is barely visible because of these outliers. We conclude that we
will cut off the data at transaction counts less than 200 and redo the box plot to see if further
reduction is needed.

The min of nb-transaction is 1.

The max of nb_transaction is 1344.

The max of 1344 is significantly greater than the mean + standard_deviation of nb_transaction
(5.6 + 9.1).

There are two possible explanation for this :

• There is a number of corrupted data that needs to be removed.

• there is a number of outliers in the the nb_transaction values that needs to be assesed
and possibly removed by performing an outlier detecion analysis
pdf_cutoff = pdf[pdf['nb_transaction'] < 200]
[Link](figsize=(10, 6))
[Link](pdf_cutoff['nb_transaction'], vert=False)
[Link]('Number of Transactions')
[Link]('Box Plot of Number of Transactions (with cut-off < 200)')
[Link]()

The tail seems continuous enough to consider that this data is now free of outliers.

The Q1-Q3 range is relatively small, suggesting that the majority of the number of transactions
data is compact.

Missing values
from [Link] import isnan, col
[Link](col('nb_transaction').isNull() |
isnan(col('nb_transaction'))).count()
1501

There are 1472 rows with missing number of transactions. Since sum_achat is most likely
strongly correlated to number of transactions, we'll most likely replace the missing number of
transaction with sum_achat

Sum_achat Colmun
[Link]('sum_achat').describe().show()

+-------+--------------------+
|summary| sum_achat|
+-------+--------------------+
| count| 149700|
| mean|0.021739130434782608|
| stddev| 0.14744195615489714|
| min| -0,13625610743506|
| max| unknown|
+-------+--------------------+

df = [Link](df['sum_achat'] != 'V_8')

[Link]('sum_achat').describe().show()

+-------+--------------------+
|summary| sum_achat|
+-------+--------------------+
| count| 149700|
| mean|0.021739130434782608|
| stddev| 0.14744195615489714|
| min| -0,13625610743506|
| max| unknown|
+-------+--------------------+

[Link](col('sum_achat').desc()).first()

Row(num_crypted=u"tBlcyAYmF6gu3gD9d4wYlOH9x1KmgcIV1zD-
rgU3NPOgyOJtE_SVytEvvd13rJtonbLarfzCNPHaizNZXomgRww=='",
class_client=70.0, subclass_client=106.0, class_service=1.0,
life_time=1.0, type_sim=2.0, nb_transaction=1.0, sum_achat=u'unknown')

df = [Link]("sum_achat", regexp_replace(col("sum_achat"), ",",

"."))
df = [Link]("sum_achat", col("sum_achat").cast("float"))

[Link]('sum_achat').describe().show()
+-------+--------------------+
|summary| sum_achat|
+-------+--------------------+
| count| 145352|
| mean| 0.02064955400476394|
| stddev|0.027451362598597866|
| min| -0.13625611|
| max| 1.0|
+-------+--------------------+

The average sum_achat is 0.020 transactions.

The standard deviation is approximately 0.027 transaction.

This is considered quite high since the mean sum_achat is 0.020.

This high value indicates a significant spread in the sum_achat values, meaning there is
considerable variability around the mean.

There seems to be negative values in the sum_achat column that needs to be dropped kno<ing
that the sum_achat can only be positive.

Outliers analysis
from [Link] import col, regexp_replace
df = [Link]("sum_achat", regexp_replace(col("sum_achat"), ",",
"."))
pdf = [Link]()

pdf['sum_achat'] = pdf['sum_achat'].astype(float)

# Drop any rows with missing values in sum_achat

pdf = [Link](subset=['sum_achat'])

# Draw a box plot for the 'sum_achat' column

[Link](figsize=(10, 6))
[Link](pdf['sum_achat'], vert=False)
[Link]('Sum Achat')
[Link]('Box Plot of Sum Achat')
[Link]()
the Q1-Q3 range (the box) is relatively small, indicating that most of the sum_achat values are
clustered closely together. This suggests that the majority of the data points fall within a narrow
range.

The plot shows a long tail extending from the box to higher values, indicating that there are
numerous data points that are significantly larger than the majority.

The tail becomes discontinuous around the 0.5 mark, meaning that the values beyond 0.5 can be
considered as outliers. We’ll be droping the rows with sum_achat > 0.5

Missing values
from [Link] import isnan, col
[Link](col('sum_achat').isNull() | isnan(col('sum_achat'))).count()

4348

Data Cleaning
df_cleaned = [Link]("Desktop/PFE/Services/[Link]",
schema=schema, header=False, sep=',')
# drop title row
df_cleaned = df_cleaned.filter(df_cleaned.num_crypted != "Achat
Service OTN")

# Cast sum_achat to float

from [Link] import col, regexp_replace
df_cleaned = df_cleaned.withColumn("sum_achat",
regexp_replace(col("sum_achat"), ",", "."))
df_cleaned = df_cleaned.withColumn("sum_achat",
col("sum_achat").cast("float"))

# Missing values
df_cleaned = df_cleaned.dropna(subset=['subclass_client', 'life_time',
'nb_transaction'])

# Outliers removal
df_cleaned = df_cleaned.filter(col("nb_transaction") <= 200)
df_cleaned = df_cleaned.filter(col("sum_achat") <= 0.5)

Data Transformation
Normalize the Numerical Columns
from [Link] import StandardScaler

stages = []
for column in numerical_columns:
assembler = VectorAssembler(inputCols=[column], outputCol=column +
"_vec")
scaler = StandardScaler(inputCol=column + "_vec", outputCol=column
+ "_scaled")
stages += [assembler, scaler]

One-Hot Encode the Categorical Columns

from [Link] import StringIndexer, OneHotEncoder,
VectorAssembler
from [Link] import Pipeline

categorical_columns = ["class_client", "subclass_client",

"class_service", "type_sim"]
numerical_columns = ["life_time", "nb_transaction", "sum_achat"]

# Index and One-Hot Encode the Categorical Columns

for column in categorical_columns:
indexer = StringIndexer(inputCol=column, outputCol=column +
"_index")
encoder = OneHotEncoder(inputCol=column + "_index",
outputCol=column + "_encoded")
stages += [indexer, encoder]

pipeline = Pipeline(stages=stages)
df_encoded = [Link](df_cleaned).transform(df_cleaned)

# Assemble Features including both scaled numerical columns and

encoded categorical columns
scaled_columns = [column + "_scaled" for column in numerical_columns]
encoded_columns = [column + "_encoded" for column in
categorical_columns]
feature_columns = encoded_columns + scaled_columns # Combine encoded
and scaled columns

assembler = VectorAssembler(inputCols=feature_columns,
outputCol="features")
df_features = [Link](df_encoded)

df_features.show()

+--------------------+------------+---------------+-------------
+---------+--------+--------------+------------+-------------
+--------------------+------------------+---------------------
+--------------------+--------------------+------------------
+--------------------+---------------------+-----------------------
+-------------------+---------------------+--------------
+----------------+--------------------+
| num_crypted|class_client|subclass_client|class_service|
life_time|type_sim|nb_transaction| sum_achat|life_time_vec|
life_time_scaled|nb_transaction_vec|nb_transaction_scaled|
sum_achat_vec| sum_achat_scaled|class_client_index|
class_client_encoded|subclass_client_index|subclass_client_encoded|
class_service_index|class_service_encoded|type_sim_index|
type_sim_encoded| features|
+--------------------+------------+---------------+-------------
+---------+--------+--------------+------------+-------------
+--------------------+------------------+---------------------
+--------------------+--------------------+------------------
+--------------------+---------------------+-----------------------
+-------------------+---------------------+--------------
+----------------+--------------------+
|cXHXGWtboQO3-UtA9...| 70| 124| 1|
47| 2| 1|0.0070966724| [47.0]|
[1.234824130535631]| [1.0]| [0.11982080568581...|
[0.00709667243063...|[0.2617748998764478]| 0.0|
(11,[0],[1.0])| 1.0| (56,[1],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,12,67,71,7...|
|TQgqROQQmRYzbFBQ6...| 70| 106| 1|
11| 2| 1|0.0023655575| [11.0]|
[0.28900139225302]| [1.0]| [0.11982080568581...|
[0.00236555747687...|[0.08725829995881...| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|qdWT1RW43fvrO-eyS...| 70| 109| 1|
88| 2| 9| 0.029806023| [88.0]|
[2.31201113802416]| [9.0]| [1.078387251172312]|
[0.02980602346360...| [1.09945455199813]| 0.0|
(11,[0],[1.0])| 4.0| (56,[4],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,15,67,71,7...|
|mn-xISgfFqf7Yx4X-...| 70| 124| 1|
47| 2| 10| 0.014193345| [47.0]|
[1.234824130535631]| [10.0]| [1.1982080568581246]|
[0.01419334486126...|[0.5235497997528956]| 0.0|
(11,[0],[1.0])| 1.0| (56,[1],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,12,67,71,7...|
|xgTk1_o9FTP5fddPZ...| 70| 85| 1|
142| 1| 1|7.8851916E-4| [142.0]|
[3.7307452454480763]| [1.0]| [0.11982080568581...|
[7.88519158959388...|[0.02908609998627...| 0.0|
(11,[0],[1.0])| 17.0| (56,[17],[1.0])|
0.0| (4,[0],[1.0])| 1.0| (1,[],[])|(75,
[0,28,67,72,7...|
|C5Zv431UiQZaXLXBl...| 70| 106| 1|
3| 2| 4|0.0048888186| [3.0]|
[0.0788185615235509]| [4.0]| [0.4792832227432498]|
[0.00488881859928...|[0.18033381304414...| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|JCnEAQKbI7iMqTBou...| 70| 106| 1|
13| 2| 28| 0.15423435| [13.0]|
[0.34154709993538...| [28.0]| [3.3549825592027487]|
[0.15423434972763...| [5.689241239763652]| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|A3-Y0rOfzmvuIMF9R...| 70| 126| 1|
76| 2| 3|0.0070966724| [76.0]|
[1.9967368919299564]| [3.0]| [0.35946241705743...|
[0.00709667243063...|[0.2617748998764478]| 0.0|
(11,[0],[1.0])| 8.0| (56,[8],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,19,67,71,7...|
|GFm2TAh3yr0la30Lv...| 70| 106| 1|
1| 2| 1|0.0013483678| [1.0]|
[0.02627285384118...| [1.0]| [0.11982080568581...|
[0.00134836777579...|[0.04973723149183...| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|XTsoXXs6bjggR2_G5...| 70| 108| 1|
100| 1| 8|0.0113546755| [100.0]|
[2.6272853841183634]| [8.0]| [0.9585664454864996]|
[0.01135467551648...|[0.41883982606084...| 0.0|
(11,[0],[1.0])| 2.0| (56,[2],[1.0])|
0.0| (4,[0],[1.0])| 1.0| (1,[],[])|(75,
[0,13,67,72,7...|
|NMLfEprQR47HEwLy5...| 70| 106| 1|
1| 2| 1|0.0014193344| [1.0]|
[0.02627285384118...| [1.0]| [0.11982080568581...|
[0.00141933443956...|[0.05235497825760...| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|zNhfpnL1mpeUrfTGG...| 70| 108| 1|
83| 2| 1|0.0070966724| [83.0]|
[2.1806468688182417]| [1.0]| [0.11982080568581...|
[0.00709667243063...|[0.2617748998764478]| 0.0|
(11,[0],[1.0])| 2.0| (56,[2],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,13,67,71,7...|
|IqY_nsyMG3fbFXvs1...| 70| 106| 1|
13| 2| 1|0.0014193344| [13.0]|
[0.34154709993538...| [1.0]| [0.11982080568581...|
[0.00141933443956...|[0.05235497825760...| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|KLwD9dDXfDO2_2HAY...| 70| 106| 1|
20| 2| 1|6.3081534E-4| [20.0]|
[0.5254570768236727]| [1.0]| [0.11982080568581...|
[6.30815338809043...|[0.02326888041843...| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|CTzoQ0AritYwGnixA...| 95| 137| 1|
67| 2| 1|7.8851916E-4| [67.0]|
[1.7602812073593035]| [1.0]| [0.11982080568581...|
[7.88519158959388...|[0.02908609998627...| 1.0|
(11,[1],[1.0])| 3.0| (56,[3],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[1,14,67,71,7...|
|59R3KaAWVWU4Hr6iQ...| 70| 108| 1|
85| 2| 4| 0.052830786| [85.0]|
[2.233192576500609]| [4.0]| [0.4792832227432498]|
[0.05283078551292...|[1.9487687677875996]| 0.0|
(11,[0],[1.0])| 2.0| (56,[2],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,13,67,71,7...|
|ZwZzZrlJZx1Y0w4aw...| 70| 106| 1|
26| 2| 3|0.0070966724| [26.0]|
[0.6830941998707745]| [3.0]| [0.35946241705743...|
[0.00709667243063...|[0.2617748998764478]| 0.0|
(11,[0],[1.0])| 0.0| (56,[0],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,11,67,71,7...|
|T9OPU9EQd5CTwXWnP...| 70| 109| 1|
93| 1| 4| 0.00946223| [93.0]|
[2.443375407230078]| [4.0]| [0.4792832227432498]|
[0.00946222990751...|[0.34903319983526...| 0.0|
(11,[0],[1.0])| 4.0| (56,[4],[1.0])|
0.0| (4,[0],[1.0])| 1.0| (1,[],[])|(75,
[0,15,67,72,7...|
|8lni8H4V6Zav6NnNe...| 103| 113| 1|
28| 2| 16| 0.042719264| [28.0]|
[0.7356399075531418]| [16.0]| [1.9171328909729992]|
[0.04271926358342...| [1.575785138267396]| 9.0|
(11,[9],[1.0])| 11.0| (56,[11],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[9,22,67,71,7...|
|dLVzZxwaSHtr9zIV1...| 70| 108| 1|
100| 2| 5| 0.091862485| [100.0]|
[2.6272853841183634]| [5.0]| [0.5991040284290623]|
[0.09186248481273...| [3.388530751461751]| 0.0|
(11,[0],[1.0])| 2.0| (56,[2],[1.0])|
0.0| (4,[0],[1.0])| 0.0| (1,[0],[1.0])|(75,
[0,13,67,71,7...|
+--------------------+------------+---------------+-------------
+---------+--------+--------------+------------+-------------
+--------------------+------------------+---------------------
+--------------------+--------------------+------------------
+--------------------+---------------------+-----------------------
+-------------------+---------------------+--------------
+----------------+--------------------+
only showing top 20 rows
Clustering
Elbow method
import [Link] as plt

wcss = []
range_clusters = range(2, 15) # Define the range for k

for k in range_clusters:
kmeans = KMeans(k=k, seed=1)
model = [Link](df_features)
[Link]([Link])

# Plot the results to find the elbow point

[Link](figsize=(10, 6))
[Link](range_clusters, wcss, marker='o')
[Link]('Elbow Method For Optimal k')
[Link]('Number of clusters')
[Link]('WCSS')
[Link]([range_clusters[0], range_clusters[-1]], [wcss[0], wcss[-1]],
color='red', linestyle='--')
[Link]()

import numpy as np
# Function to calculate the distance from a point to a line
def distance_to_line(point, line_start, line_end):
point = [Link](point)
line_start = [Link](line_start)
line_end = [Link](line_end)
return [Link]([Link](line_end-line_start, line_start-point) /
[Link](line_end-line_start))

# Compute the distances from the line connecting the first and last
points
line_start = [range_clusters[0], wcss[0]]
line_end = [range_clusters[-1], wcss[-1]]
distances = [distance_to_line([k, wcss[i]], line_start, line_end) for
i, k in enumerate(range_clusters)]

# Find the optimal number of clusters as the one with the maximum
distance
optimal_k = range_clusters[[Link](distances)]
optimal_k

distances

[0.0,
3.1902479135290926,
4.22021843795659,
3.5852288854481835,
4.57261642372215,
4.421662828553438,
4.078817689014636,
3.3117429661876425,
2.4228863848843774,
1.973793520961397,
1.3512932673014124,
0.4361865166906377,
0.0]

The optimal number of clusters is 6 since its the furthest from the red line.

We'll cluster the Data into 6 groups in an attempt of building/identifying better user
profile/personas groups.
from [Link] import KMeans
kmeans = KMeans(k=optimal_k, seed=1)
model = [Link](df_features)
predictions = [Link](df_features)

# Show the Results

[Link](*(categorical_columns + numerical_columns +
["prediction"])).show()

+------------+---------------+-------------+--------+---------
+--------------+------------+----------+
|class_client|subclass_client|class_service|type_sim|life_time|
nb_transaction| sum_achat|prediction|
+------------+---------------+-------------+--------+---------
+--------------+------------+----------+
| 70| 124| 1| 2| 47|
1|0.0070966724| 1|
| 70| 106| 1| 2| 11|
1|0.0023655575| 0|
| 70| 109| 1| 2| 88|
9| 0.029806023| 2|
| 70| 124| 1| 2| 47|
10| 0.014193345| 1|
| 70| 85| 1| 1| 142|
1|7.8851916E-4| 2|
| 70| 106| 1| 2| 3|
4|0.0048888186| 0|
| 70| 106| 1| 2| 13|
28| 0.15423435| 5|
| 70| 126| 1| 2| 76|
3|0.0070966724| 2|
| 70| 106| 1| 2| 1|
1|0.0013483678| 0|
| 70| 108| 1| 1| 100|
8|0.0113546755| 2|
| 70| 106| 1| 2| 1|
1|0.0014193344| 0|
| 70| 108| 1| 2| 83|
1|0.0070966724| 2|
| 70| 106| 1| 2| 13|
1|0.0014193344| 0|
| 70| 106| 1| 2| 20|
1|6.3081534E-4| 0|
| 95| 137| 1| 2| 67|
1|7.8851916E-4| 4|
| 70| 108| 1| 2| 85|
4| 0.052830786| 2|
| 70| 106| 1| 2| 26|
3|0.0070966724| 0|
| 70| 109| 1| 1| 93|
4| 0.00946223| 2|
| 103| 113| 1| 2| 28|
16| 0.042719264| 4|
| 70| 108| 1| 2| 100|
5| 0.091862485| 5|
+------------+---------------+-------------+--------+---------
+--------------+------------+----------+
only showing top 20 rows

Clusters exploration : Persona Extraction

Cluster 0
cluster_0_df = [Link]([Link] == 0)

Numercial Columns Comparison

from [Link] import lit

df_summary = df_cleaned.describe()
cluster_0_summary = cluster_0_df.describe()
common_columns = ["summary", "life_time", "nb_transaction",
"sum_achat"]
df_summary_selected =
df_summary.select(common_columns).withColumn("source",
lit("original"))
cluster_0_summary_selected =
cluster_0_summary.select(common_columns).withColumn("source",
lit("cluster_0"))
comparison_summary =
df_summary_selected.union(cluster_0_summary_selected)
comparison_summary_pd = comparison_summary.toPandas()
cols = ['source'] + [col for col in comparison_summary_pd.columns if
col != 'source']
comparison_summary_pd = comparison_summary_pd[cols]
comparison_summary_pd =
comparison_summary_pd.sort_values(by=['summary', 'source'])
comparison_summary_pd

source summary life_time nb_transaction \

5 cluster_0 count 55128 55128
0 original count 141307 141307
9 cluster_0 max 62 22
4 original max 154 174
6 cluster_0 mean 9.492453925409954 3.5653569873748365
1 original mean 42.89374907117128 5.589871697792749
8 cluster_0 min 1 1
3 original min 1 1
7 cluster_0 stddev 10.539010542176188 3.870916151388448
2 original stddev 38.0621003734457 8.345795993244655

sum_achat
5 55128
0 141307
9 0.06134679
4 0.46104714
6 0.013819638618474076
1 0.020591187240199666
8 0.0
3 0.0
7 0.014885433804150553
2 0.027109827695412994

Numerical Columns comparison conclusion

Cluster 0 likely represents a segment of clients who are either new or minimally engaged with
the service. Their lower transaction counts and purchase amounts suggest they might be
exploring the service without committing fully, or they might be low-budget clients.
Understanding this cluster can help in tailoring strategies to increase their engagement and
spending, such as personalized offers, targeted communication, or loyalty programs.

Categorical Columns Comparison

from [Link] import col

def compare_percentage_distribution(df_cleaned, cluster_df,

column_name):
total_count_df = df_cleaned.count()
total_count_cluster = cluster_df.count()

df_class_percentage = df_cleaned.groupBy(column_name).count() \
.withColumn('percentage', (col('count') / total_count_df) *
100) \
.orderBy(col('percentage'), ascending=False)

cluster_class_percentage = cluster_df.groupBy(column_name).count()
\
.withColumn('percentage', (col('count') / total_count_cluster)
* 100) \
.orderBy(col('percentage'), ascending=False)

df_class_percentage =
df_class_percentage.withColumnRenamed('count', 'total_count') \
.withColumnRenamed('percentage', 'total_percentage')

cluster_class_percentage =
cluster_class_percentage.withColumnRenamed('count', 'cluster_count') \
.withColumnRenamed('percentage', 'cluster_percentage')

comparison_df = df_class_percentage.join(cluster_class_percentage,
on=column_name, how='outer')
 return comparison_df.toPandas()

# Class Client Distribution

compare_percentage_distribution(df_cleaned, cluster_0_df,
"class_client")

class_client total_count total_percentage cluster_count \

0 81 4496 3.181725 2711.0
1 103 23 0.016277 NaN
2 96 1202 0.850630 60.0
3 92 3 0.002123 1.0
4 72 1073 0.759340 5.0
5 102 1873 1.325483 16.0
6 82 42 0.029723 8.0
7 73 260 0.183997 39.0
8 70 121319 85.854912 52264.0
9 95 10034 7.100851 NaN
10 83 10 0.007077 NaN
11 71 972 0.687864 24.0

cluster_percentage
0 4.917646
1 NaN
2 0.108838
3 0.001814
4 0.009070
5 0.029023
6 0.014512
7 0.070744
8 94.804818
9 NaN
10 NaN
11 0.043535

# Subclass client Distribution

compare_percentage_distribution(df_cleaned, cluster_0_df,
"subclass_client")

subclass_client total_count total_percentage cluster_count \

0 148 70 0.049538 1.0
1 85 564 0.399131 2.0
2 137 7526 5.325992 NaN
3 133 66 0.046707 23.0
4 108 9432 6.674829 43.0
5 34 58 0.041045 NaN
6 115 1 0.000708 NaN
7 126 2528 1.789013 47.0
8 28 80 0.056614 NaN
9 103 5 0.003538 NaN
10 12 64 0.045291 4.0
11 93 15 0.010615 NaN
12 111 1936 1.370067 794.0
13 47 1 0.000708 NaN
14 132 13 0.009200 2.0
15 152 37 0.026184 29.0
16 13 41 0.029015 3.0
17 20 460 0.325532 1.0
18 57 39 0.027599 NaN
19 339 307 0.217257 147.0
20 235 293 0.207350 NaN
21 48 2 0.001415 NaN
22 61 1 0.000708 NaN
23 127 3548 2.510845 3081.0
24 88 15 0.010615 NaN
25 107 1050 0.743063 4.0
26 114 4 0.002831 NaN
27 136 4 0.002831 2.0
28 129 1 0.000708 NaN
29 77 3 0.002123 NaN
30 102 86 0.060860 7.0
31 113 1566 1.108225 NaN
32 237 141 0.099783 128.0
33 143 159 0.112521 108.0
34 338 892 0.631250 539.0
35 98 379 0.268210 1.0
36 341 2873 2.033162 2576.0
37 145 11 0.007784 1.0
38 109 7441 5.265840 23.0
39 105 1016 0.719002 14.0
40 11 239 0.169135 48.0
41 33 57 0.040338 9.0
42 150 740 0.523682 NaN
43 68 8 0.005661 NaN
44 106 56804 40.198999 45341.0
45 116 3633 2.570998 31.0
46 147 1451 1.026842 NaN
47 123 3 0.002123 NaN
48 119 284 0.200981 73.0
49 131 32 0.022646 7.0
50 149 2406 1.702676 1934.0
51 124 32482 22.986830 9.0
52 46 1 0.000708 NaN
53 144 80 0.056614 1.0
54 153 79 0.055907 75.0
55 104 82 0.058030 19.0
56 36 198 0.140120 1.0

cluster_percentage
0 0.001814
1 0.003628
2 NaN
3 0.041721
4 0.078000
5 NaN
6 NaN
7 0.085256
8 NaN
9 NaN
10 0.007256
11 NaN
12 1.440284
13 NaN
14 0.003628
15 0.052605
16 0.005442
17 0.001814
18 NaN
19 0.266652
20 NaN
21 NaN
22 NaN
23 5.588811
24 NaN
25 0.007256
26 NaN
27 0.003628
28 NaN
29 NaN
30 0.012698
31 NaN
32 0.232187
33 0.195908
34 0.977725
35 0.001814
36 4.672762
37 0.001814
38 0.041721
39 0.025395
40 0.087070
41 0.016326
42 NaN
43 NaN
44 82.246771
45 0.056233
46 NaN
47 NaN
48 0.132419
49 0.012698
50 3.508199
51 0.016326
52 NaN
53 0.001814
54 0.136047
55 0.034465
56 0.001814

# class service Distribution

compare_percentage_distribution(df_cleaned, cluster_0_df,
"class_service")

class_service total_count total_percentage cluster_count \

0 1 133998 94.827574 52262
1 3 138 0.097660 111
2 5 10 0.007077 5
3 4 640 0.452915 288
4 2 6521 4.614775 2462

cluster_percentage
0 94.801190
1 0.201350
2 0.009070
3 0.522421
4 4.465970

# Type sim Distribution

compare_percentage_distribution(df_cleaned, cluster_0_df, "type_sim")

type_sim total_count total_percentage cluster_count

cluster_percentage
0 1 10260 7.260787 36
0.065303
1 2 131047 92.739213 55092
99.934697

Categorical Columns comparison conclusion

• class_client Distribution: Class 95: Previously represented 7% of the total
population but is now completely absent in the cluster, indicating a significant shift.

• subclass_client Distribution: Subclass 106: Previously constituted 40% of the

total population and now stands at 82% in the cluster, showing a dramatic increase
and strong association with the cluster.

• class_service Distribut No notable changes were observed, indicating that the

distribution of class_service remains relatively consistent between the total
population and the cluster.
 • type_sim Distribution: Type Sim 1: Previously made up 7% of the total
population but is now completely absent in the cluster, suggesting a significant shift
away from this type within the cluste

Conclusion
The most notable changes in the cluster include the complete absence of class_client
95 and type_sim 1, as well as the dramatic increase in subclass_client 106 from 40%
to 82%. These shifts highlight specific trends and associations within the clustered
group that could inform targeted strategies and [Link]:emainrventions.

Cluster 0 Persona/Behavior:
Persona: Cluster 0 comprises clients predominantly characterized by their membership in
subclass_client 106, who are new or exhibit minimal engagement with the service. They do not
belong to class_client 95 or use type_sim 1.

Behavior: These clients likely exhibit a cautious approach to using the service, exploring without
full commitment or having budget constraints. Their lower transaction counts and purchase
amounts further emphasize this cautious and budget-conscious behavior.

Cluster 1
cluster_1_df = [Link]([Link] == 1)

Numercial Columns Comparison

from [Link] import lit

df_summary = df_cleaned.describe()
cluster_1_summary = cluster_1_df.describe()
common_columns = ["summary", "life_time", "nb_transaction",
"sum_achat"]
df_summary_selected =
df_summary.select(common_columns).withColumn("source",
lit("original"))
cluster_1_summary_selected =
cluster_1_summary.select(common_columns).withColumn("source",
lit("cluster_0"))
comparison_summary =
df_summary_selected.union(cluster_1_summary_selected)
comparison_summary_pd = comparison_summary.toPandas()
cols = ['source'] + [col for col in comparison_summary_pd.columns if
col != 'source']
comparison_summary_pd = comparison_summary_pd[cols]
comparison_summary_pd =
comparison_summary_pd.sort_values(by=['summary', 'source'])
comparison_summary_pd
 source summary life_time nb_transaction \
5 cluster_0 count 28829 28829
0 original count 141307 141307
9 cluster_0 max 97 22
4 original max 154 174
6 cluster_0 mean 54.877623226612094 4.694335564882583
1 original mean 42.89374907117128 5.589871697792749
8 cluster_0 min 4 1
3 original min 1 1
7 cluster_0 stddev 14.500714391696485 4.403776291908017
2 original stddev 38.0621003734457 8.345795993244655

sum_achat
5 28829
0 141307
9 0.06134679
4 0.46104714
6 0.014255520952511175
1 0.020591187240199666
8 0.0
3 0.0
7 0.0146081887161619
2 0.027109827695412994

Numerical Columns comparison conclusion

Cluster 1 consists of customers who, on average, have longer lifetimes but are less varied in their
engagement compared to the entire dataset. Despite their longer engagement, they exhibit
lower transaction counts and spending amounts. This cluster likely includes more consistent but
moderately engaged customers.

Categorical Columns Comparison

# Class Client Distribution
from [Link] import col
compare_percentage_distribution(df_cleaned, cluster_1_df,
"class_client")

class_client total_count total_percentage cluster_count \

0 81 4496 3.181725 664.0
1 103 23 0.016277 NaN
2 96 1202 0.850630 37.0
3 92 3 0.002123 NaN
4 72 1073 0.759340 7.0
5 102 1873 1.325483 28.0
6 82 42 0.029723 3.0
7 73 260 0.183997 31.0
8 70 121319 85.854912 27872.0
9 95 10034 7.100851 185.0
10 83 10 0.007077 NaN
11 71 972 0.687864 2.0

cluster_percentage
0 2.303236
1 NaN
2 0.128343
3 NaN
4 0.024281
5 0.097124
6 0.010406
7 0.107531
8 96.680426
9 0.641715
10 NaN
11 0.006937

# Subclass client Distribution

compare_percentage_distribution(df_cleaned, cluster_1_df,
"subclass_client")

subclass_client total_count total_percentage cluster_count \

0 148 70 0.049538 27.0
1 85 564 0.399131 3.0
2 137 7526 5.325992 NaN
3 133 66 0.046707 22.0
4 108 9432 6.674829 32.0
5 34 58 0.041045 2.0
6 115 1 0.000708 NaN
7 126 2528 1.789013 1011.0
8 28 80 0.056614 NaN
9 103 5 0.003538 NaN
10 12 64 0.045291 11.0
11 93 15 0.010615 2.0
12 111 1936 1.370067 479.0
13 47 1 0.000708 NaN
14 132 13 0.009200 4.0
15 152 37 0.026184 NaN
16 13 41 0.029015 2.0
17 20 460 0.325532 3.0
18 57 39 0.027599 3.0
19 339 307 0.217257 NaN
20 235 293 0.207350 NaN
21 48 2 0.001415 NaN
22 61 1 0.000708 NaN
23 127 3548 2.510845 43.0
24 88 15 0.010615 NaN
25 107 1050 0.743063 5.0
26 114 4 0.002831 NaN
27 136 4 0.002831 1.0
28 129 1 0.000708 1.0
29 77 3 0.002123 NaN
30 102 86 0.060860 16.0
31 113 1566 1.108225 NaN
32 237 141 0.099783 NaN
33 143 159 0.112521 NaN
34 338 892 0.631250 NaN
35 98 379 0.268210 3.0
36 341 2873 2.033162 1.0
37 145 11 0.007784 NaN
38 109 7441 5.265840 33.0
39 105 1016 0.719002 141.0
40 11 239 0.169135 59.0
41 33 57 0.040338 20.0
42 150 740 0.523682 NaN
43 68 8 0.005661 NaN
44 106 56804 40.198999 141.0
45 116 3633 2.570998 21.0
46 147 1451 1.026842 NaN
47 123 3 0.002123 NaN
48 119 284 0.200981 64.0
49 131 32 0.022646 9.0
50 149 2406 1.702676 14.0
51 124 32482 22.986830 26653.0
52 46 1 0.000708 NaN
53 144 80 0.056614 NaN
54 153 79 0.055907 NaN
55 104 82 0.058030 3.0
56 36 198 0.140120 NaN

cluster_percentage
0 0.093656
1 0.010406
2 NaN
3 0.076312
4 0.110999
5 0.006937
6 NaN
7 3.506885
8 NaN
9 NaN
10 0.038156
11 0.006937
12 1.661521
13 NaN
14 0.013875
15 NaN
16 0.006937
17 0.010406
18 0.010406
19 NaN
20 NaN
21 NaN
22 NaN
23 0.149155
24 NaN
25 0.017344
26 NaN
27 0.003469
28 0.003469
29 NaN
30 0.055500
31 NaN
32 NaN
33 NaN
34 NaN
35 0.010406
36 0.003469
37 NaN
38 0.114468
39 0.489091
40 0.204655
41 0.069375
42 NaN
43 NaN
44 0.489091
45 0.072843
46 NaN
47 NaN
48 0.221999
49 0.031219
50 0.048562
51 92.452045
52 NaN
53 NaN
54 NaN
55 0.010406
56 NaN

# class service Distribution

compare_percentage_distribution(df_cleaned, cluster_1_df,
"class_service")

class_service total_count total_percentage cluster_count \

0 1 133998 94.827574 28452.0
1 3 138 0.097660 1.0
2 5 10 0.007077 NaN
3 4 640 0.452915 54.0
4 2 6521 4.614775 322.0
 cluster_percentage
0 98.692289
1 0.003469
2 NaN
3 0.187311
4 1.116931

# Type sim Distribution

compare_percentage_distribution(df_cleaned, cluster_1_df, "type_sim")

type_sim total_count total_percentage cluster_count

cluster_percentage
0 1 10260 7.260787 167
0.579278
1 2 131047 92.739213 28662
99.420722

Categorical Columns Comparison Conclusion

class_client Distribution:
• Class 95: Previously represented 7.10% of the total population but is now completely
absent in the cluster, indicating a significant exclusion of this class from the cluster's
composition.

subclass_client Distribution:
• Subclass 106: Previously constituted 40.20% of the total population and now stands at
0.49% in the cluster, showing a dramatic decrease and less association with the cluster.

class_service Distribution:
• No notable changes were observed, indicating that the distribution of class_service
remains relatively consistent between the total population and the cluster.

type_sim Distribution:
• Type Sim 1: Previously made up 7.26% of the total population but is now completely
absent in the cluster, suggesting a significant shift away from this type within the cluster.

Conclusion
The most notable changes in the cluster include the complete absence of class_client 95
and type_sim 1, as well as the dramatic decrease in subclass_client 106 from 40.20% to
0.49%. These shifts highlight specific trends and associations within the clustered group that
could inform targeted strategies and interventions. Understanding these changes can guide
efforts to tailor marketing strategies or service offerings to better align with the characteristics
of the clustered group.

Cluster 1 Persona/Behavior:
Persona:
Cluster 1 consists of customers who exhibit longer engagement periods compared to the entire
dataset. They show variability in their categorical profiles, particularly in class_client and
subclass_client, suggesting diverse customer segments within the cluster.

Behavior:

These customers demonstrate longer engagement with the service but lower transaction counts
and spending amounts compared to the overall dataset. They are characterized by consistent
but moderate engagement levels, indicating a stable relationship with the service offerings.
Despite spending less and making fewer transactions on average, their extended engagement
suggests loyalty and continued interest in the service.

Cluster 2
cluster_2_df = [Link]([Link] == 2)

Numercial Columns Comparison

from [Link] import lit

df_summary = df_cleaned.describe()
cluster_2_summary = cluster_2_df.describe()
common_columns = ["summary", "life_time", "nb_transaction",
"sum_achat"]
df_summary_selected =
df_summary.select(common_columns).withColumn("source",
lit("original"))
cluster_2_summary_selected =
cluster_2_summary.select(common_columns).withColumn("source",
lit("cluster_2"))
comparison_summary =
df_summary_selected.union(cluster_2_summary_selected)
comparison_summary_pd = comparison_summary.toPandas()
cols = ['source'] + [col for col in comparison_summary_pd.columns if
col != 'source']
comparison_summary_pd = comparison_summary_pd[cols]
comparison_summary_pd =
comparison_summary_pd.sort_values(by=['summary', 'source'])
comparison_summary_pd

source summary life_time nb_transaction \

5 cluster_2 count 28142 28142
0 original count 141307 141307
9 cluster_2 max 154 28
4 original max 154 174
6 cluster_2 mean 100.14572525051524 4.450145689716438
1 original mean 42.89374907117128 5.589871697792749
8 cluster_2 min 46 1
3 original min 1 1
7 cluster_2 stddev 17.650532225043108 4.356242722722819
2 original stddev 38.0621003734457 8.345795993244655

sum_achat
5 28142
0 141307
9 0.08476581
4 0.46104714
6 0.014659374975396287
1 0.020591187240199666
8 0.0
3 0.0
7 0.014796826469914143
2 0.027109827695412994

Numerical Columns comparison conclusion

Cluster 2 likely represents a segment of clients who have a significantly longer lifetime
engagement with the service compared to the overall dataset. However, despite their longer
lifetimes, they exhibit lower transaction counts and spending amounts.

Categorical Columns Comparison

# Class Client Distribution
from [Link] import col
compare_percentage_distribution(df_cleaned, cluster_2_df,
"class_client")

class_client total_count total_percentage cluster_count \

0 81 4496 3.181725 719.0
1 103 23 0.016277 NaN
2 96 1202 0.850630 55.0
3 92 3 0.002123 NaN
4 72 1073 0.759340 155.0
5 102 1873 1.325483 9.0
6 82 42 0.029723 21.0
7 73 260 0.183997 97.0
8 70 121319 85.854912 26910.0
9 95 10034 7.100851 87.0
10 83 10 0.007077 4.0
11 71 972 0.687864 85.0

cluster_percentage
0 2.554900
1 NaN
2 0.195437
3 NaN
4 0.550778
5 0.031981
6 0.074622
7 0.344681
8 95.622202
9 0.309146
10 0.014214
11 0.302040

# Subclass client Distribution

compare_percentage_distribution(df_cleaned, cluster_2_df,
"subclass_client")

subclass_client total_count total_percentage cluster_count \

0 148 70 0.049538 30.0
1 85 564 0.399131 508.0
2 137 7526 5.325992 129.0
3 133 66 0.046707 NaN
4 108 9432 6.674829 8484.0
5 34 58 0.041045 21.0
6 115 1 0.000708 NaN
7 126 2528 1.789013 990.0
8 28 80 0.056614 72.0
9 103 5 0.003538 5.0
10 12 64 0.045291 38.0
11 93 15 0.010615 4.0
12 111 1936 1.370067 291.0
13 47 1 0.000708 1.0
14 132 13 0.009200 NaN
15 152 37 0.026184 NaN
16 13 41 0.029015 17.0
17 20 460 0.325532 410.0
18 57 39 0.027599 23.0
19 339 307 0.217257 NaN
20 235 293 0.207350 NaN
21 48 2 0.001415 1.0
22 61 1 0.000708 1.0
23 127 3548 2.510845 60.0
24 88 15 0.010615 7.0
25 107 1050 0.743063 937.0
26 114 4 0.002831 1.0
27 136 4 0.002831 NaN
28 129 1 0.000708 NaN
29 77 3 0.002123 3.0
30 102 86 0.060860 53.0
31 113 1566 1.108225 120.0
32 237 141 0.099783 NaN
33 143 159 0.112521 NaN
34 338 892 0.631250 NaN
35 98 379 0.268210 343.0
36 341 2873 2.033162 NaN
37 145 11 0.007784 NaN
38 109 7441 5.265840 6715.0
39 105 1016 0.719002 649.0
40 11 239 0.169135 95.0
41 33 57 0.040338 10.0
42 150 740 0.523682 NaN
43 68 8 0.005661 7.0
44 106 56804 40.198999 3540.0
45 116 3633 2.570998 3315.0
46 147 1451 1.026842 14.0
47 123 3 0.002123 2.0
48 119 284 0.200981 103.0
49 131 32 0.022646 NaN
50 149 2406 1.702676 18.0
51 124 32482 22.986830 1067.0
52 46 1 0.000708 NaN
53 144 80 0.056614 NaN
54 153 79 0.055907 NaN
55 104 82 0.058030 45.0
56 36 198 0.140120 13.0

cluster_percentage
0 0.106602
1 1.805131
2 0.458390
3 NaN
4 30.147111
5 0.074622
6 NaN
7 3.517874
8 0.255845
9 0.017767
10 0.135029
11 0.014214
12 1.034042
13 0.003553
14 NaN
15 NaN
16 0.060408
17 1.456897
18 0.081728
19 NaN
20 NaN
21 0.003553
22 0.003553
23 0.213204
24 0.024874
25 3.329543
26 0.003553
27 NaN
28 NaN
29 0.010660
30 0.188331
31 0.426409
32 NaN
33 NaN
34 NaN
35 1.218819
36 NaN
37 NaN
38 23.861133
39 2.306162
40 0.337574
41 0.035534
42 NaN
43 0.024874
44 12.579063
45 11.779547
46 0.049748
47 0.007107
48 0.366001
49 NaN
50 0.063961
51 3.791486
52 NaN
53 NaN
54 NaN
55 0.159903
56 0.046194

# class service Distribution

compare_percentage_distribution(df_cleaned, cluster_2_df,
"class_service")

class_service total_count total_percentage cluster_count \

0 1 133998 94.827574 25377
1 3 138 0.097660 18
2 5 10 0.007077 5
3 4 640 0.452915 83
4 2 6521 4.614775 2659

cluster_percentage
0 90.174828
1 0.063961
2 0.017767
3 0.294933
4 9.448511

# Type sim Distribution

compare_percentage_distribution(df_cleaned, cluster_2_df, "type_sim")
 type_sim total_count total_percentage cluster_count
cluster_percentage
0 1 10260 7.260787 9228
32.790846
1 2 131047 92.739213 18914
67.209154

Categorical Columns Comparison Conclusion

class_client Distribution:
• Class 95: Previously represented 7.1% of the total population but is now completely
absent in the cluster, indicating a significant absence in this particular segment within
the cluster.
• Class 70: Maintains a high presence, representing 85.85% of the total population and
95.62% in the cluster, showing consistency across the dataset and the cluster.
• Class 73: Was 0.18% of the total population and is now at 0.34% in the cluster,
indicating a slight increase in representation within the cluster.

subclass_client Distribution:
• Subclass 106: Previously constituted 40.20% of the total population and now stands at
12.58% in the cluster, showing a notable decrease in representation within the cluster.
• Subclass 109: Increased significantly from 5.27% to 23.86%, indicating a strong
presence and association with the cluster.
• Subclass 108: Maintains a high presence, representing 6.67% of the total population and
30.15% in the cluster, showing consistency across the dataset and the cluster.

class_service Distribution:
• No notable changes were observed, indicating that the distribution of class_service
remains relatively consistent between the total population and the cluster.

type_sim Distribution:
• Type Sim 1: Previously made up 7.26% of the total population but is now completely
absent in the cluster, suggesting a significant shift away from this type within the cluster.
• Type Sim 2: Represents 92.74% of the total population and 67.21% in the cluster,
indicating a lower representation in the cluster compared to the total population.

Conclusion
The most notable changes in the cluster include the absence of class_client 95 and
type_sim 1, as well as significant shifts in subclass_client distributions. Specifically,
subclass_client 106 saw a decrease, while subclass_client 109 showed a notable
increase, reflecting specific trends and associations within the clustered group. These insights
can inform targeted strategies and interventions tailored to the characteristics of each cluster.
Cluster 2 Persona/Behavior:
Persona:

Cluster 2 represents a segment of clients characterized by longer lifetimes and distinct

categorical profiles, particularly in class_client, subclass_client, class_service, and
type_sim. This cluster shows specific preferences and behaviors within these categories,
reflecting their unique segment within the customer base.

Behavior:

Cluster 2 likely represents a segment of clients who have a significantly longer lifetime
engagement with the service compared to the overall dataset. They exhibit lower transaction
counts and spending amounts despite their extended engagement periods. This behavior
suggests a cautious or less frequent purchasing pattern, possibly indicating a focus on long-
term relationship building with the service provider. The cluster's composition indicates a mix of
consistent and varied categorical preferences, highlighting a complex customer profile with
potential opportunities for targeted engagement strategies to enhance transaction frequency
and spending levels.

Cluster 3
cluster_3_df = [Link]([Link] == 3)

Numercial Columns Comparison

from [Link] import lit

df_summary = df_cleaned.describe()
cluster_3_summary = cluster_3_df.describe()
common_columns = ["summary", "life_time", "nb_transaction",
"sum_achat"]
df_summary_selected =
df_summary.select(common_columns).withColumn("source",
lit("original"))
cluster_3_summary_selected =
cluster_3_summary.select(common_columns).withColumn("source",
lit("cluster_3"))
comparison_summary =
df_summary_selected.union(cluster_3_summary_selected)
comparison_summary_pd = comparison_summary.toPandas()
cols = ['source'] + [col for col in comparison_summary_pd.columns if
col != 'source']
comparison_summary_pd = comparison_summary_pd[cols]
comparison_summary_pd =
comparison_summary_pd.sort_values(by=['summary', 'source'])
comparison_summary_pd
 source summary life_time nb_transaction \
5 cluster_3 count 6725 6725
0 original count 141307 141307
9 cluster_3 max 154 174
4 original max 154 174
6 cluster_3 mean 42.39375464684015 33.96104089219331
1 original mean 42.89374907117128 5.589871697792749
8 cluster_3 min 1 17
3 original min 1 1
7 cluster_3 stddev 35.21880560084209 15.844799554426414
2 original stddev 38.0621003734457 8.345795993244655

sum_achat
5 6725
0 141307
9 0.40119854
4 0.46104714
6 0.05003317081363401
1 0.020591187240199666
8 6.3081534E-4
3 0.0
7 0.031060748769071393
2 0.027109827695412994

Numerical Columns comparison conclusion

Cluster 3 is characterized by clients with a moderate average lifetime of approximately 42 days,
indicating some engagement with the service.

Their behavior shows significantly higher transaction counts averaging around 34 transactions
and a slightly higher spending amount compared to the overall dataset.

Categorical Columns Comparison

# Class Client Distribution
from [Link] import col
compare_percentage_distribution(df_cleaned, cluster_3_df,
"class_client")

class_client total_count total_percentage cluster_count \

0 81 4496 3.181725 79.0
1 103 23 0.016277 NaN
2 96 1202 0.850630 26.0
3 92 3 0.002123 NaN
4 72 1073 0.759340 23.0
5 102 1873 1.325483 51.0
6 82 42 0.029723 3.0
7 73 260 0.183997 7.0
8 70 121319 85.854912 6255.0
9 95 10034 7.100851 254.0
10 83 10 0.007077 NaN
11 71 972 0.687864 27.0

cluster_percentage
0 1.174721
1 NaN
2 0.386617
3 NaN
4 0.342007
5 0.758364
6 0.044610
7 0.104089
8 93.011152
9 3.776952
10 NaN
11 0.401487

# Subclass client Distribution

compare_percentage_distribution(df_cleaned, cluster_3_df,
"subclass_client")

subclass_client total_count total_percentage cluster_count \

0 148 70 0.049538 NaN
1 85 564 0.399131 39.0
2 137 7526 5.325992 165.0
3 133 66 0.046707 1.0
4 108 9432 6.674829 397.0
5 34 58 0.041045 NaN
6 115 1 0.000708 NaN
7 126 2528 1.789013 171.0
8 28 80 0.056614 4.0
9 103 5 0.003538 NaN
10 12 64 0.045291 2.0
11 93 15 0.010615 NaN
12 111 1936 1.370067 75.0
13 47 1 0.000708 NaN
14 132 13 0.009200 NaN
15 152 37 0.026184 NaN
16 13 41 0.029015 NaN
17 20 460 0.325532 36.0
18 57 39 0.027599 NaN
19 339 307 0.217257 NaN
20 235 293 0.207350 8.0
21 48 2 0.001415 NaN
22 61 1 0.000708 NaN
23 127 3548 2.510845 135.0
24 88 15 0.010615 NaN
25 107 1050 0.743063 47.0
26 114 4 0.002831 NaN
27 136 4 0.002831 NaN
28 129 1 0.000708 NaN
29 77 3 0.002123 NaN
30 102 86 0.060860 1.0
31 113 1566 1.108225 26.0
32 237 141 0.099783 NaN
33 143 159 0.112521 NaN
34 338 892 0.631250 8.0
35 98 379 0.268210 15.0
36 341 2873 2.033162 64.0
37 145 11 0.007784 NaN
38 109 7441 5.265840 301.0
39 105 1016 0.719002 51.0
40 11 239 0.169135 11.0
41 33 57 0.040338 NaN
42 150 740 0.523682 21.0
43 68 8 0.005661 NaN
44 106 56804 40.198999 2717.0
45 116 3633 2.570998 134.0
46 147 1451 1.026842 8.0
47 123 3 0.002123 NaN
48 119 284 0.200981 23.0
49 131 32 0.022646 NaN
50 149 2406 1.702676 169.0
51 124 32482 22.986830 2082.0
52 46 1 0.000708 NaN
53 144 80 0.056614 NaN
54 153 79 0.055907 2.0
55 104 82 0.058030 4.0
56 36 198 0.140120 8.0

cluster_percentage
0 NaN
1 0.579926
2 2.453532
3 0.014870
4 5.903346
5 NaN
6 NaN
7 2.542751
8 0.059480
9 NaN
10 0.029740
11 NaN
12 1.115242
13 NaN
14 NaN
15 NaN
16 NaN
17 0.535316
18 NaN
19 NaN
20 0.118959
21 NaN
22 NaN
23 2.007435
24 NaN
25 0.698885
26 NaN
27 NaN
28 NaN
29 NaN
30 0.014870
31 0.386617
32 NaN
33 NaN
34 0.118959
35 0.223048
36 0.951673
37 NaN
38 4.475836
39 0.758364
40 0.163569
41 NaN
42 0.312268
43 NaN
44 40.401487
45 1.992565
46 0.118959
47 NaN
48 0.342007
49 NaN
50 2.513011
51 30.959108
52 NaN
53 NaN
54 0.029740
55 0.059480
56 0.118959

# class service Distribution

compare_percentage_distribution(df_cleaned, cluster_3_df,
"class_service")

class_service total_count total_percentage cluster_count \

0 1 133998 94.827574 6257.0
1 3 138 0.097660 4.0
2 5 10 0.007077 NaN
3 4 640 0.452915 NaN
4 2 6521 4.614775 464.0
 cluster_percentage
0 93.040892
1 0.059480
2 NaN
3 NaN
4 6.899628

# Type sim Distribution

compare_percentage_distribution(df_cleaned, cluster_3_df, "type_sim")

type_sim total_count total_percentage cluster_count

cluster_percentage
0 1 10260 7.260787 410
6.096654
1 2 131047 92.739213 6315
93.903346

Categorical Columns comparison conclusion

Class Client Distribution:

Class 95: Previously represented 7% of the total population but is now completely absent in the
cluster, indicating a significant shift away from this class.

Subclass Client Distribution:

Subclass 106: Previously constituted 40% of the total population and now stands at 82% in the
cluster, showing a dramatic increase and strong association with the cluster. Subclass 107:
Previously represented 40% of the total population but now stands at 40% in the cluster,
showing a moderate decrease and moderate association with the cluster. Subclass 72:
Previously constituted 40% of the total population and now stands at 40% in the cluster,
showing a dramatic increase and strong association with the cluster.

Class Service Distribution:

No notable changes were observed, indicating that the distribution of class_service remains
relatively consistent between the total population and the cluster.

Type Sim Distribution: Type Sim 1: Previously made up 7% of the total population but is now
completely absent in the cluster, suggesting a significant shift away from this type within the
cluster.

Conclusion
The most notable changes in the cluster include the complete absence of class_client 95 and
type_sim 1, as well as the dramatic increase in subclass_client 106 from 40% to 82%. These
shifts highlight specific trends and associations within the clustered group that could inform
targeted strategies and interventions.
Cluster 3 Persona/Behavior:
Persona: Cluster 3 represents clients characterized by a predominant membership in
subclass_client 106, showing a moderate engagement level with the service. They largely
belong to class_client 70 and predominantly use type_sim 2.

Behavior: These clients tend to have moderate transaction counts and purchase amounts,
indicating a balanced engagement with the service. They exhibit a preference for consistent
usage without extremely high or low activity levels. Understanding this cluster can help tailor
strategies to maintain their moderate engagement and potentially increase their participation
through targeted loyalty programs or personalized offers.

Cluster 4
cluster_4_df = [Link]([Link] == 4)

Numercial Columns Comparison

from [Link] import lit

df_summary = df_cleaned.describe()
cluster_4_summary = cluster_4_df.describe()
common_columns = ["summary", "life_time", "nb_transaction",
"sum_achat"]
df_summary_selected =
df_summary.select(common_columns).withColumn("source",
lit("original"))
cluster_4_summary_selected =
cluster_4_summary.select(common_columns).withColumn("source",
lit("cluster_4"))
comparison_summary =
df_summary_selected.union(cluster_4_summary_selected)
comparison_summary_pd = comparison_summary.toPandas()
cols = ['source'] + [col for col in comparison_summary_pd.columns if
col != 'source']
comparison_summary_pd = comparison_summary_pd[cols]
comparison_summary_pd =
comparison_summary_pd.sort_values(by=['summary', 'source'])
comparison_summary_pd

source summary life_time nb_transaction \

5 cluster_4 count 13664 13664
0 original count 141307 141307
9 cluster_4 max 95 23
4 original max 154 174
6 cluster_4 mean 36.61109484777518 3.444672131147541
1 original mean 42.89374907117128 5.589871697792749
8 cluster_4 min 1 1
3 original min 1 1
7 cluster_4 stddev 22.766358070584612 3.5564083170975302
2 original stddev 38.0621003734457 8.345795993244655

sum_achat
5 13664
0 141307
9 0.06308153
4 0.46104714
6 0.011601299268702505
1 0.020591187240199666
8 0.0
3 0.0
7 0.013173007050880654
2 0.027109827695412994

Numerical Columns comparison conclusion

Cluster 4 consists of clients who show a moderate level of engagement with the service,
characterized by an average lifetime of approximately 37 days. They exhibit lower transaction
counts and spending amounts compared to the overall dataset, suggesting cautious and
possibly budget-conscious behavior. This cluster may represent clients who are exploring the
service without fully committing or who have varying levels of engagement due to budget
constraints.

Categorical Columns Comparison

# Class Client Distribution
from [Link] import col
compare_percentage_distribution(df_cleaned, cluster_4_df,
"class_client")

class_client total_count total_percentage cluster_count \

0 81 4496 3.181725 208.0
1 103 23 0.016277 20.0
2 96 1202 0.850630 937.0
3 92 3 0.002123 1.0
4 72 1073 0.759340 828.0
5 102 1873 1.325483 1705.0
6 82 42 0.029723 4.0
7 73 260 0.183997 54.0
8 70 121319 85.854912 NaN
9 95 10034 7.100851 9123.0
10 83 10 0.007077 4.0
11 71 972 0.687864 780.0

cluster_percentage
0 1.522248
1 0.146370
2 6.857436
3 0.007319
4 6.059719
5 12.478044
6 0.029274
7 0.395199
8 NaN
9 66.766686
10 0.029274
11 5.708431

# Subclass client Distribution

compare_percentage_distribution(df_cleaned, cluster_4_df,
"subclass_client")

subclass_client total_count total_percentage cluster_count \

0 148 70 0.049538 2.0
1 85 564 0.399131 NaN
2 137 7526 5.325992 6990.0
3 133 66 0.046707 9.0
4 108 9432 6.674829 3.0
5 34 58 0.041045 24.0
6 115 1 0.000708 1.0
7 126 2528 1.789013 82.0
8 28 80 0.056614 NaN
9 103 5 0.003538 NaN
10 12 64 0.045291 3.0
11 93 15 0.010615 1.0
12 111 1936 1.370067 93.0
13 47 1 0.000708 NaN
14 132 13 0.009200 5.0
15 152 37 0.026184 NaN
16 13 41 0.029015 15.0
17 20 460 0.325532 NaN
18 57 39 0.027599 2.0
19 339 307 0.217257 3.0
20 235 293 0.207350 277.0
21 48 2 0.001415 NaN
22 61 1 0.000708 NaN
23 127 3548 2.510845 18.0
24 88 15 0.010615 5.0
25 107 1050 0.743063 1.0
26 114 4 0.002831 2.0
27 136 4 0.002831 NaN
28 129 1 0.000708 NaN
29 77 3 0.002123 NaN
30 102 86 0.060860 4.0
31 113 1566 1.108225 1297.0
32 237 141 0.099783 NaN
33 143 159 0.112521 NaN
34 338 892 0.631250 20.0
35 98 379 0.268210 1.0
36 341 2873 2.033162 6.0
37 145 11 0.007784 4.0
38 109 7441 5.265840 1.0
39 105 1016 0.719002 97.0
40 11 239 0.169135 11.0
41 33 57 0.040338 1.0
42 150 740 0.523682 702.0
43 68 8 0.005661 1.0
44 106 56804 40.198999 2048.0
45 116 3633 2.570998 NaN
46 147 1451 1.026842 1383.0
47 123 3 0.002123 1.0
48 119 284 0.200981 7.0
49 131 32 0.022646 9.0
50 149 2406 1.702676 22.0
51 124 32482 22.986830 269.0
52 46 1 0.000708 NaN
53 144 80 0.056614 74.0
54 153 79 0.055907 NaN
55 104 82 0.058030 1.0
56 36 198 0.140120 169.0

cluster_percentage
0 0.014637
1 NaN
2 51.156323
3 0.065867
4 0.021956
5 0.175644
6 0.007319
7 0.600117
8 NaN
9 NaN
10 0.021956
11 0.007319
12 0.680621
13 NaN
14 0.036593
15 NaN
16 0.109778
17 NaN
18 0.014637
19 0.021956
20 2.027225
21 NaN
22 NaN
23 0.131733
24 0.036593
25 0.007319
26 0.014637
27 NaN
28 NaN
29 NaN
30 0.029274
31 9.492096
32 NaN
33 NaN
34 0.146370
35 0.007319
36 0.043911
37 0.029274
38 0.007319
39 0.709895
40 0.080504
41 0.007319
42 5.137588
43 0.007319
44 14.988290
45 NaN
46 10.121487
47 0.007319
48 0.051230
49 0.065867
50 0.161007
51 1.968677
52 NaN
53 0.541569
54 NaN
55 0.007319
56 1.236827

# class service Distribution

compare_percentage_distribution(df_cleaned, cluster_4_df,
"class_service")

class_service total_count total_percentage cluster_count \

0 1 133998 94.827574 13004.0
1 3 138 0.097660 3.0
2 5 10 0.007077 NaN
3 4 640 0.452915 61.0
4 2 6521 4.614775 596.0

cluster_percentage
0 95.169789
1 0.021956
2 NaN
3 0.446429
4 4.361827
# Type sim Distribution
compare_percentage_distribution(df_cleaned, cluster_4_df, "type_sim")

type_sim total_count total_percentage cluster_count

cluster_percentage
0 1 10260 7.260787 161
1.178279
1 2 131047 92.739213 13503
98.821721

Categorical Columns comparison conclusion

class_client Distribution:

Class 95: Previously represented 7.10% of the total population but now only 0.64% in the
cluster, suggesting a notable decrease. Class 103: Previously represented 0.016% of the total
population but now accounts for 0.15% in the cluster, indicating a slight increase. Class 96, 72,
102, 73, 71: These classes show varying degrees of representation in the cluster compared to the
total population, with notable shifts in their percentages.

subclass_client Distribution:

Subclass 106: Previously constituted 40.20% of the total population and now stands at 14.99%
in the cluster, showing a dramatic decrease and association with the cluster. Subclass 137:
Previously made up 5.33% of the total population but is now completely absent in the cluster,
indicating a significant shift. Subclass 107, 150, 147: These subclasses show significant
representation in the cluster compared to the total population, with notable increases in their
percentages.

class_service Distribution:

No notable changes were observed, indicating that the distribution of class_service remains
relatively consistent between the total population and the cluster. The majority class (class 1)
continues to dominate both in the total population (94.83%) and the cluster (95.17%).

type_sim Distribution:

Type Sim 1: Previously made up 7.26% of the total population but now only 1.18% in the cluster,
suggesting a significant shift away from this type within the cluster. Type Sim 2: Previously
constituted 92.74% of the total population and now stands at 98.82% in the cluster, indicating a
strong association and near exclusivity within the cluster.

Conclusion
The most notable changes in Cluster 4 include the dramatic decrease in subclass_client 106
from 40.20% to 14.99%, the significant decrease of class_client 95 from 7.10% to 0.64%, and
the shift away from type_sim 1, which previously made up 7.26% of the total population. These
shifts highlight specific trends and associations within the clustered group that could inform
targeted strategies and interventions aimed at addressing the preferences and behaviors of
these segments.
Cluster 4 Persona/Behavior:
Persona:

Cluster 4 represents a diverse group of clients characterized by their significant association with
subclasses like 107, 150, and 147, which have seen substantial increases in representation within
the cluster. This cluster shows a notable shift away from type_sim 1 and a strong preference for
type_sim 2, indicating a clear preference for a specific type within the service offerings.

Behavior:

These clients exhibit a varied engagement pattern across different classes and subclasses. They
show a decreased presence of class_client 95 and an increased presence of subclass_client 107,
150, and 147. Their behavior suggests a focused interest in specific service types (type_sim 2)
while showing reduced interest in others (type_sim 1). This segmentation could imply different
usage patterns or needs within this cluster, highlighting opportunities for targeted service
enhancements or tailored marketing strategies to further engage and retain these clients.

Cluster 5
cluster_5_df = [Link]([Link] == 5)

Numercial Columns Comparison

from [Link] import lit

df_summary = df_cleaned.describe()
cluster_5_summary = cluster_5_df.describe()
common_columns = ["summary", "life_time", "nb_transaction",
"sum_achat"]
df_summary_selected =
df_summary.select(common_columns).withColumn("source",
lit("original"))
cluster_5_summary_selected =
cluster_5_summary.select(common_columns).withColumn("source",
lit("cluster_5"))
comparison_summary =
df_summary_selected.union(cluster_5_summary_selected)
comparison_summary_pd = comparison_summary.toPandas()
cols = ['source'] + [col for col in comparison_summary_pd.columns if
col != 'source']
comparison_summary_pd = comparison_summary_pd[cols]
comparison_summary_pd =
comparison_summary_pd.sort_values(by=['summary', 'source'])
comparison_summary_pd

source summary life_time nb_transaction \

5 cluster_5 count 8819 8819
0 original count 141307 141307
9 cluster_5 max 154 60
4 original max 154 174
6 cluster_5 mean 39.93275881619231 6.498695997278603
1 original mean 42.89374907117128 5.589871697792749
8 cluster_5 min 1 1
3 original min 1 1
7 cluster_5 stddev 33.395326063697254 5.767758430317427
2 original stddev 38.0621003734457 8.345795993244655

sum_achat
5 8819
0 141307
9 0.46104714
4 0.46104714
6 0.09403788259041486
1 0.020591187240199666
8 0.050701782
3 0.0
7 0.03577127354465978
2 0.027109827695412994

Numerical Columns comparison conclusion

Cluster 5 likely represents a segment of clients who are moderately engaged with the service.
Despite having a lower count of transactions and purchase amounts compared to the original
dataset, they exhibit a moderate lifetime engagement. Their mean values for lifetime, number of
transactions, and purchase amount are 39.93, 6.50, and 0.094 respectively, compared to 42.89,
5.59, and 0.021 for the original dataset.

This cluster shows variability in their engagement levels, indicating that they may be exploring
the service with varying levels of commitment or budget constraints.

Categorical Columns Comparison

# Class Client Distribution
from [Link] import col
compare_percentage_distribution(df_cleaned, cluster_5_df,
"class_client")

class_client total_count total_percentage cluster_count \

0 81 4496 3.181725 115
1 103 23 0.016277 3
2 96 1202 0.850630 87
3 92 3 0.002123 1
4 72 1073 0.759340 55
5 102 1873 1.325483 64
6 82 42 0.029723 3
7 73 260 0.183997 32
8 70 121319 85.854912 8018
9 95 10034 7.100851 385
10 83 10 0.007077 2
11 71 972 0.687864 54

cluster_percentage
0 1.304003
1 0.034017
2 0.986506
3 0.011339
4 0.623653
5 0.725706
6 0.034017
7 0.362853
8 90.917338
9 4.365574
10 0.022678
11 0.612314

# Subclass client Distribution

compare_percentage_distribution(df_cleaned, cluster_5_df,
"subclass_client")

subclass_client total_count total_percentage cluster_count \

0 148 70 0.049538 10.0
1 85 564 0.399131 12.0
2 137 7526 5.325992 242.0
3 133 66 0.046707 11.0
4 108 9432 6.674829 473.0
5 34 58 0.041045 11.0
6 115 1 0.000708 NaN
7 126 2528 1.789013 227.0
8 28 80 0.056614 4.0
9 103 5 0.003538 NaN
10 12 64 0.045291 6.0
11 93 15 0.010615 8.0
12 111 1936 1.370067 204.0
13 47 1 0.000708 NaN
14 132 13 0.009200 2.0
15 152 37 0.026184 8.0
16 13 41 0.029015 4.0
17 20 460 0.325532 10.0
18 57 39 0.027599 11.0
19 339 307 0.217257 157.0
20 235 293 0.207350 8.0
21 48 2 0.001415 1.0
22 61 1 0.000708 NaN
23 127 3548 2.510845 211.0
24 88 15 0.010615 3.0
25 107 1050 0.743063 56.0
26 114 4 0.002831 1.0
27 136 4 0.002831 1.0
28 129 1 0.000708 NaN
29 77 3 0.002123 NaN
30 102 86 0.060860 5.0
31 113 1566 1.108225 123.0
32 237 141 0.099783 13.0
33 143 159 0.112521 51.0
34 338 892 0.631250 325.0
35 98 379 0.268210 16.0
36 341 2873 2.033162 226.0
37 145 11 0.007784 6.0
38 109 7441 5.265840 368.0
39 105 1016 0.719002 64.0
40 11 239 0.169135 15.0
41 33 57 0.040338 17.0
42 150 740 0.523682 17.0
43 68 8 0.005661 NaN
44 106 56804 40.198999 3017.0
45 116 3633 2.570998 132.0
46 147 1451 1.026842 46.0
47 123 3 0.002123 NaN
48 119 284 0.200981 14.0
49 131 32 0.022646 7.0
50 149 2406 1.702676 249.0
51 124 32482 22.986830 2402.0
52 46 1 0.000708 1.0
53 144 80 0.056614 5.0
54 153 79 0.055907 2.0
55 104 82 0.058030 10.0
56 36 198 0.140120 7.0

cluster_percentage
0 0.113392
1 0.136070
2 2.744075
3 0.124731
4 5.363420
5 0.124731
6 NaN
7 2.573988
8 0.045357
9 NaN
10 0.068035
11 0.090713
12 2.313187
13 NaN
14 0.022678
15 0.090713
16 0.045357
17 0.113392
18 0.124731
19 1.780247
20 0.090713
21 0.011339
22 NaN
23 2.392562
24 0.034017
25 0.634993
26 0.011339
27 0.011339
28 NaN
29 NaN
30 0.056696
31 1.394716
32 0.147409
33 0.578297
34 3.685225
35 0.181426
36 2.562649
37 0.068035
38 4.172809
39 0.725706
40 0.170087
41 0.192766
42 0.192766
43 NaN
44 34.210228
45 1.496768
46 0.521601
47 NaN
48 0.158748
49 0.079374
50 2.823449
51 27.236648
52 0.011339
53 0.056696
54 0.022678
55 0.113392
56 0.079374

# class service Distribution

compare_percentage_distribution(df_cleaned, cluster_5_df,
"class_service")

class_service total_count total_percentage cluster_count \

0 1 133998 94.827574 8646.0
1 3 138 0.097660 1.0
2 5 10 0.007077 NaN
3 4 640 0.452915 154.0
4 2 6521 4.614775 18.0
 cluster_percentage
0 98.038326
1 0.011339
2 NaN
3 1.746230
4 0.204105

# Type sim Distribution

compare_percentage_distribution(df_cleaned, cluster_5_df, "type_sim")

type_sim total_count total_percentage cluster_count

cluster_percentage
0 1 10260 7.260787 258
2.925502
1 2 131047 92.739213 8561
97.074498

Categorical Columns comparison conclusionon

class_client Distribution:
• Class 95: Previously represented 7% of the total population but is now completely
absent in the cluster, indicating a significant shift.
• Class 109: Decreased from 7.1% to 4.37% in the cluster, suggesting a slight decrease in
representation.
• Class 8: Maintained a high presence, decreasing slightly from 85.85% to 90.92%,
indicating consistency as the dominant class.

subclass_client Distribution:
• Subclass 106: Previously constituted 40.2% of the total population and now stands at
34.21% in the cluster, showing a decrease in representation but still prominent.
• Subclass 38: Decreased from 5.27% to 4.17% in the cluster, indicating a slight decrease
in representation.
• Subclass 3: Increased significantly from 0.02% to 1.5% in the cluster, showing a notable
shift in representation.

class_service Distribution:
• No notable changes were observed, indicating that the distribution of class_service
remains relatively consistent between the total population and the cluster.

type_sim Distribution:
• Type Sim 1: Previously made up 7.26% of the total population but is now completely
absent in the cluster, suggesting a significant shift away from this type within the cluster.
• Type Sim 2: Increased from 92.74% to 97.07% in the cluster, indicating a strong
association with this type within the cluster.
Conclusion
The most notable changes in the cluster include the complete absence of class_client 95
and type_sim 1, as well as the significant decrease in representation of subclass_client
106 and increase in subclass_client 3. These shifts highlight specific trends and
associations within the clustered group that could inform targeted strategies and interventions.
Understanding these changes can guide personalized approaches and optimizations tailored to
different client and service types within the cluster.

Cluster 5 Persona/Behavior:
Persona:

Cluster 5 comprises clients who exhibit moderate engagement levels with the service. They
show variability in their categorical profiles, indicating diverse preferences across different client
and service type categories.

Behavior:

These clients demonstrate a consistent pattern of moderate engagement with the service. They
are characterized by a balanced approach in their interaction, reflecting a mix of exploration and
commitment to the service offerings.