Adaptive clustering algorithm for
determining the optimal partition size
Clustering is a machine learning technique for grouping similar objects together. Given a set of
objects, we can use a clustering algorithm to classify each object into a specific group. Objects
that belong to the same group should have similar, while objects in different groups have highly
dissimilar properties.
Clustering is a method of unsupervised learning, where the data we want to describe is not
labeled. We do not know much information about the expected outcome. The algorithm only has
the data and it should perform clustering in the best way possible.
The agglomerative clustering is the most common type of hierarchical clustering used to group
objects in clusters based on their similarity. It is a bottom-up approach that treats each object as
a singleton cluster, and then successively merges pairs of clusters until all clusters have been
merged into a single cluster that contains all objects.
In order to decide which clusters should be combined, a measure of dissimilarity between
groups of objects is required. This is achieved by using an appropriate metric (a measure of
distance between two objects), and a linkage criterion which specifies the distance between two
clusters.
Hierarchical clustering does not require a prespecified number of clusters. We have the
possibility to select which number of clusters fits our input data the most since the algorithm
builds a tree. However, in some cases we may want to stop merging clusters together at a given
point to save computational resources.
In these cases, a number of stopping criteria can be used to determine the cutting point:
number criterion and distance criterion.
It is a rather difficult problem to determine the optimal number of clusters in an input data,
regardless of the clustering method we choose to use. There are several methods in the
literature that provide a solution to this problem: the elbow method and the silhouette method.