NPTEL Data Analytics with Python Overview
Uploaded by
D BarikNPTEL Data Analytics with Python Overview
Uploaded by
D Barik- Assignment Overview
- Questionnaire
Common questions
Powered by AIPredefining the number of clusters in K-means can significantly impact the clustering outcome since an inappropriate choice can lead to suboptimal clusters. This requirement contrasts with methods like hierarchical clustering, which determine clusters as part of the process and adaptively find the structure in data. The predetermined number of clusters influences K-means' performance in terms of convergence and computational efficiency but may overlook the data's natural structure .
The possible termination conditions in the K-means clustering algorithm include: a fixed number of iterations; no change in observations' assignment to clusters between iterations (except in cases of a bad local minimum); no change in centroids between successive iterations; and when the residual sum of squares (RSS) falls below a pre-defined threshold .
The main difference between partitional clustering and hierarchical clustering is the approach used to form clusters. Partitional clustering, like K-means, assigns data points to a fixed number of clusters, seeking to optimize clusters in a flat structure. In contrast, hierarchical clustering involves creating a tree-based representation (dendrogram) of clusters by either merging smaller clusters into larger ones (agglomerative) or dividing a single cluster into smaller clusters (divisive).
Hierarchical clustering techniques can be agglomerative or divisive based on the direction of building the dendrogram. Agglomerative clustering starts with each data point as a separate cluster and merges clusters iteratively, while divisive clustering starts with all data points in a single cluster and divides them into smaller clusters. Both approaches reflect a bottom-up and top-down strategy respectively for organizing the data .
The average-link metric in hierarchical clustering calculates the dissimilarity between clusters by averaging the pairwise distances between all pairs of objects from the two clusters. Unlike single-link or complete-link metrics, it provides a more balanced measure of cluster similarity, reducing sensitivity to noise and outliers .
Hierarchical clustering is most useful for exploratory data analysis because it does not require a preset number of clusters and provides a visual representation of how data points are grouped together at various levels of similarity. This flexibility helps in gaining insights into the inherent structure of the data .
The formula for computing dissimilarity between two objects for categorical variables is given by D(i,j) = (p - m) / p, where p denotes the number of parameters, and m represents the number of matches between the two objects .
K-means clustering is a partitioning method because it divides the dataset into mutually exclusive subsets so that the similarity within a subset is higher compared to that between different subsets. It measures cluster similarity based on the mean value of the objects within each cluster, which serves as the cluster's centroid .
Standard deviation is more affected by an outlier than mean absolute deviation due to the squaring of differences from the mean, which amplifies the influence of outliers more than the absolute differences used in mean absolute deviation. The larger squared values of deviations contribute significantly to higher standard deviations in the presence of outliers, making it more responsive to extreme values .
In a dataset, the presence of an outlier affects the standard deviation more significantly than it does the mean absolute deviation. This occurs because standard deviation measures dispersion based on the squared differences from the mean, which amplifies the effect of extreme values like outliers .
