Scribd
Upload
13 views3 pages

k-NN Classification of Iris Dataset

The document outlines a program that implements the k-Nearest Neighbors (k-NN) algorithm to classify flowers in the Iris dataset. It evaluates the model's performance using accuracy and F1-score for different values of k, both for regular and weighted k-NN. The results show high accuracy and F1-scores for both methods across various k values.

Uploaded by

1bi22ai010
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
13 views3 pages

k-NN Classification of Iris Dataset

The document outlines a program that implements the k-Nearest Neighbors (k-NN) algorithm to classify flowers in the Iris dataset. It evaluates the model's performance using accuracy and F1-score for different values of k, both for regular and weighted k-NN. The results show high accuracy and F1-scores for both methods across various k values.

Uploaded by

1bi22ai010
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Program 4

Develop a program to load the Iris dataset. Implement the k-Nearest Neighbors (k-NN)
algorithm for classifying flowers based on their features. Split the dataset into training and
testing sets and evaluate the model using metrics like accuracy and F1-score. Test it for
different values of 𝑘 (e.g., k=1,3,5) and evaluate the accuracy. Extend the k-NN algorithm to
assign weights based on the distance of neighbors (e.g., 𝑤𝑒𝑖𝑔ℎ𝑡=1/𝑑2 ). Compare the
performance of weighted k-NN and regular k-NN on a synthetic or real-world dataset.

import numpy as np
import [Link] as plt
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from [Link] import accuracy_score, f1_score
from collections import Counter
#distance formula
def euclidean_distance(x1, x2):
return [Link]([Link]((x1 - x2) ** 2))

class KNN:
def __init__(self, k=3, weighted=False):
self.k = k
[Link] = weighted

def fit(self, X_train, y_train):


self.X_train = X_train
self.y_train = y_train

def predict(self, X_test):


predictions = [self._predict(x) for x in X_test]
return [Link](predictions)

def _predict(self, x):


distances = [euclidean_distance(x, x_train) for x_train in self.X_train]
k_indices = [Link](distances)[:self.k]
k_nearest_labels = [self.y_train[i] for i in k_indices]

if [Link]:
weights = [1 / (distances[i] ** 2 + 1e-5) for i in k_indices]
class_votes = {}
for label, weight in zip(k_nearest_labels, weights):
class_votes[label] = class_votes.get(label, 0) + weight
return max(class_votes, key=class_votes.get)
else:
most_common = Counter(k_nearest_labels).most_common(1)
return most_common[0][0]
# Load the Iris dataset
iris = load_iris()
X, y = [Link], [Link]
# Standardize features
scaler = StandardScaler()
X = scaler.fit_transform(X)
# Split dataset into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Evaluate for different k values
k_values = [1, 3, 5]
for k in k_values:
knn = KNN(k=k, weighted=False)
[Link](X_train, y_train)
y_pred = [Link](X_test)
acc = accuracy_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred, average='weighted')
print(f'k={k}, Regular k-NN -> Accuracy: {acc:.4f}, F1-score: {f1:.4f}')
# Evaluate Weighted k-NN
for k in k_values:
knn_weighted = KNN(k=k, weighted=True)
knn_weighted.fit(X_train, y_train)
y_pred_weighted = knn_weighted.predict(X_test)
acc_weighted = accuracy_score(y_test, y_pred_weighted)
f1_weighted = f1_score(y_test, y_pred_weighted, average='weighted')
print(f'k={k}, Weighted k-NN -> Accuracy: {acc_weighted:.4f}, F1-score: {f1_weighted:.4f}')

output:
k=1, Regular k-NN -> Accuracy: 0.9667, F1-score: 0.9664
k=3, Regular k-NN -> Accuracy: 1.0000, F1-score: 1.0000
k=5, Regular k-NN -> Accuracy: 1.0000, F1-score: 1.0000
k=1, Weighted k-NN -> Accuracy: 0.9667, F1-score: 0.9664
k=3, Weighted k-NN -> Accuracy: 1.0000, F1-score: 1.0000
k=5, Weighted k-NN -> Accuracy: 1.0000, F1-score: 1.0000

Common questions

Powered by AI

When choosing the optimal value of 'k' for a k-NN model, considerations might include the nature of the data (e.g., noise levels and distribution), computational resources (as larger values of 'k' require more computation), and desired balance between bias and variance (small 'k' may lead to low bias but high variance, and vice versa). Cross-validation can be used to empirically determine the best 'k' by evaluating performance metrics like accuracy and F1-score across different 'k' values. Domain-specific knowledge about the dataset and problem context can also guide the selection of 'k'.

As demonstrated by the test results, changing the value of 'k' in the k-NN algorithm affects its performance. For k=1, there was a slight reduction in accuracy (0.9667 instead of 1.0000 for k=3 and k=5). This suggests that a smaller 'k' might be more sensitive to noise in the data and potentially overfit, whereas larger values, like k=3 and k=5, resulted in perfect classification performance, likely due to better generalization across the test set.

The Euclidean distance metric is used in the k-NN algorithm to measure the similarity (or dissimilarity) between instances. By calculating the straight-line distance between feature vectors, it helps determine which neighbors are closest to a given test instance. It is widely used due to its simplicity and effectiveness in capturing natural notions of similarity in many problem domains.

The regular k-NN algorithm can be sensitive to noise and outliers, as each neighbor in the 'k' selected contributes equally to the classification decision, potentially magnifying the impact of any anomalous instances. The weighted k-NN algorithm mitigates this effect by assigning lower influence to neighbors that are farther away, thus reducing the influence of outliers in the decision-making process. This generally makes the weighted version less sensitive to noise.

In the provided k-NN implementation, ties in voting are implicitly handled by returning the most common class among the nearest neighbors. The Counter class from the Python collections library is used to count occurrences of each class label among the nearest neighbors, and the most_common() method is then employed to determine the label with the highest frequency. If there is a tie, Python’s Counter resolves it by returning the first one encountered with the highest count, ensuring deterministic behavior.

For the values of k (1, 3, 5) tested, both regular and weighted k-NN algorithms showed similar performance in terms of accuracy and F1-score. For k=1, both achieved an accuracy of 0.9667 and an F1-score of 0.9664. For k=3 and k=5, both methods achieved perfect accuracy and F1-scores of 1.0000.

The k-Nearest Neighbors (k-NN) algorithm classifies a test instance by identifying the 'k' closest training instances to that instance, using a distance metric like Euclidean distance. It assigns the class label that is most common among those 'k' neighbors to the test instance. The process can be adjusted to incorporate weights by assigning each neighbor a weight inversely proportional to its distance from the test instance, specifically using a formula such as weight = 1/(d^2), where 'd' is the Euclidean distance. This allows closer neighbors to have a higher influence on the classification result.

Weighted k-NN can offer advantages over regular k-NN when dealing with datasets where the proximity of neighbors is a more reliable indicator of class similarity than their sheer number. In cases where the feature space is unevenly populated or where closer neighbors are more indicative of the decision boundary, the weight applied to each neighbor inversely proportional to its distance allows the model to emphasize these more relevant neighbors, leading to improved predictive performance.

Feature standardization benefits the k-NN algorithm by ensuring that all features contribute equally to the distance calculations, preventing features with larger ranges from dominating the calculation. In practice, this is achieved by scaling the features to have a mean of 0 and a standard deviation of 1, using tools like the StandardScaler from the sklearn library in Python.

Using numpy and libraries like sklearn offers several advantages when implementing a k-NN algorithm. Numpy provides efficient array operations and mathematical functions, facilitating quick computation of distances. Sklearn offers robust data preprocessing tools such as StandardScaler for feature standardization and modules for dataset handling, performance evaluation, and model development, significantly reducing the time and effort needed to implement and test machine learning models.

You might also like