Scribd
Upload
2 views4 pages

Program 2

The document outlines the implementation of basic Natural Language Processing techniques, specifically a Levenshtein Distance algorithm to measure word differences and a custom tokenizer for text preprocessing. The Levenshtein Distance algorithm uses a 2D matrix to calculate edit distances between two words, while the custom tokenizer removes punctuation, converts text to lowercase, and splits sentences into tokens. The provided Python program includes user input for both functionalities and displays the results.

Uploaded by

24f2000672
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
2 views4 pages

Program 2

The document outlines the implementation of basic Natural Language Processing techniques, specifically a Levenshtein Distance algorithm to measure word differences and a custom tokenizer for text preprocessing. The Levenshtein Distance algorithm uses a 2D matrix to calculate edit distances between two words, while the custom tokenizer removes punctuation, converts text to lowercase, and splits sentences into tokens. The provided Python program includes user input for both functionalities and displays the results.

Uploaded by

24f2000672
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Aim :

To implement basic Natural Language Processing (NLP) techniques by developing:

1. A Levenshtein Distance algorithm to measure the difference between two words.

2. A custom tokenizer to preprocess text by removing punctuation and splitting


sentences into meaningful words.

Algorithm:
Edit Distance (Levenshtein Distance)

1. Start the program and input two words from the user.

2. Create a 2D matrix based on the lengths of both words.

3. Initialize the first row and first column with incremental values (representing
insertions and deletions).

4. Compare characters of both words one by one.

5. If characters match, copy the diagonal value; otherwise, take 1 plus the minimum of
insert, delete, and replace operations.

6. The final value in the last cell of the matrix gives the edit distance.

Custom Tokenizer

1. Input a sentence from the user.

2. Remove all punctuation marks from the sentence.

3. Convert the text into lowercase.

4. Split the sentence into words using whitespace.

5. Return and display the list of tokens.


Program
import string

# -------------------------------

# 1. Levenshtein Distance Function

# -------------------------------

def edit_distance(s1, s2):

m = len(s1)

n = len(s2)

dp = [[0 for _ in range(n + 1)] for _ in range(m + 1)]

# Initialize

for i in range(m + 1):

dp[i][0] = i

for j in range(n + 1):

dp[0][j] = j

# Fill DP table

for i in range(1, m + 1):

for j in range(1, n + 1):

if s1[i - 1] == s2[j - 1]:

dp[i][j] = dp[i - 1][j - 1]

else:

dp[i][j] = 1 + min(

dp[i - 1][j], # delete

dp[i][j - 1], # insert

dp[i - 1][j - 1] # replace


)

return dp[m][n]

# -------------------------------

# 2. Custom Tokenizer

# -------------------------------

def custom_tokenizer(text):

text = [Link]([Link]("", "", [Link]))

text = [Link]()

tokens = [Link]()

return tokens

# -------------------------------

# MAIN PROGRAM (USER INPUT)

# -------------------------------

if __name__ == "__main__":

print("\n=== NLP LAB PROGRAM (USER INPUT VERSION) ===")

# ----- Edit Distance -----

word1 = input("\nEnter first word: ")

word2 = input("Enter second word: ")

print("\n--- Edit Distance Result ---")

print("Word 1:", word1)


print("Word 2:", word2)

print("Edit Distance:", edit_distance(word1, word2))

# ----- Tokenizer -----

sentence = input("\nEnter a sentence: ")

print("\n--- Tokenizer Result ---")

print("Input:", sentence)

print("Tokens:", custom_tokenizer(sentence))

Output:

Enter first word: kitten


Enter second word: sitting

Edit Distance: 3

Enter a sentence: Hello! NLP is fun, isn't it?

Tokens: ['hello', 'nlp', 'is', 'fun', 'isnt', 'it']

You might also like