Scribd
Upload
2 views1 page

Program 1

The document provides a Python program that utilizes the NLTK library for tokenization of text into sentences and words. It includes installation instructions for NLTK, downloading necessary tokenizers, and demonstrates how to perform both sentence and word tokenization on a sample text about Natural Language Processing. The output displays the results of the tokenization processes.

Uploaded by

galetilohitha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
2 views1 page

Program 1

The document provides a Python program that utilizes the NLTK library for tokenization of text into sentences and words. It includes installation instructions for NLTK, downloading necessary tokenizers, and demonstrates how to perform both sentence and word tokenization on a sample text about Natural Language Processing. The output displays the results of the tokenization processes.

Uploaded by

galetilohitha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

1.

Write a python program to perform tokenization by word and sentence using NLTK

# Install NLTK (Run this only once in Google Colab)


!pip install nltk

# Import NLTK
import nltk

# Download required tokenizer


[Link]('punkt')
[Link]('punkt_tab') # Required for newer NLTK versions

# Import tokenizers
from [Link] import word_tokenize, sent_tokenize

# Input text
text = """
Natural Language Processing (NLP) is a branch of Artificial Intelligence.
It helps computers understand human language.
Python provides the NLTK library for NLP tasks.
Tokenization is the first step in text preprocessing.
"""

# -------------------------------
# Sentence Tokenization
# -------------------------------
sentences = sent_tokenize(text)

print("Sentence Tokenization:")
for i, sentence in enumerate(sentences, start=1):
print(f"{i}. {sentence}")

# -------------------------------
# Word Tokenization
# -------------------------------
words = word_tokenize(text)

print("\nWord Tokenization:")
print(words)

You might also like