Scribd
Upload
7 views4 pages

Lexical Analysis with Python Examples

The document provides examples of lexical analysis in Python using NLTK and SpaCy for word and sentence tokenization, as well as custom tokenization methods. It includes code snippets for various tokenization techniques, such as using regular expressions and classifying tokens by type. Additionally, it demonstrates how SpaCy can be utilized for advanced lexical analysis, including part-of-speech tagging and dependency parsing.

Uploaded by

krishna Jyothi
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
7 views4 pages

Lexical Analysis with Python Examples

The document provides examples of lexical analysis in Python using NLTK and SpaCy for word and sentence tokenization, as well as custom tokenization methods. It includes code snippets for various tokenization techniques, such as using regular expressions and classifying tokens by type. Additionally, it demonstrates how SpaCy can be utilized for advanced lexical analysis, including part-of-speech tagging and dependency parsing.

Uploaded by

krishna Jyothi
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Lexical Analysis in Python – Simple Examples

1. Word Tokenization using NLTK

📌 Code:

import nltk
from [Link] import word_tokenize

[Link]('punkt') # Download tokenizer models

text = "Lexical analysis breaks text into tokens."

# Tokenize into words


tokens = word_tokenize(text)
print("Word Tokens:", tokens)

Output:

Word Tokens: ['Lexical', 'analysis', 'breaks', 'text', 'into', 'tokens', '.']

Explanation: Breaks sentence into words and punctuation.

2. Sentence Tokenization using NLTK

Code:

from [Link] import sent_tokenize

text = "Lexical analysis is important. It is the first step in NLP."

sentences = sent_tokenize(text)
print("Sentence Tokens:", sentences)
Output:

['Lexical analysis is important.', 'It is the first step in NLP.']

Explanation: Breaks paragraph into individual sentences.

3. Tokenization using Regular Expressions

Code:

from [Link] import RegexpTokenizer

text = "Don't forget to tokenize numbers like 100 and punctuation too!"

tokenizer = RegexpTokenizer(r'\w+')
tokens = [Link](text)
print("Regex Tokens:", tokens)

Output:

['Don', 't', 'forget', 'to', 'tokenize', 'numbers', 'like', '100', 'and', 'punctuation',
'too']

Explanation: Breaks based on word patterns (\w+ matches letters and


numbers).

4. Custom Tokenizer using split()

Code:

text = "This is a simple lexical analyzer."


tokens = [Link]() # Splits on spaces
print("Tokens:", tokens)
Output:

['This', 'is', 'a', 'simple', 'lexical', 'analyzer.']

Explanation: Basic whitespace tokenizer; doesn't separate punctuation.

5. Token Classification (Lexeme to Token Type)

Code:

def classify_token(token):
if [Link]():
return "WORD"
elif [Link]():
return "NUMBER"
elif token in ['.', ',', '!', '?']:
return "PUNCTUATION"
else:
return "UNKNOWN"

tokens = ['Hello', '123', '.', '@home']

classified = [(token, classify_token(token)) for token in tokens]


print("Token Types:", classified)

Output:

[('Hello', 'WORD'), ('123', 'NUMBER'), ('.', 'PUNCTUATION'), ('@home',


'UNKNOWN')]

Explanation: Shows how tokens are categorized into types like WORD,
NUMBER, etc.
6. Using SpaCy for Lexical Analysis (Advanced)

Code:

import spacy

nlp = [Link]("en_core_web_sm")
text = "Apple is looking at buying U.K. startup for $1 billion."

doc = nlp(text)
for token in doc:
print(f"{[Link]:<12} {token.pos_:<10} {token.dep_:<10}")

Output:

Sample output with token text, POS tags, and dependency parsing

Explanation: SpaCy not only tokenizes but also shows POS tags and
dependency.

You might also like