Scribd
Upload
27 views5 pages

Python NLTK Parsing Techniques

This document contains 3 code samples demonstrating natural language processing techniques using the NLTK library in Python. The first sample performs morphological analysis using stemming to extract word roots. The second sample performs lexical analysis using tokenization and lemmatization. The third sample generates parse trees using a context-free grammar to represent syntactic structure.

Uploaded by

Joven Ramos
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
27 views5 pages

Python NLTK Parsing Techniques

This document contains 3 code samples demonstrating natural language processing techniques using the NLTK library in Python. The first sample performs morphological analysis using stemming to extract word roots. The second sample performs lexical analysis using tokenization and lemmatization. The third sample generates parse trees using a context-free grammar to represent syntactic structure.

Uploaded by

Joven Ramos
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Sample Program using Python 3.

Sample 1: Morphological Analysis using NLTK Library


You can find the code on this link:
[Link]

# This script gives you an idea how stemming has been placed by using NLTK library.
# It is part of morphological analysis

from [Link] import PorterStemmer

word = "unexpected"
text = "disagreement"
text1 = "disagree"
text2 = "agreement"
text3 = "quirkiness"
text4 = "historical"
text5 = "canonical"
text6 = "happiness"
text7 = "unkind"
text8 = "dogs"
text9 = "expected"

def stemmer_porter():
port = PorterStemmer()
print("\nDerivational Morphemes")
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print ("\nInflectional Morphemes")
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print ("\nSome examples")
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))
print (" ".join([[Link](i) for i in [Link]()]))

if __name__ == "__main__":
stemmer_porter()

Sample 2: Lexical Analysis using Tokenization and Lemmatization of NLTK


Library
You can find the code on this link:
[Link]

# This script gives you an idea how tokenization and lemmatization has been placed by using NLTK.
# It is part of lexical analysis
from [Link] import word_tokenize
from [Link] import WordNetLemmatizer

def wordtokenization():
content = """Stemming is funnier than a bummer says the sushi loving computer scientist.
She really wants to buy cars. She told me angrily. It is better for you.
Man is walking. We are meeting tomorrow. You really don't know..!"""
print (word_tokenize(content))
def wordlemmatization():
wordlemma = WordNetLemmatizer()
print ([Link]('cars'))
print ([Link]('walking',pos='v'))
print ([Link]('meeting',pos='n'))
print ([Link]('meeting',pos='v'))
print ([Link]('better',pos='a'))
print ([Link]('is',pos='v'))
print ([Link]('funnier',pos='a'))
print ([Link]('expected',pos='v'))
print ([Link]('fantasized',pos='v'))

if __name__ =="__main__":
wordtokenization()
print ("\n")
print ("----------Word Lemmatization----------")
wordlemmatization()

Sample 3: Syntactical Analysis using Parse Tree Algorithm of NLTK Library


You can find the code on this link:
[Link]

# This script is for generating parsing tree by using NLTK.


# NLTK gives us tree representation of stanford parser.
import nltk
from nltk import CFG
from [Link] import *
from collections import defaultdict
# Part 1: Define a grammar and generate parse result using NLTK
def definegrammar_pasrereult():
Grammar = [Link]("""
S -> NP VP
PP -> P NP
NP -> Det N | Det N PP | 'I'
VP -> V NP | VP PP
Det -> 'an' | 'my'
N -> 'elephant' | 'pajamas'
V -> 'shot'
P -> 'in'
""")
sent = "I shot an elephant".split()
parser = [Link](Grammar)
trees = [Link](sent)
for tree in trees:
print(tree)

# Part 2: Draw the parse tree


def draw_parser_tree():
dp1 = Tree('dp', [Tree('d', ['the']), Tree('np', ['dog'])])
dp2 = Tree('dp', [Tree('d', ['the']), Tree('np', ['cat'])])
vp = Tree('vp', [Tree('v', ['chased']), dp2])
tree = Tree('s', [dp1, vp])
print(tree)
print(tree.pformat_latex_qtree())
tree.pretty_print()

if __name__ == "__main__":
print("\n--------Parsing result as per defined grammar-------")
definegrammar_pasrereult()
print("\n--------Drawing Parse Tree-------")
draw_parser_tree()

Common questions

Powered by AI

A parse tree captures the syntactic structure of a sentence according to a specified grammar, detailing the relationships between words and phrases in a hierarchical form. This representation is significant for NLP because it facilitates the understanding of a sentence's grammatical structure, enabling more advanced tasks such as semantic analysis, machine translation, and syntactic error correction. NLTK uses a formal grammar to create parse trees that visually represent these structures .

Lexical analysis through tokenization and lemmatization simplifies text data and unifies word forms. Tokenization breaks down text into manageable pieces, while lemmatization normalizes words to their canonical forms, reducing redundancy and improving the relevancy of search and analytic queries. These processes enable more effective information retrieval and sentiment analysis by standardizing data for machine learning models, thus enhancing NLP tasks .

Stemming and lemmatization are both techniques used to reduce words to their base or root form. Stemming cuts off derived or inflected word endings to arrive at the root form, often producing non-existent words. For instance, the NLTK's PorterStemmer reduces 'cars' to 'car', 'walking' to 'walk'. In contrast, lemmatization considers the morphological analysis of words, returning the base form, which is a valid word, as seen in the lemmatization of 'better' to 'good'. NLTK implements lemmatization using the WordNetLemmatizer .

Parse trees represent the syntactic structure of a sentence based on a formal grammar. Using NLTK, the parse tree illustrates hierarchical relationships between different parts of a sentence, such as phrases and clauses. This visual representation helps in understanding how a sentence is constructed, showing the roles of each word and phrase in the context of the overall sentence. It aids in syntactic analysis, machine translation, and grammar checking .

Morphological analysis through stemming reduces words to their base forms, which helps in text processing by simplifying the data. This benefits information retrieval systems by consolidating various word forms and making it easier to match user queries with relevant documents. The efficiency in matching increases due to the reduction in word variability, enhancing search accuracy and retrieval speed .

Lematization returns the base form of a word by analyzing its context and inflection, which requires understanding its part of speech. For example, 'meeting' is lemmatized to 'meet' when used as a verb but remains 'meeting' as a noun. This is crucial as it maintains the correct word form depending on its syntactic role, providing a more accurate understanding of text data while reducing ambiguity in NLP tasks such as translation and sentiment analysis .

Tokenization divides text into individual units called tokens, typically words, which allows for more manageable and analyzable snippets of data. This is beneficial for various NLP tasks like parsing and information retrieval. In the provided content, tokenization enables the breaking down of complex sentences into a sequence of discrete words, facilitating further processing like lemmatization or morphological analysis .

Defining grammar is crucial as it establishes the framework rules that dictate how the parse tree of a sentence is constructed. It ensures that each sentence element is placed correctly according to syntactic rules. In NLTK, a defined grammar allows the parser to recognize parts of speech and hierarchical structures, making it possible to analyze complex sentence constructions logically and consistently, which is essential for applications like language modeling and parsing in computational linguistics .

Parse trees offer a hierarchical visual representation of sentence structure, which makes understanding and analyzing complex syntax more intuitive. They provide clear delineations between subordinate and coordinate clauses, and the roles of sentence constituents, which is more challenging with other syntactic structures. Parse trees allow detailed analyses and are beneficial for linguistic theory testing, automated parsing systems, and computational linguistics research. They enable the observation of recursive sentence structures, enhancing complex grammar understanding .

The PorterStemmer reduces 'happiness' to 'happi' and 'unkind' to 'unkind'. This approach demonstrates how PorterStemmer simplifies words by removing derivational morphemes such as suffixes to reveal a base or root form, though it may not always return a valid English word. This process assists in grouping words with similar meanings and parts of speech .

You might also like