Scribd
Upload
5 views13 pages

Lexical Analysis and Tokenization Guide

The document describes the lexical analysis of a program through class and object diagrams. It includes class diagrams showing the Symbol Table, Lexical Analyzer, and MyFrame classes used in the analysis. Flow diagrams demonstrate the process of lexical analysis from the source program through token output. Collaboration and sequence diagrams illustrate the interaction between the programmer, input/output screens, and lexical analyzer class.

Uploaded by

Jaswant Malviya
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
5 views13 pages

Lexical Analysis and Tokenization Guide

The document describes the lexical analysis of a program through class and object diagrams. It includes class diagrams showing the Symbol Table, Lexical Analyzer, and MyFrame classes used in the analysis. Flow diagrams demonstrate the process of lexical analysis from the source program through token output. Collaboration and sequence diagrams illustrate the interaction between the programmer, input/output screens, and lexical analyzer class.

Uploaded by

Jaswant Malviya
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOC, PDF, TXT or read online on Scribd

OBJRCT DIAGRAM

Stdem o String input="int a,b; float c;"; S tring key="int a,b; float c;", void init(); void analysis(String); void m ain(); void breakdatatype("int a,b");

Myframe JTextA rea="int a,b; floar c;" Myfram e(); void actionP erformedActionE vent); S tring getS tring(File);

S tringTokenizer S tringTokenizer("int a,b;"); hasM oreTokens(); nextToken(); countTokens()

String indexOf(int); valueAt(int); charAt(int); length(); subString(S tring); compareTo();

Datatype int,float Identifier a,b,c

Class Diagram
Symbol Table Lineno.,Token,Type addRow();

Lexical Myframe mf; Myframe(); void main();

Myframe JButton,JTextArea,JmenuBar, JPanel,JFileChooser Myframe(); void actionPerformedActionEvent); String getString(File); Read Source File

Stdemo String input,in,key,a,ch; void init(); void analysis(String); void main(); void breakdatatype(String); void breakcontrolkeywords(String); void breakgeneralkeywords(String); void breakinbuiltfunction(String); void breakbracket(String); void breakuserfunction(String);

String indexOf(int); valueAt(int); charAt(int); length(); subString(String); compareTo();

StringTokenizer StringTokenizer(); hasMoreTokens(); nextToken(); countTokens()

Root of Abstract Tokens

Flow Chart

Start

Source Program

To Input Screen

Lexical Analysis

Tokens

To Output Screen Tabular Form of Tokens

End

Flow Diagram
For User Input

Interface

For Edit TextArea For New Program

Analysis Lexical Analysis Tokens Output Table

Token in Tabular Format

Context diagram

Programmer

To Parser

stream of token input logic Lexical Analyser o/p screen

Input screen

Table of Token

Test Model

"if(x>3.1) printf("Hello"); " Text stream Lexical Analyzer

Token stream IF-Keyword Bracket L Identifier X OP ">" Number 3.1 bracket -R

SWIMLANE DIAGRAM
Start

Input screen

Write program

Decision

Lexical Analysis

Edit

Output screen

Token

Finish

System Architecture
Instantiate Lexical Analyzer

Find reserve word

yes

Add reserve word

no Find operator yes Add operator

Token request

more token

o/p next token

no

no

System Overview

Input Screen

Interface

Use Classes

Input Screen

Lexical Analyzer

Output Token

Collaboration diagram
String tokenizer

write program

Lexical Analyzer

Programmer

i/p screen

o/p token

i/p program Tabular format

Tokens

Activity diagram

Programmer

Write program

Inspection

Error in program

Decision

Input source code

Lexical analyser

Token

to parser

Sequence diagram

Programmer

Lexical Analyser

System (classes)

Input program

Use classes

Java classes Output Token

Usecase diagram
Open New Save

Construct Program

Lexical Analysis

User

View Tabular Form

You might also like