Python Tokens
Definition:
• In Python, tokens are the smallest units of a program.
Just like words make sentences in English, tokens make Python
programs.
• Whenever we write a Python program, the Python interpreter breaks
it into tokens before execution.
Types of Tokens in Python
• Keywords
• Identifiers
• Literals
• Operators
• Punctuators / Separators
1. Keywords
• Keywords are reserved words in Python.
• They have special meaning and cannot be used as variable names.
• Examples: if, else, while, for, break, True, False, None, etc.
• Example:
if True:
print("This is a keyword example")
2. Identifiers
• Identifiers are names used to identify variables, functions, classes, etc.
• Rules for identifiers:
1. Must start with a letter (a-z, A-Z) or an underscore (_).
2. Can contain letters, digits (0-9), and underscore.
3. Cannot use keywords as identifiers.
4. Python is case-sensitive (Var and var are different).
Example
name = "Priyanshu" # identifier = name
_age = 22 # identifier = _age
3. Literals
• Literals are fixed values assigned to variables.
• Types of literals in Python:
• String Literals – "Hello", 'World'
• Numeric Literals
• Integer → 10, -5
• Float → 3.14, -0.99
• Complex → 2+3j
• Boolean Literals – True, False
• Special Literal – None
4. Operators
• Operators are symbols used to perform operations.
• Types of operators:
1. Arithmetic → +, -, *, /, %, **, //
2. Relational → ==, !=, >, <, >=, <=
3. Logical → and, or, not
4. Assignment → =, +=, -=, *=, /=
5. Bitwise → &, |, ^, ~, <<, >>
6. Membership → in, not in
5. Punctuators (Separators)
• These are symbols used to structure Python programs.
Symbol Meaning
() Parentheses (function call, tuples)
[] Square brackets (lists, indexing)
{} Curly braces (dictionaries, sets)
: Colon (used in loops, functions, classes)
, Comma (separates values)
. Dot (object/member access)
; Semicolon (rarely used, separates statements on one line)
# Comment symbol
""" """ Triple quotes (multi-line string or docstring)
Blocks in Python
• A block is a group of statements that are executed together.
• In Python, blocks are defined by indentation (spaces or tabs) instead
of {} like in C, C++, or Java.
• Example: functions, loops, if-else statements all contain blocks.
•
# Example of block inside if
x = 10
if x > 5:
print("x is greater than 5") # Block starts here
print("This also belongs to the same block")
print("Outside the block") # Not part of if block
Indentation in Python
• Indentation means spaces at the beginning of a line.
• In Python, indentation is used to define blocks of code.
• Without proper indentation, Python will give an IndentationError.
• Example
for i in range(3):
print("Hello") # This line is indented → inside loop
print("World") # Same block
print("Done") # Not indented → outside loop
Character Set in Python
• Character set is the set of valid characters that can be used in Python
programs.
• Python supports Unicode, so it can handle many languages and symbols.
Characters allowed in Python:
• Letters: A-Z, a-z
• Digits: 0-9
• Special symbols: + - * / % = ( ) [ ] { } @ # , . : etc.
• Whitespace characters: space, tab, newline
• Other symbols: Unicode characters like π, √