Python Tokens and Character Sets

In Python, every program is formed using valid characters and tokens. The character set
defines which characters are allowed in a Python program, while tokens represent the
smallest meaningful units such as keywords, identifiers, literals, operators, and symbols.

Character Set

A character set is the collection of valid characters that a programming language

understands. Python supports a wide range of characters, making it flexible and easy to
use. Python character set includes:

• Alphabets: A–Z, a–z

• Digits: 0–9

• Special symbols: + - * / % = @ # $ & _ etc.

• Whitespace characters: space, tab, newline

• Unicode characters: Python supports full Unicode

These characters are used to form keywords, variables, expressions and statements.

Tokens

A token is the smallest meaningful unit in a Python program. Python code is interpreted by
breaking it into tokens. Python has the following types of tokens:

1. Keywords

Keywords are reserved words with special meaning in Python. They cannot be used as
variable or function names. Examples of keywords: if, else, for, while, break, continue, True,
False, import, class

for x in range(1, 6):

if x < 4:

continue

break

Here, for, if, continue and break are keywords.

2. Identifiers

Identifiers are names given to variables, functions, classes, etc. Rules for identifiers:
 • Can contain letters, digits, and _

• Cannot start with a digit

• Cannot be a keyword

• Case-sensitive

name = "Python"

_count = 10

# Valid identifiers: name, _count

# Invalid identifiers: 2num, my-name, for

3. Literals (Values)

Literals are the fixed values or data items used in a source code. Python supports different
types of literals:

3.1 String Literals

Represent text values enclosed in quotes.

msg = "Hello Python"

print(msg)

Output: Hello Python

3.2 Numeric Literals

Represent integer or decimal numbers.

a = 10

b = 3.5

print(a)

print(b)

Output:
10
3.5

3.3 Boolean Literals

Represent logical values: True or False.

is_valid = True

3.4 Special Literal

None represents the absence of a value.

value = None

3.5 Collection Literals

Represent grouped data such as lists, tuples, dictionaries and sets.

a = [1, 2, 3] # List

tup = (1, 2) # Tuple

d = {"a": 1} # Dictionary

s = {1, 2, 3} # Set

print(a)

print(tup)

print(d)

print(s)

Output:
[1, 2, 3]
(1, 2)
{'a': 1}
{1, 2, 3}

4. Operators

These are the tokens responsible to perform an operation in an expression. The variables
on which operation is applied are called operands.

a=5

b=2

print(a + b) # Addition

print(~a) # Bitwise NOT

Output:
7
-6

5. Punctuators

These are the symbols that are used in Python to organize the structures, statements, and
expressions. Some of the Punctuators are: [ ] { } ( ) @ -= += *= //= **== = , etc.

Python Variables and Naming

In Python, variables are used to store data that can be referenced and manipulated during
program execution. A variable is essentially a name that is assigned to a value.

• Unlike Java and many other languages, Python variables do not require explicit
declaration of type.

• The type of the variable is inferred based on the value assigned.

x=5

name = "Samantha"

print(x)

print(name)

Output:
5
Samantha

Rules for Naming Variables

To use variables effectively, we must follow Python's naming rules:

• Variable names can only contain letters, digits and underscores (_).

• A variable name cannot start with a digit.

• Variable names are case-sensitive like myVar and myvar are different.

• Avoid using Python keywords like if, else, for as variable names.

Below listed variable names are valid:

age = 21

_colour = "lilac"
total_score = 90

Below listed variables names are invalid:

# 1name = "Error" # Starts with a digit

# class = 10 # 'class' is a reserved keyword

# user-name = "Doe" # Contains a hyphen

Assigning Values to Variables

Basic Assignment: Variables in Python are assigned values using the = operator.

x=5

y = 3.14

z = "Hi"

Dynamic Typing: Python variables are dynamically typed, meaning the same variable can
hold different types of values during execution.

x = 10

x = "Now a string"

Multiple Assignments

Assigning Same Value: Python allows assigning the same value to multiple variables in a
single line, which can be useful for initializing variables with the same value.

a = b = c = 100

print(a, b, c)

Output: 100 100 100

Assigning Different Values: We can assign different values to multiple variables

simultaneously, making the code concise and easier to read.

x, y, z = 1, 2.5, "Python"

print(x, y, z)

Output: 1 2.5 Python

Type Casting a Variable

Type casting refers to the process of converting the value of one data type into another.
Python provides several built-in functions to facilitate casting, including int(), float() and
str() among others. Basic casting functions are:

• int(): Converts compatible values to an integer.

• float(): Transforms values into floating-point numbers.

• str(): Converts any data type into a string.

s = "10"

n = int(s)

cnt = 5

f = float(cnt)

age = 25

s2 = str(age)

print(n)

print(f)

print(s2)

Output:
10
5.0
25

Type of Variable

In Python, we can determine the type of a variable using the type() function. This built-in
function returns the type of the object passed to it.

n = 42

f = 3.14

s = "Hello, World!"
li = [1, 2, 3]

d = {'key': 'value'}

bool = True

print(type(n)) #

print(type(f)) #

print(type(s)) #

print(type(li)) #

print(type(d)) #

print(type(bool)) #

Concept of Object Reference

Let us assign a variable x to value 5.

x=5

When x = 5 is executed, Python creates an object to represent the value 5 and makes x
reference this object.

Now, let's assign another variable y to the variable x.

y=x

This statement creates y and references the same object as x, not x itself. This is called a
Shared Reference, where multiple variables reference the same object.

Now, if we write

x = 'Python'

Python creates a new object for the value "Python" and makes x reference this new object.

The variable y remains unchanged, still referencing the original object 5. Now, If we assign a
new value to y:

y = "Computer"

Python creates yet another object for "Computer" and updates y to reference it.
The original object 5 no longer has any references and becomes eligible for garbage
collection.

Key Points:

• Python variables hold references to objects, not the actual objects themselves.

• Reassigning a variable does not affect other variables referencing the same object
unless explicitly updated.

Deleting a Variable

We can remove a variable from the namespace using the del keyword. This deletes the
variable and frees up the memory it was using.

x = 10

del x

print(x) # This will raise a NameError

Explanation:

• del x removes the variable x from memory.

• After deletion, trying to access the variable x results in a NameError indicating that
the variable no longer exists.

Practical Examples

1. Swapping Two Variables

Using multiple assignments, we can swap the values of two variables without needing a
temporary variable.

a, b = 5, 10

a, b = b, a

print(a, b)

Output: 10 5

2. Counting Characters in a String

Assign the results of multiple operations on a string to variables in one line.

word = "Python"
length = len(word)

print("Length of the word:", length)

Output: Length of the word: 6

1. What is the output of the following code?

x=5

y=x

x = 10

print(y)

a) 5 b) 10 c) Error d) None

2. Which of the following is NOT a valid Python variable name?

a) my_var b) _private c) 2nd_var d) var2

3. What is the data type of the following literal: 3.14j

a) int b) float c) complex d) None