Module 19 - AUTOMATION WITH SELENIUM

Welcome to Module 19! Selenium WebDriver is a widely used open-source framework for
automating web browsers. It allows you to write scripts that interact with web pages just like
a human user would: clicking buttons, filling forms, navigating pages, and extracting
information.

Chapter 1: Setting Up and Basic Interactions

This chapter covers the installation of Selenium, how to set up your WebDriver, and perform
fundamental tasks like opening URLs and automating simple searches.

1.1 Installation

To use Selenium with Python, you need two main components:

1. Selenium Python Library: Install it using pip:

Bash

pip install selenium

2. WebDriver Executable: Selenium automates browsers using browser-specific

"WebDrivers." You need to download the WebDriver executable for the browser you
want to automate. Common WebDrivers include:
o ChromeDriver: For Google Chrome (most common).
 Download from: [Link]
 Important: Download the version that matches your installed Chrome
browser version.
o GeckoDriver: For Mozilla Firefox.
 Download from: [Link]
o MSEdgeDriver: For Microsoft Edge.
 Download from: [Link]
edge/tools/webdriver/

Setup (Crucial!):

o Recommended (Simple): Place the downloaded WebDriver executable (e.g.,

[Link] for Windows, chromedriver for Linux/macOS) in a
directory that is included in your system's PATH environment variable. This
allows Python to find it automatically.
o Alternative (Explicit Path): If you don't want to modify your PATH, you can
specify the exact path to the WebDriver executable in your Python code. We'll
use this method in our examples for clarity.

1
1.2 Opening a URL

After setting up, the first step is to launch a browser and navigate to a URL.

 Steps:
1. Import webdriver from selenium.
2. Create an instance of the WebDriver (e.g., [Link]()).
3. Use the [Link]() method to open a URL.
4. Use [Link]() to pause execution (useful for observing actions, but in
real scenarios, use explicit waits).
5. Always close the browser at the end using [Link]().
 Example:

Python

from selenium import webdriver

from [Link] import Service
import time

# --- IMPORTANT ---

# Replace 'path/to/your/[Link]' with the actual path to
your ChromeDriver executable
# Example: C:/Users/YourUser/Downloads/chromedriver-
win64/[Link]
# Or: /usr/local/bin/chromedriver on macOS/Linux if you installed it
there
CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
# Set up the WebDriver service
service = Service(CHROMEDRIVER_PATH)

# Initialize the Chrome WebDriver

driver = [Link](service=service)
print("Chrome browser launched successfully.")

# Open Google's website

[Link]("[Link]
print("Navigated to [Link]")

# Pause for a few seconds to see the page

[Link](3)

# Close the browser

[Link]()
print("Browser closed.")

except Exception as e:
print(f"An error occurred: {e}")
print("Please ensure ChromeDriver path is correct and Chrome
browser is installed.")
print("Also, check that ChromeDriver version matches your Chrome
browser version.")

2
1.3 Automating Google Search

Now, let's automate a common task: performing a search on Google. This involves finding an
HTML element (the search bar), typing text into it, and then submitting the form.

 Key Concepts:
o find_element(): Locates a single HTML element on the page.
o By class: Used to specify the method for finding an element (e.g., [Link],
[Link], [Link]).
o send_keys(): Simulates typing text into an input field.
o submit(): Submits the form the element belongs to (if it's an input within a
form).
o click(): Simulates a mouse click on an element.
 Example:

Python

from selenium import webdriver

from [Link] import Service
from [Link] import By # Import By for finding
elements
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)
[Link]("[Link]

# Give some time for the page to load

[Link](2)

# 1. Find the search input field by its 'name' attribute

# Inspect Google's search bar: it usually has name="q"
search_box = driver.find_element([Link], "q")
print("Found search box.")

# 2. Type "Selenium Python" into the search box

search_box.send_keys("Selenium Python")
print("Typed 'Selenium Python' into search box.")

# 3. Submit the search query

# Option A: Use .submit() on the input element
search_box.submit()
print("Search submitted.")

# Option B (Alternative): Find the search button and click it

# search_button = driver.find_element([Link], "btnK") # 'btnK'
is a common name for Google search button
# search_button.click()
# print("Search button clicked.")

# Pause to view search results

[Link](5)

[Link]()

3
 except Exception as e:
print(f"An error occurred during Google search automation: {e}")
print("Ensure the search box 'name' attribute ('q') or button
'name' attribute ('btnK') is correct.")
1.4 Automating Navigations

Selenium allows you to navigate browser history just like a user would.

 Methods:
o [Link](): Navigates back to the previous page in the browser history.
o [Link](): Navigates forward to the next page in the browser
history.
o [Link](): Refreshes the current page.
 Example:

Python

from selenium import webdriver

from [Link] import Service
from [Link] import By
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)

# 1. Navigate to the first page

[Link]("[Link]
print("Page 1: [Link]")
[Link](2)

# 2. Navigate to a second page (e.g., performing a search)

search_box = driver.find_element([Link], "q")
search_box.send_keys("Python official website")
search_box.submit()
print("Page 2: Google Search Results")
[Link](3)

# 3. Click on the first search result (e.g., [Link])

# We'll use a CSS selector here, often reliable for first link
first_result_link = driver.find_element(By.CSS_SELECTOR,
"div#search a h3")
first_result_link.click()
print("Page 3: Navigated to [Link] (or similar first
result)")
[Link](3)

# 4. Go back to the previous page (Google Search Results)

[Link]()
print("Navigated back to Page 2 (Google Search Results)")
[Link](3)

# 5. Go forward to the next page ([Link] again)

[Link]()
print("Navigated forward to Page 3 ([Link])")

4
 [Link](3)

# 6. Refresh the current page

[Link]()
print("Page 3 refreshed.")
[Link](3)

[Link]()

except Exception as e:
print(f"An error occurred during navigation automation: {e}")

Chapter 2: Locating Elements and Extracting Data

This chapter focuses on the various sophisticated methods Selenium provides for finding
specific elements on a web page and extracting their data.

2.5 Finding Elements by Different Methods

Selenium offers several strategies to locate elements. Choosing the right method depends on
the HTML structure and the uniqueness of the element.

 [Link]: Locates an element by its id attribute. IDs are supposed to be unique on a

page.
o Example: driver.find_element([Link], "loginButton")
 [Link]: Locates an element by its name attribute.
o Example: driver.find_element([Link], "username")
 By.CLASS_NAME: Locates elements by their class attribute. Note: if an element has
multiple classes (e.g., class="btn primary large"), you can only use one class
name at a time with this method. For multiple classes, use CSS Selector or XPath.
o Example: driver.find_element(By.CLASS_NAME, "product-title")
 By.TAG_NAME: Locates elements by their HTML tag name (e.g., div, a, input, p).
o Example: driver.find_element(By.TAG_NAME, "h1")
 By.LINK_TEXT: Locates an <a> (anchor) element by its exact visible text.
o Example: driver.find_element(By.LINK_TEXT, "Click here for more
details")
 By.PARTIAL_LINK_TEXT: Locates an <a> element by partial matching of its visible
text.
o Example: driver.find_element(By.PARTIAL_LINK_TEXT, "more
details")
 By.CSS_SELECTOR: Locates elements using CSS selectors (similar to how CSS styles
elements). Very powerful and often preferred for complex selections over XPath due
to readability and performance.
o Examples:
 #id_name (by ID)
 .class_name (by class)
 tag_name (by tag)
 tag_name[attribute='value'] (by tag and attribute)
 parent > child (direct child)
 ancestor descendant (any descendant)

5
 o Example: driver.find_element(By.CSS_SELECTOR, "[Link]-card
> [Link]")
 [Link]: Locates elements using XPath expressions. Extremely powerful and
flexible for navigating the HTML DOM structure.
o find_element() returns the first matching element.
o find_elements() returns a list of all matching elements.
 Example (Demonstrating various By methods): (We'll use a dummy website for
demonstration, or you can inspect elements on any public site like
[Link])

Python

from selenium import webdriver

from [Link] import Service
from [Link] import By
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)
[Link]("[Link] # A good site for
practicing
[Link](2)

print("\n--- Finding Elements ---")

# By ID (if exists, [Link] doesn't have many

prominent IDs)
# Try to find a hypothetical element:
try:
# This ID likely won't exist on [Link], just for
demo
non_existent_element = driver.find_element([Link],
"nonExistentId")
print(f"Found by ID (this should fail):
{non_existent_element.tag_name}")
except Exception:
print("Element with ID 'nonExistentId' not found (as
expected).")

# By Class Name: Find the first book's price

try:
price_element = driver.find_element(By.CLASS_NAME,
"price_color")
print(f"Found by CLASS_NAME (first price):
{price_element.text}")
except Exception as e:
print(f"Error finding by CLASS_NAME: {e}")

# By Tag Name: Find all <h3> tags (often book titles)

h3_tags = driver.find_elements(By.TAG_NAME, "h3")
print(f"Found {len(h3_tags)} elements by TAG_NAME (h3). First
one: {h3_tags[0].text if h3_tags else 'N/A'}")

# By Link Text: Find a specific link, e.g., "Poetry" category

try:

6
 poetry_link = driver.find_element(By.LINK_TEXT, "Poetry")
print(f"Found by LINK_TEXT: {poetry_link.text}")
# poetry_link.click() # Uncomment to click
# [Link](2)
# [Link]()
except Exception as e:
print(f"Error finding by LINK_TEXT: {e}")

# By Partial Link Text: Find a link containing "Fantasy"

try:
fantasy_link_partial =
driver.find_element(By.PARTIAL_LINK_TEXT, "Fantasy")
print(f"Found by PARTIAL_LINK_TEXT:
{fantasy_link_partial.text}")
except Exception as e:
print(f"Error finding by PARTIAL_LINK_TEXT: {e}")

# By CSS Selector: Find all book titles

# CSS selector: article with class 'product_pod', then h3, then a
book_titles_css = driver.find_elements(By.CSS_SELECTOR,
"article.product_pod h3 a")
print(f"Found {len(book_titles_css)} elements by CSS_SELECTOR
(book titles). First one: {book_titles_css[0].get_attribute('title')
if book_titles_css else 'N/A'}")

[Link](3)
[Link]()

except Exception as e:
print(f"An error occurred: {e}")
2.6 Selecting Links

This is a specific application of find_element() using By.LINK_TEXT or

By.PARTIAL_LINK_TEXT, followed by click().

 Example:

Python

from selenium import webdriver

from [Link] import Service
from [Link] import By
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)
[Link]("[Link]
[Link](2)

# Find the 'Travel' category link and click it

try:
travel_link = driver.find_element(By.LINK_TEXT, "Travel")
print(f"Found link: '{travel_link.text}'. Clicking now...")
travel_link.click()
print("Navigated to Travel category.")

7
 [Link](3)
except Exception as e:
print(f"Error finding/clicking 'Travel' link: {e}")
print("The link text 'Travel' might not exist or the page
structure changed.")

[Link]()

except Exception as e:
print(f"An error occurred: {e}")
2.7 Refreshing the Page

Already demonstrated in section 1.4, [Link]() simply reloads the current page in
the browser.

 Example (Simple Refresh):

Python

from selenium import webdriver

from [Link] import Service
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)
[Link]("[Link]
print("Page loaded.")
[Link](3)

[Link]()
print("Page refreshed.")
[Link](3)

[Link]()

except Exception as e:
print(f"An error occurred: {e}")
2.8 Finding Element by XPath

XPath (XML Path Language) is a powerful query language for selecting nodes from an XML
document (and HTML documents can be treated as XML). It allows you to navigate through
elements and attributes in the DOM structure, making it very flexible for finding elements
that might not have unique IDs or classes.

 XPath Syntax Basics:

o /: Selects from the root node.
o //: Selects nodes from the current node that match the selection, no matter
where they are.
o .: Selects the current node.
o ..: Selects the parent of the current node.
o @attribute: Selects an attribute.
o tagname: Selects all nodes with the specified tag name.

8
 o tagname[condition]: Selects tags that meet a certain condition.
 [@id='someid']: Element with a specific ID.
 [@class='someclass']: Element with a specific class.
 [contains(@class, 'partial_class')]: Element whose class
attribute contains a substring.
 [text()='Some Text']: Element whose visible text is exactly 'Some
Text'.
 [contains(text(), 'Partial Text')]: Element whose visible text
contains 'Partial Text'.
 [index]: Selects the element at a specific index (1-based).
 Example:

Python

from selenium import webdriver

from [Link] import Service
from [Link] import By
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)
[Link]("[Link]
[Link](2)

print("\n--- Finding Elements by XPath ---")

# XPath 1: Find the first <h1> tag on the page (absolute path)
try:
h1_xpath = driver.find_element([Link],
"/html/body/div/div/div/div/section/div[2]/ol/li/article/div[2]/h3/a"
)
# Note: This is an example of an absolute XPath, which is
brittle.
# A more robust XPath for the first book title might be:
first_book_title_xpath = driver.find_element([Link],
"//article[@class='product_pod'][1]/h3/a")
print(f"Found first book title by XPath:
{first_book_title_xpath.get_attribute('title')}")
except Exception as e:
print(f"Error finding first book title by XPath: {e}")

# XPath 2: Find all book prices (using contains for class, more
robust than exact match)
all_prices_xpath = driver.find_elements([Link],
"//p[contains(@class, 'price_color')]")
print(f"Found {len(all_prices_xpath)} prices by XPath. First one:
{all_prices_xpath[0].text if all_prices_xpath else 'N/A'}")

# XPath 3: Find a specific book by its title text

try:
book_by_text_xpath = driver.find_element([Link],
"//a[contains(text(), 'A Light in the Attic')]")
print(f"Found book by partial link text XPath:
{book_by_text_xpath.text}")

9
 except Exception as e:
print(f"Error finding book by text XPath: {e}")

[Link](3)
[Link]()

except Exception as e:
print(f"An error occurred: {e}")

o Tip: You can often get XPath from your browser's developer tools (Inspect
Element -> Right Click -> Copy -> Copy XPath or Copy Full XPath). Be
cautious with "Full XPath" as it's often too specific and brittle.

2.9 Getting Data

Once you have located an element, you can extract various types of information from it.

 [Link]: Returns the visible (inner) text of the element, including the text of its
sub-elements, excluding any leading/trailing whitespace.
 element.get_attribute('attribute_name'): Returns the value of a specified
HTML attribute (e.g., href, src, value, title, class, id).
 element.tag_name: Returns the HTML tag name of the element (e.g., 'div', 'a',
'input').
 element.is_displayed(): Returns True if the element is visible on the page, False
otherwise.
 element.is_enabled(): Returns True if the element is enabled (not disabled),
False otherwise.
 element.is_selected(): Returns True if the element (e.g., checkbox, radio button,
option in a dropdown) is selected, False otherwise.
 Example:

Python

from selenium import webdriver

from [Link] import Service
from [Link] import By
import time

CHROMEDRIVER_PATH = 'path/to/your/[Link]'

try:
service = Service(CHROMEDRIVER_PATH)
driver = [Link](service=service)
[Link]("[Link]
attic_1000/[Link]") # Navigate to a specific book page
[Link](2)

print("\n--- Getting Data from Elements ---")

# Get the product title

try:
product_title_element = driver.find_element(By.TAG_NAME,
"h1")
product_title = product_title_element.text
print(f"Product Title: {product_title}")

10
 print(f" Tag Name: {product_title_element.tag_name}")
print(f" Is Displayed?:
{product_title_element.is_displayed()}")
except Exception as e:
print(f"Error getting product title: {e}")

# Get the price

try:
price_element = driver.find_element(By.CLASS_NAME,
"price_color")
product_price = price_element.text
print(f"Product Price: {product_price}")
except Exception as e:
print(f"Error getting product price: {e}")

# Get the description (often in a <p> tag after an <h3> or <h4>)

try:
# This XPath targets the <p> tag that is a sibling to an <h3>
with text 'Product Description'
description_element = driver.find_element([Link],
"//h2[text()='Product Description']/following-sibling::p")
product_description = description_element.text[:200] + "..."
# Get first 200 chars
print(f"Product Description (first 200 chars):
{product_description}")
except Exception as e:
print(f"Error getting product description: {e}")

# Get the 'href' attribute of the "Add to basket" button (or any
link)
try:
# Example using a button if it had an href, or a link like
'All products'
# Let's target the 'Add to basket' button which doesn't have
an href for demo
add_to_basket_button = driver.find_element(By.CLASS_NAME,
"btn-primary")
# For demonstration, we'll try to get its 'type' attribute
button_type = add_to_basket_button.get_attribute("type")
print(f"Add to Basket Button Type: {button_type}")
print(f"Add to Basket Button Text:
{add_to_basket_button.text}")
except Exception as e:
print(f"Error getting button attributes: {e}")

[Link](3)
[Link]()

except Exception as e:
print(f"An error occurred: {e}")

11