Chapter 02: Programming and Automation
Advanced Programming in Python
Chapter 02: Programming and Automation
1
Chapter 02: Programming and Automation
Table of Contents
Table of Contents ......................................................................................................................................................2
1. Introduction ................................................................................................................................................3
2. Principles of automation........................................................................................................................4
3. Manipulating Files with Python ..........................................................................................................4
4. Library openpyxl .......................................................................................................................................5
4.1. Detailed Description of This Library (openpyxl): ..................................................................6
4.1.1. Create an Excel file with openpyxl ..........................................................................................7
4.1.2. Read data form an Excel file with openpyxl ........................................................................7
5. Library Pandas ...........................................................................................................................................8
5.1. Data loading: ..........................................................................................................................................8
5.2. Data exploration: .................................................................................................................................8
5.3. Performance analysis: .......................................................................................................................8
5.4. Visualizing results: ..............................................................................................................................8
5.5. Identifying problems: ........................................................................................................................9
6. Definitions and examples of automation (Sending emails).....................................................9
7. File and folder manipulation module: OS, Shutil ...................................................................... 11
7.1. Python OS Module ............................................................................................................................ 11
7.2. function for Os: .................................................................................................................................. 11
8. Visualizing data (Matplotlib) ............................................................................................................ 14
9. Visualizing data (Seaborn) ................................................................................................................. 19
2
Chapter 02: Programming and Automation
Definitions
Python File
and examples Use libraries to: Browse a
Principles of libraries for manipulation
of automation folder ([Link])
Task automation: with Python
(sending
Automation OS, shutil: file
emails, etc.)
and folder
manipulation
Check for the existence of a
file or folder
Openpyxl or ([Link])
pandas:
working with
Excel or CSV
files Create or delete folders
([Link], [Link])
Visualizing data:
Matplotlib, Seaborn,
1. Introduction
❑ The Python language is a versatile, high-level, interpreted programming language,
valued for its simplicity and readability.
❑ It is often used for web development, data analysis, artificial intelligence, task
automation, and many other areas.
❑ Python has a vast standard library and an active community, making it easy for
developers of all skill levels to learn and use.
❑ Automating tasks with Python is a very popular use of the language, as it
can simplify and speed up many repetitive tasks.
❑ Python is the ideal language for automating tasks such as reading data from Excel files,
transcribing them to a web page, extracting tables from websites, generating automatic data
reports, aggregating multiple data sources, automatically extracting and generating
information from PDF files, etc.
3
Chapter 02: Programming and Automation
2. Principles of automation.
Task automation is a core concept in computer science and engineering.
It aims to replace repetitive and error-prone manual operations with automated, efficient,
and reliable scripts.
In civil engineering and scientific research, automation is particularly valuable due to the
frequent handling of large datasets and routine analytical or reporting tasks.
Automation provides major benefits such as:
➢ Time savings by reducing manual effort.
➢ Improved accuracy through consistent execution.
➢ Reproducibility of results across multiple analyses or experiments.
Python is one of the most powerful tools for automation thanks to:
Its simplicity and readability, making it easy to learn and apply.
A rich ecosystem of libraries (e.g., os, shutil, pandas, openpyxl, etc.) that support
automation across data processing, file management, and reporting.
3. Manipulating Files with Python
File manipulation is one of the most common tasks in automation.
Openpyxl and Pandas are two Python libraries used for data manipulation, but they serve
different purposes.
Openpyxl is primarily used to read and
write Excel files (in .xlsx format).
• It allows you to manipulate spreadsheets,
format cells, add charts, and perform other
operations specific to Excel files.
4
Chapter 02: Programming and Automation
Pandas, on the other hand, is a more general-purpose library for data analysis.
• It offers powerful data structures, such as DataFrames, that make it easier to manipulate,
analyze, and visualize data.
• Pandas can also read and write Excel files, but it is primarily used for more complex data
processing operations, such as filtering, aggregation, and transformation.
You might wonder why a library is named
after an adorable animal, but Pandas
actually stands for "Panel Data," a term
borrowed from econometrics.
In summary,
• If you work primarily with Excel files, OpenPyXL is an excellent choice.
• If you need to perform more advanced data analysis, Pandas is the best option.
4. Library openpyxl
Installing Reading
About the and using Excel files Writing to
‘openpyxl’ the using the an Excel
library ‘openpyxl’ ‘openpyxl’ file
library library
5
Chapter 02: Programming and Automation
4.1. Detailed Description of This Library (openpyxl):
The openpyxl library is a powerful Python tool designed for reading, writing, and
manipulating Excel files in the .xlsx format.
• Reading and Writing Excel Files:
Load an existing Excel file, read its data, and create new Excel files.
• Working with Worksheets:
Rename, delete, or create new sheets.
• Cell Manipulation:
Define values, formats, styles, background colors of cells, etc.
• Iteration over Rows and Columns:
openpyxl provides methods to iterate through rows and columns of worksheets.
• Formulas:
Add and manage formulas within cells.
• Styles and Formats:
Apply styles and formats to cells, modify fonts, colors, borders, etc.
• Managing Images and Charts:
Insert images and charts into Excel files and customize them.
• Sheet and Cell Protection:
Add passwords to protect sheets or specific cells, restricting access or modification.
• Advanced Manipulation:
Beyond basic functionalities, openpyxl also supports advanced operations such as filters,
table creation, comment management, and more.
6
Chapter 02: Programming and Automation
4.1.1. Create an Excel file with openpyxl
Objective: Create an Excel file with just some simple data in it.
Step 1: Install the library
pip install openpyxl
Step 2: Create a simple Excel file
import openpyxl
Result in simple [Link]
4.1.2. Read data form an Excel file with openpyxl
Objective: Read the data it wrote into the file example_simple.xlsx
Result:
Ahmed a 28 ans
Ali a 32 ans
7
Chapter 02: Programming and Automation
5. Library Pandas
Example of using the Pandas library in an industrial context
Production data analysis: Imagine you work in a factory and you want to analyze
production data to improve efficiency.
5.1. Data loading:
You have a CSV file containing information on daily production, such as the number of units
produced, machine operating time, and downtime.
5.2. Data exploration:
You can take a look at the first few lines of your DataFrame to understand its structure.
5.3. Performance analysis:
Suppose you want to calculate production yield, which is the number of units produced per
operating hour.
5.4. Visualizing results:
You can use graphs to visualize returns over time.
8
Chapter 02: Programming and Automation
5.5. Identifying problems:
You can also filter days where yield was particularly low to identify potential problems.
6. Definitions and examples of automation (Sending emails)
smtplib is a built-in Python module that allows you to send emails using the SMTP (Simple
Mail Transfer Protocol).
In other words: It's Python's tool for sending emails from a script.
What is smtplib in Python?
9
Chapter 02: Programming and Automation
• Connect to an SMTP server (like Gmail, Outlook, etc.)
• Log in with a username and password
• Send simple (text) or complex (HTML, attachments, etc.) emails
• Handle sending errors
Extremely basic example
Important:
• smtplib only handles sending → to build the email (subject, text, HTML,
attachments...), you use [Link] (MIMEText, MIMEMultipart, etc.).
• For Gmail, remember to enable SMTP and use an application password (not your
main password!)
10
Chapter 02: Programming and Automation
7. File and folder manipulation module: OS, Shutil
7.1. Python OS Module
✓ It is possible to automate many operating system tasks.
✓ Python's OS module provides functions for creating and deleting directories (folders),
retrieving their contents, modifying them, identifying the current directory, and managing
the computer's file system hierarchy. It is called OS, short for "operating system".
Module os
The OS module: file (and
directory) manipulation
1) Browse
2) Rename
3) Create/delete
directories
No need to install it!
os comes with Python. You can use it directly: import os
7.2. function for Os:
File and directory management is an essential part of automation and system programming
in Python. The os and [Link] modules provide powerful functions to interact with the
operating system, allowing programmers to create, rename, move, and delete files or
directories, as well as retrieve information about them. These functions make it possible to
write scripts that organize data, maintain file structures, or execute system commands
automatically. The following Python example illustrates the practical use of key functions
11
Chapter 02: Programming and Automation
such as getcwd(), chdir(), mkdir(), rename(), remove(), rmdir(), listdir()., and their [Link]
counterparts — exists(), isfile(), isdir(), split(), and join().
Example:
12
Chapter 02: Programming and Automation
13
Chapter 02: Programming and Automation
8. Visualizing data (Matplotlib)
Matplotlib is one of the most powerful and widely used data visualization libraries in
Python. It provides a flexible and comprehensive set of tools for creating high-quality static,
animated, and interactive plots. With Matplotlib, users can easily transform raw data into
visual insights through various chart types such as line graphs, bar charts, histograms,
scatter plots, and pie charts. The library is particularly valuable for engineers, data
scientists, and researchers because it integrates smoothly with other scientific libraries like
NumPy and Pandas, allowing direct visualization of numerical or tabular data. By offering
precise control over plot elements—such as colors, labels, titles, legends, and axes—
Matplotlib enables the creation of professional and publication-ready graphics for analysis,
reports, and presentations
Example01:
Result:
14
Chapter 02: Programming and Automation
Example02:
15
Chapter 02: Programming and Automation
Result:
16
Chapter 02: Programming and Automation
Example 03:
17
Chapter 02: Programming and Automation
Result:
18
Chapter 02: Programming and Automation
9. Visualizing data (Seaborn)
Seaborn is a powerful Python library for data visualization, built on top of Matplotlib. It
provides a high-level interface for creating attractive and informative statistical graphics
with minimal code. Unlike Matplotlib, which often requires detailed configuration, Seaborn
comes with beautiful default styles and color themes that make charts visually appealing
and publication-ready. It is also tightly integrated with pandas DataFrames, allowing users
to visualize data directly from datasets used in data analysis. Seaborn is widely used in
scientific research, engineering, and data science because it simplifies the creation of
complex plots such as scatter plots, boxplots, violin plots, and heatmaps, making it an
essential tool for effective statistical analysis and communication of results.
Example:
19
Chapter 02: Programming and Automation
Result:
20