Python File I/O and Database Management
Python File I/O and Database Management
Topic Covers:
File Handling:
Introduction,
Create,
Read,
Write
Delete File
Database connection using MYSQL,
Creating,
Searching and Drop Tables,
Record Manipulation:
o Select,
o Insert,
o Update,
o Delete,
o Searching,
o Sorting,
o Join
File Handling
The key function for working with files in Python is the open() function.
The open() function takes two parameters; filename, and mode.
There are four different methods (modes) for opening a file:
"r" Read Default value. Opens a file for reading, error if the file does not exist
"a" Append Opens a file for appending, creates the file if it does not exist
"w" Write Opens a file for writing, creates the file if it does not exist
"x" Create Creates the specified file, returns an error if the file exists
In addition you can specify if the file should be handled as binary or text mode
"t" Text Default value. Text mode
"b" Binary Binary mode (e.g. images)
Syntax
To open a file for reading it is enough to specify the name of the file: f
= open("[Link]")
Because "r" for read, and "t" for text are the default values, you do not need to specify
them.
Note: Make sure the file exists, or else you will get an error.
Hello! Welcome to
[Link] This file is
for testing purposes.
Good Luck!
Example
f = open("[Link]", "r")
print([Link]())
By default the read() method returns the whole text, but you can also specify
how many character you want to return:
Example
Return the 5 first characters of the file:
f = open("[Link]", "r")
print([Link](5))
Read Lines
You can return one line by using the readline() method:
Example
f = open("[Link]", "r")
print([Link]())
By calling readline() two times, you can read the two first lines
Example
Read two lines of the file:
f = open("[Link]", "r")
print([Link]())
print([Link]())
By looping through the lines of the file, you can read the whole file, line by line:
Example
Open the file "[Link]" and append content to the file:
f = open("[Link]", "a")
[Link]("Now the file has one more line!")
Example
Open the file "[Link]" and overwrite the content:
Example
Create a file called "[Link]":
f = open("[Link]", "x")
Example
Create a new file if it does not exist:
Example
Loop through the file line by line:
Delete a File
To delete a file, you must import the OS module, and run its [Link]() function:
Example
Remove the file "[Link]":
import os
[Link]("[Link]")
Example
Check if file exist, then delete it:
import os
if [Link]("[Link]"):
[Link]("[Link]")
else:
print("The file does not exist")
Delete Folder
To delete an entire folder, use the [Link]() method:
Example
import os
[Link]("myfolder")
In this section of the tutorial, we will discuss Python - MySQL connectivity, and we will
perform the database operations in python. We will also cover the Python connectivity with
the databases like MongoDB and SQLite later in this tutorial.
Install [Link]
To connect the python application with the MySQL database, we must import the
[Link] module in the program.
The [Link] is not a built-in module that comes with the python installation. We need
to install it to get it working.
[Link]
d5411298bcacbd309f96/[Link] to download the source code.
3. Open the terminal (CMD for windows) and change the present working directory to the
source code directory.
$ cd mysql-connector-python-8.0.13/
4. Run the file named [Link] with python (python3 in case you have also installed python 2)
with the parameter build.
This will take a bit of time to install mysql-connector for python. We can verify the installation
once the process gets over by importing mysql-connector on the python shell.
Database Connection
In this section of the tutorial, we will discuss the steps to connect the python application to the
database.
There are the following steps to connect a python application to our database.
To create a connection between the MySQL database and the python application, the connect()
method of [Link] module is used.
Pass the database details like HostName, username, and the database password in the method
call. The method returns the connection object.
Example
import [Link]
Output:
Here, we must notice that we can specify the database name in the connect() method if we want
to connect to a specific database.
Example
import [Link]
Output:
The cursor object can be defined as an abstraction specified in the Python DB-API 2.0. It
facilitates us to have multiple separate working environments through the same connection to
the database. We can create the cursor object by calling the 'cursor' function of the connection
object. The cursor object is an important aspect of executing queries to the databases.
<my_cur> = [Link]()
Example
import [Link]
#Create the connection object
myconn = [Link](host = "localhost", user = "root",passwd = “”,
database = "mydb")
print(myconn)
print(cur)
Output:
Example
import [Link]
try:
dbs = [Link]("show databases")
except:
[Link]()
for x in cur:
print(x)
[Link]()
Output:
('EmployeeDB',)
('Test',)
('TestDB',)
('information_schema',)
('javatpoint',)
('javatpoint1',)
('mydb',)
('mysql',)
('performance_schema',)
('testDB',)
Creating the new database
The new database can be created by using the following SQL query.
try:
#creating a new database
[Link]("create database PythonDB2")
#getting the list of all the databases which will now include the new database PythonDB
dbs = [Link]("show databases")
except:
[Link]()
for x in cur:
print(x)
[Link]()
Output:
('EmployeeDB',)
('PythonDB',)
('Test',)
('TestDB',)
('anshika',)
('information_schema',)
('javatpoint',)
('javatpoint1',)
('mydb',)
('mydb1',)
('mysql',)
('performance_schema',)
('testDB',)
In this section of the tutorial, we will create the new table Employee. We have to mention the
database name while establishing the connection object.
We can create the new table by using the CREATE TABLE statement of SQL. In our database
PythonDB, the table Employee will have the four columns, i.e., name, id, salary, and
department_id initially.
> create table Employee (name varchar(20) not null, id int primary key, salary float not n
ull, Dept_Id int not null)
Example
import [Link]
try:
#Creating a table with name Employee having four columns i.e., name, id, salary, and
department id
dbs = [Link]("create table Employee(name varchar(20) not null, id int(20) not null
primary key, salary float not null, Dept_id int not null)")
except:
[Link]()
[Link]()
Now, we may check that the table Employee is present in the database.
Alter Table
Sometimes, we may forget to create some columns, or we may need to update the table schema.
The alter statement used to alter the table schema if required. Here, we will add the column
branch_name to the table Employee. The following SQL query is used for this purpose.
Example
import [Link]
try:
#adding a column branch name to the table Employee
[Link]("alter table Employee add branch_name varchar(20) not null")
except:
[Link]()
[Link]()
Insert Operation
Adding a record to the table
The INSERT INTO statement is used to add a record to the table. In python, we can mention
the format specifier (%s) in place of values.
We provide the actual values in the form of tuple in the execute() method of the cursor.
Example
import [Link]
#Create the connection object
myconn = [Link](host = "localhost", user = "root",passwd = "",
database = "PythonDB")
#creating the cursor object
cur = [Link]()
sql = "insert into Employee(name, id, salary, dept_id, branch_name) values (%s, %s, %s,
%s, %s)"
try:
#inserting the values into the table
[Link](sql,val)
except:
[Link]()
print([Link],"record inserted!")
[Link]()
Output:
1 record inserted!
We can also insert multiple rows at once using the python script. The multiple rows are
mentioned as the list of various tuples.
Each element of the list is treated as one particular row, whereas each element of the tuple is
treated as one particular column value (attribute).
Example
import [Link]
try:
#inserting the values into the table
[Link](sql,val)
except:
[Link]()
[Link]()
Output:
3 records inserted!
Row ID
In SQL, a particular row is represented by an insertion id which is known as row id. We can
get the last inserted row id by using the attribute lastrowid of the cursor object.
Example
import [Link]
#Create the connection object
myconn = [Link](host = "localhost", user = "root",passwd = "",
database = "PythonDB")
#creating the cursor object
cur = [Link]()
sql = "insert into Employee(name, id, salary, dept_id, branch_name) values (%s, %s, %s,
%s, %s)"
val = ("Mike",105,28000,202,"Guyana")
try:
#inserting the values into the table
[Link](sql,val)
#getting rowid
print([Link],"record inserted! id:",[Link])
except:
[Link]()
[Link]()
Output:
Python provides the fetchall() method returns the data stored inside the table in the form of
rows. We can iterate the result to get the individual rows.
In this section of the tutorial, we will extract the data from the database by using the python
script. We will also format the output to print it on the console.
Example
import [Link]
try:
#Reading the Employee data
[Link]("select * from Employee")
for x in result:
print(x);
except:
[Link]()
[Link]()
Output:
We can read the specific columns by mentioning their names instead of using star (*).
In the following example, we will read the name, id, and salary from the Employee table and
print it on the console.
Example
import [Link]
#Create the connection object
myconn = [Link](host = "localhost", user = "root",passwd = "",
database = "PythonDB")
#creating the cursor object
cur = [Link]()
try:
#Reading the Employee data
[Link]("select name, id, salary from Employee")
Output:
The fetchone() method is used to fetch only one row from the table. The fetchone() method
returns the next row of the result-set.
Example
import [Link]
try:
#Reading the Employee data
[Link]("select name, id, salary from Employee")
except:
[Link]()
[Link]()
Output:
We can format the result by iterating over the result produced by the fetchall() or fetchone()
method of cursor object since the result exists as the tuple object which is not readable.
Example
import [Link]
try:
print("Name id Salary");
for row in result:
print("%s %d %d"%(row[0],row[1],row[2]))
except:
[Link]()
[Link]()
Output:
Name id Salary
John 101 25000
John 102 25000
David 103 25000
Nick 104 90000
Mike 105 28000
We can restrict the result produced by the select statement by using the where clause. This will
extract only those columns which satisfy the where condition.
try:
#Reading the Employee data
[Link]("select name, id, salary from Employee where name like 'J%'")
print("Name id Salary");
[Link]()
Output:
Name id Salary
John 101 25000
John 102 25000
try:
#Reading the Employee data
[Link]("select name, id, salary from Employee where id in (101,102,103)")
print("Name id Salary");
[Link]()
Output:
Name id Salary
John 101 25000
John 102 25000
David 103 2500
The ORDER BY clause is used to order the result. Consider the following example.
Example
import [Link]
try:
#Reading the Employee data
[Link]("select name, id, salary from Employee order by name")
print("Name id Salary");
Output:
Name id Salary
David 103 25000
John 101 25000
John 102 25000
Mike 105 28000
Nick 104 90000
Order by DESC
Example
import [Link]
try:
#Reading the Employee data
[Link]("select name, id, salary from Employee order by name desc")
except:
[Link]()
[Link]()
Output:
Name id Salary
Nick 104 90000
Mike 105 28000
John 101 25000
John 102 25000
David 103 25000
Update Operation
The UPDATE-SET statement is used to update any column inside the table. The following
SQL query is used to update a column.
Example
import [Link]
try:
#updating the name of the employee whose id is 110
[Link]("update Employee set name = 'alex' where id = 110")
[Link]()
except:
[Link]()
[Link]()
Delete Operation
The DELETE FROM statement is used to delete a specific record from the table. Here, we
must impose a condition using WHERE clause otherwise all the records from the table will be
removed.
The following SQL query is used to delete the employee detail whose id is 110 from the table.
Example
import [Link]
try:
#Deleting the employee details whose id is 110
[Link]("delete from Employee where id = 110")
[Link]()
except:
[Link]()
[Link]()
Join Operation
We can combine the columns from two or more tables by using some common column among
them by using the join statement.
We have only one table in our database, let's create one more table Departments with two
columns department_id and department_name.
create table Departments (Dept_id int(20) primary key not null, Dept_Name varchar(20)
not null);
As we have created a new table Departments as shown in the above image. However, we
haven't yet inserted any value inside it.
Let's insert some Departments ids and departments names so that we can map this to our
Employee table.
Let's look at the values inserted in each of the tables. Consider the following image.
Now, let's create a python script that joins the two tables on the common column, i.e., dept_id.
Example
import [Link]
try:
#joining the two tables on departments_id
[Link]("select [Link], [Link], [Link], [Link]
_id, Departments.Dept_Name from Departments join Employee on Departments.Dept_id
= Employee.Dept_id")
print("ID Name Salary Dept_Id Dept_Name")
for row in cur:
print("%d %s %d %d %s"%(row[0], row[1],row[2],row[3],row[4]))
except:
[Link]()
[Link]()
Output:
Right Join
Right join shows all the columns of the right-hand side table as we have two tables in the
database PythonDB, i.e., Departments and Employee. We do not have any Employee in the
table who is not working for any department (Employee for which department id is null).
However, to understand the concept of right join let's create the one.
This will insert an employee Alex who doesn't work for any department (department id is null).
Now, we have an employee in the Employee table whose department id is not present in the
Departments table. Let's perform the right join on the two tables now.
Example
import [Link]
try:
#joining the two tables on departments_id
result = [Link]("select [Link], [Link], [Link], Departme
nts.Dept_id, Departments.Dept_Name from Departments right join Employee on Depart
ments.Dept_id = Employee.Dept_id")
except:
[Link]()
[Link]()
Output:
Left Join
The left join covers all the data from the left-hand side table. It has just opposite effect to the
right join. Consider the following example.
Example
import [Link]
try:
#joining the two tables on departments_id
result = [Link]("select [Link], [Link], [Link], Departme
nts.Dept_id, Departments.Dept_Name from Departments left join Employee on Departme
nts.Dept_id = Employee.Dept_id")
print("ID Name Salary Dept_Id Dept_Name")
for row in cur:
print(row[0]," ", row[1]," ",row[2]," ",row[3]," ",row[4])
except:
[Link]()
[Link]()
Output:
Performing Transactions
Transactions ensure the data consistency of the database. We have to make sure that more than
one applications must not modify the records while performing the database operations. The
transactions have the following properties.
1. Atomicity
Either the transaction completes, or nothing happens. If a transaction contains 4 queries then
all these queries must be executed, or none of them must be executed.
2. Consistency
The database must be consistent before the transaction starts and the database must also be
consistent after the transaction is completed.
3. Isolation
Intermediate results of a transaction are not visible outside the current transaction.
4. Durability
Once a transaction was committed, the effects are persistent, even after a system failure.
Python provides the commit() method which ensures the changes made to
All the operations that modify the records of the database do not take place until the commit()
is called.
The rollback() method is used to revert the changes that are done to the database. This method
is useful in the sense that, if some error occurs during the database operations, we can rollback
that transaction to maintain the database consistency.
[Link]()
We need to close the database connection once we have done all the operations regarding the
database. Python provides the close() method. The syntax to use the close() method is given
below.
[Link]()
In the following example, we are deleting all the employees who are working for the CS
department.
Example
import [Link]
try:
[Link]("delete from Employee where Dept_id = 201")
[Link]()
print("Deleted !")
except:
print("Can't delete !")
[Link]()
[Link]()
Output:
Deleted !