Useful Python

Chapter 1: Python as Glue

Python is a versatile and powerful language that can be used for a wide variety of tasks. One of the most common use cases for Python is as a glue language: it helps us combine skills and programs we already know how to use by allowing us to easily convert data from one format to another. This means that we can take data in one format that we don’t have tools to manipulate and change it into data for tools that we’re comfortable with. Whether we need to process a CSV, web page, or JSON file, Python can help us get the data into a format we can use.

For example, we might use Python to pull data from a web page and put it into Excel, where we already know how to manipulate it. We might also read a CSV file downloaded from a website, calculate the totals from it, and then output the data in JSON format.

Who This Series is For

In this series of tutorials, we’re not looking at data science. That is, this isn’t about doing heavy statistical or numerical calculations on data we’ve received. Python is one of the industry-standard tools for doing calculations like these—using libraries such as NumPy and pandas—and there are plenty of resources available for learning data science (such as Become a Python Data Scientist).

In this series, we’ll looking at how to convert data from one form to another so that we can then go on to manipulate it.

We’re also going to assume a little knowledge of Python and programming already—such as what a variable is, what a dictionary is, and how to import a module. To start learning Python (or any other programming language) from scratch, check out SitePoint’s programming tutorials. The Python wiki also has a list of Python programming tutorials for programmers.

Getting Started

We’ll start this tutorial by looking at how to read data and then how to write it to a different format.

Reading Data

Let’s try an example. Let’s imagine we’re an author on a tour of local libraries, talking to people about our books. We’ve been given plymouth-libraries.json—a JSON file of all the public libraries in the town of Plymouth in the UK—and we want to explore this dataset a little and convert it into something we can read in Excel or Google Sheets, because we know about Excel.

First, let’s read the contents of the JSON file into a Python data structure:

import json

with open(plymouth-libraries.json) as fp:

    library_data = json.load(fp)

Now let’s explore this data a little in Python code to see what it contains:

print(library_data.keys())

This will print dict_keys(['type', 'name', 'crs', 'features']), which are the top-level keys from this file.

Similarly:

print(library_data[features][0][properties][LibraryName])

This will print Central Library, which is the LibraryName value in properties for the first entry in the features list in the JSON file.

This is the most basic, and most common, use of Python’s built-in json module: to load some existing JSON data into a Python data structure (usually a Python dictionary, or nested set of Python dictionaries).

Bear in mind that, to keep these examples simple, this code contains no error checking. (Check out A Guide to Python Exception Handling for more on that.) But handling errors is important. For example, what would happen if the plymouth-libraries.json file didn’t exist? What we do in that situation depends on how we should react for errors. If we’re running this script by hand, Python will display the exception that occurs—in this case, a FileNotFoundError exception. Simply seeing that exception may be enough; we may not want to handle this in code at all:

$ python load-json.py

Traceback (most recent call last):

  File /home/aquarius/Scratch/fail.py, line 13, in

    open(plymouth-libraries.json)

FileNotFoundError: [Errno 2] No such file or directory: 'plymouth-libraries.json'

If we’d like to do something more than have our program terminate with an error, we can use Python’s try and except keywords (as the exception handling article above describes) to do something else of our choosing. In this case, we display a more friendly error message and then exit (because the rest of the program won’t run without the list of libraries!):

try:

    with open(plymouth-libraries.json) as fp:

        library_data = json.load(fp)

except FileNotFoundError:

    print(I couldn't find the plymouth-libraries.json file!)

    sys.exit(1)

Writing

Now we want to write that data from its Python dictionary into a different format on disk, so we can open it in Excel. For now, let’s use CSV format, which is a very simple file format that Excel understands. (If you’re thinking, Hey, why don’t we make it a full Excel file! … then read on. CSV is simpler, so we’ll do that first.) This process of taking Python data structures and writing them out as some file format is called serialization. So we’re going to serialize the data we read as JSON into CSV format.

The image below demonstrates the stages involved in serialization.

A CSV file is a text file of tabular data. Each row of the table is one line in the CSV file, with the entries in the row separated by commas. The first line of the