SCC.
111 Software Development
– Lecture 47: High performance
Python
Adrian Friday, Hansi Hettiarachchi and Nigel Davies
This lecture
• Python’s performance and how to increase it
• Subprocesses
• Compiling Python and optimising Python
applications using native code
Strengths of Python
Flexible and powerful in-built type Fast to develop, can try scripts in Extensibility and wide range of
system REPL libraries available
Arbitrary data size (ints, strings, lists etc.)
Objects, modules, inheritance etc.
Downside?
Overhead as
Type conversion
variables ‘change
based on context
type’
Magic behind the
scenes to support
abstraction This costs space
(arbitrary sized and time…
integers, arbitrary
length strings…)
Python tradeoffs
• The flexibility comes at a cost
(performance)
• Python source is translated to python
bytecode (.py -> .pyc), pyc updated on
first run
• The bytecode is interpreted at runtime
Speeding things up?
• Will will need to trade some of the
flexibility to get more performance
• Introduce fixed types so managing
memory is leaner and more efficient
• Get closer to the machine
architecture
• either use other program
components written in a
compiled language
• Or, ‘compile’ our python in
some way
Part 1: Working
with other
processes
Useful when 1) using python to ‘script’
other things, e.g. automation of routine
system tasks; 2) running toolkits written
in other languages (such as C)
[Link](command) Execute some subprocess (waits for return code)
Subprocess runs to completion, could be in C or something fast
[Link](args) Execute some subprocess (send it commands as
stdin, get results as stdout)
Subprocess runs to completion with command input and output
For example
import os
status_code = [Link]('ls')
print(f"Returned {status_code}")
[Link]
For example
import subprocess
# Don't capture output
[Link](["ls", "-l"])
# Do capture output
completed_process =
[Link](["ls", "-l", "/dev/null"],
capture_output=True)
print(completed_process)
[Link]
Analysis
• Pros and cons of process level integration
• Startup cost of process
• Verbose API designed for humans
• Difficult to maintain state outside python script
• Low level of granularity
• Difficult to efficiently pass large amounts of data efficiently in/out
• Simple to use
• Cross platform
• Very versatile
Part 2: Serious performance
When you want to write with the convenience of Python, but get the
performance of C where it matters...
Optimize for the common case…
• Actively try to create programs that utilise
multiple programming languages.
• Allow each language to excel at what it does
best.
• Write user facing scripts in Python
• Write the high-performance modules in a
language that is more computationally
efficient.
• Create Foreign Function Interfaces between
them…
[Link]
Foreign Function Interfaces
• A function that, when called, executes code
written in a different programming language.
• Provides finer grained integration of library Python Application Python modules
code into applications than process-based
approaches.
Python Virtual Machine
• C is the most common language to interface
with, as it complements Python well. Python Core APIs Python Extension APIs
Python Extension APIs
Python Extension APIs
C Implementation Python Extension APIs
C Implementation
• Typically implemented through dynamically C Implementation
C Implementation
linking C libraries into Python interpreter. C Implementation
(Java and C# can do this too) Dynamically Linked Library
Dynamically Linked Library
Dynamically Linked Library
Dynamically Linked Library
Refresher: C/C++ Libraries
• C functions are compiled into object files (.o
files). These can be grouped into libraries.
• Object files are not required to have a main
function
• These files are linked together at the final stage
of a C compilation to create a program by the
linker. This is called static linking.
• Most operating systems also support dynamic
linking.
• Libraries are linked to applications at run-time.
• Shared object files in Linux, DLLs on Windows.
CPython Extensions
• CPython extension modules are just shared object C libraries (.so files, or DLLs)
• Placed in a well know search path, defined by [Link]…
• The Python VM exposes a set of C library functions and types (classes/structs) that allow:
• Python modules and functions to be created
• Marshalling of types between C/Python
• Initialization callbacks
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "scc.h"
Example… int the_answer()
{
return 42;
}
static PyObject *
• Key aspects: scc_the_answer(PyObject *self, PyObject *args)
{
return PyLong_FromLong(the_answer());
}
• C code that makes up the functionality of static PyMethodDef scc_methods[] = {
{"the_answer", scc_the_answer, METH_VARARGS,
the module. "Determine the answer to life, the universe and everything."},
{NULL, NULL, 0, NULL} /* Sentinel */
};
• Wrappers methods for each underlying C static struct PyModuleDef sccmodule = {
PyModuleDef_HEAD_INIT,
function. Note standardized naming "scc",
NULL,
/* name of module */
/* module documentation, may be NULL */
convention and parameter list. -1, /* size of per-interpreter state of the module,
or -1 if the module keeps state in global variables. */
scc_methods
};
• Initialization callback function, that PyMODINIT_FUNC
registers a table of exposed functions to the PyInit_scc(void)
{
Python VM. PyObject *m;
m = PyModule_Create(&sccmodule);
if (m == NULL)
• Explicit functions to manage type return NULL;
conversion between C and Python. }
return m;
Using CPython Extensions…
• To create and use a C Extension:
• Compile as a dynamic library
• Include Python VM library
• Place resultant library in the
Python search path
• Use like any other Python module!
Limitations
• Powerful, but quite Complex to write
• Developer needs to handle memory management (ref counts!)
• Exception handling requirements need to be obeyed
• Only works with CPython…
Three higher level approaches
The C Foreign
Using the ctypes The Cython
Function
Library language
Interface or CFFI
Calling C directly from Python
Python C
import ctypes #include <stdio.h>
#include <stdlib.h>
lib = [Link]('./[Link]')
int fib(int n)
my_fib = [Link] {
my_fib.argtypes = [ctypes.c_int] if (n == 1)
my_fib.restype = ctypes.c_int return 1;
result = my_fib(5) return n * fib(n - 1);
print(result) }
• gcc -fPIC -c fib.c
• gcc -shared -o [Link] fib.o
CFFI sweetness
>>> import cffi
>>> [Link]("""
... int fib(int n)
... """)
>>> lib = [Link]('./[Link]')
>>> fib = [Link]
>>> fib(5)
120
Cython
def fib(n):
if n == 1:
return 1
return n * fib(n - 1)
from setuptools import setup python [Link] build_ext --inplace
from [Link] import cythonize
import fib
setup(
ext_modules=cythonize("[Link]") result = [Link](5)
) print(result)
Summary
• Python is great for working with
data
• Unless you want high performance
• Compiling vs. interpreting
• subprocess, ctype, cffi, cython…