Chapter 14.

Web Integration

I have been quoted saying that, in the future, all companies will be
Internet companies. I still believe that. More than ever, really.

— Andrew Grove

The Internet, or the Web, has evolved from some separate world into
something that is everywhere and in everything. It has become a technol-
ogy platform enabling a multitude of different use cases. From a finance
perspective, the following seem particularly noteworthy:

Data provision/gathering
Web technology allows the provision of data and the gathering thereof in a
simplified manner and generally at reduced costs; it also speeds up in general all
associated processes. Large financial data providers, like Bloomberg and
Thomson Reuters, rely heavily on the Web and related technologies to provide
the financial world with data in real time.
Trading/buying/selling
Using the Web also facilitates trading of financial securities; even private
investors today have access to professional trading facilities (e.g., online
brokers like Interactive Brokers) and can trade securities in real time.
Application providing
Models like Software-as-a-Service (SaaS) allow both small companies, like
startups, and large ones to provide applications in an efficient manner; even the
smallest outfit can today reach a global target audience at very little cost. Large
corporations benefit from web technologies, for example, when they use them
to provide internal applications that are accessible and usable via any standard
web browser, instead of installing such applications on hundreds or even
thousands of different machines.
Communication
Of course, the Web facilitates communication within organizations and across
organizations; the majority of today’s business and financial communication has
moved from paper to the Web.
Commoditization/scalability
Recent web technologies also allow for better virtualization, making web
servers and servers in general a commodity that everybody can rent at rather
low variable costs and that is easily scalable when requirements change;
computing power and storage capacity become more and more comparable to
electricity, which we are all used to getting from the plug sockets at home.
Again, Python for the Web is a broad topic in itself that cannot be cov-
ered by a single chapter in this book. However, this chapter is able to
cover a number of important topics from a finance perspective. In partic-
ular, it covers:

Web protocols
The first section shows how to transfer files via FTP and how to access
websites via HTTP .
Web plotting
Web technologies generally allow for better interactivity and for better real-time
support than standard approaches, for example, for plotting data; the second
section introduces the plotting library Bokeh to generate interactive web plots
and to realize real-time plotting of financial data.
Web applications
One of Python ’s strengths is its powerful web frameworks to develop web-
based applications; one that is really Pythonic and that has become quite
popular recently is Flask . This chapter illustrates techniques for developing
web-based applications using this framework.
Web services
Web services have become an important aspect of web-enabled applications; the
last section shows how to develop a simple web service for the valuation of
European options on the VSTOXX volatility index.

Web Basics

This section gives a rather brief overview of selected Python libraries for
working with web technologies and protocols. Several topics, like the han-
dling of email functionality with Python , are not touched upon.

ftplib

The File Transfer Protocol ( FTP ) is, as the name suggests, a protocol
to transfer files over the Web.[50] Python provides a dedicated library to
work with FTP called ftplib :

In [1]: import ftplib

import numpy as np

In what follows, we will connect to an FTP server, log in, transfer a file to
the server, transfer it back to the local machine, and delete the file on the
server. First, the connection:

In [2]: ftp = [Link]('[Link]')

Not every FTP server is password protected, but this one is:

In [3]: [Link](user='python', passwd='python')

Out[3]: '230 Login successful.'

To have a file that we can transfer, we generate a NumPy ndarray object

with some random data and save it to disk:

In [4]: [Link]('./data/array', [Link].standard_normal((100, 100)))

For the FTP file transfer to follow, we have to open the file for reading:

In [5]: f = open('./data/[Link]', 'r')

This open file can now be written, choosing here binary transfer, by the
STOR command in combination with the target filename:

In [6]: [Link]('STOR [Link]', f)

Out[6]: '226 Transfer complete.'

Let us have a look at the directory of the FTP server. Indeed, the file was
transferred:

In [7]: [Link]('LIST')

Out[7]: -rw------- 1 1001 1001 80080 Sep 29 11:05 [Link]

'226 Directory send OK.'

The other way around is pretty similar. To retrieve a distant file and to
save it to disk, we need to open a new file, this time in write mode:

In [8]: f = open('./data/array_ftp.npy', 'wb').write

Again, we choose binary transfer, and we use the RETR command for re-
trieving the file from the FTP server:

In [9]: [Link]('RETR [Link]', f)

Out[9]: '226 Transfer complete.'

Since we do not need the file on the server anymore, we can delete it:

In [10]: [Link]('[Link]')

Out[10]: '250 Delete operation successful.'

In [11]: [Link]('LIST')

Out[11]: '226 Directory send OK.'

Finally, we should close the connection to the FTP server:

In [12]: [Link]()

In the local directory there are now two files, the one that was generated
locally and the one generated by retrieving the file from the server:

In [13]: !ls -n ./data

<<<<<<< HEAD
Out[13]: insgesamt 156
-rw------- 1 1000 1000 77824 Sep 15 08:14 array_ftp.npy
 -rw------- 1 1000 1000 80080 Sep 15 08:14 [Link]
=======
Out[13]: insgesamt 156
-rw------- 1 1000 1000 77824 Sep 29 17:05 array_ftp.npy
-rw------- 1 1000 1000 80080 Sep 29 17:05 [Link]
>>>>>>> 798603793467fffcd06a9df88edf091e339dec37

In [14]: !rm -f ./data/arr*

# cleanup directory

All that has happened so far was done without encryption (i.e., was fully
insecure). Both login information and data were transferred in readable
form. However, for most applications such operations should be en-
crypted so others are not able to read the data and/or steal the login infor-
mation and do even worse things.

ftplib can connect to FTP servers securely via the function FTP_TLS .
Once such a secure connection is established, all other operations remain
the same:

In [15]: ftps = ftplib.FTP_TLS('[Link]')

In [16]: [Link](user='python', passwd='python')

Out[16]: '230 Login successful.'

In [17]: ftps.prot_p()

Out[17]: '200 PROT now Private.'

In [18]: [Link]('LIST')

Out[18]: '226 Directory send OK.'

 In [19]: [Link]()

httplib

Another important protocol, if not the most important one on the Web, is
the HyperText Transfer Protocol ( HTTP ).[51] This protocol is used
whenever a ( HTML -based) web page is displayed in the browser. The
Python library to work with HTTP is called httplib :

In [20]: import httplib

As with FTP , we first need a connection to the HTTP server:

In [21]: http = [Link]('[Link]')

Once the connection is established, we can send requests, for example

asking for the [Link] page (file):

In [22]: [Link]('GET', '/[Link]')

To test whether this was successful, use the getresponse method:

In [23]: resp = [Link]()

The returned object provides status information. Fortunately, our request

was successful:

In [24]: [Link], [Link]

Out[24]: (200, 'OK')

Equipped with the response object, we can now read the content as
follows:
 In [25]: content = [Link]()
content[:100]
# first 100 characters of the file

Out[25]: '<!doctype html>\n<html lang="en">\n\n\t<head>\n\t\t<meta charset="utf-

8">\n\n\t\t<title>Dr. Yves J. Hilpisch \xe2\x80'

Once you have the content of a particular web page, there are many po-
tential use cases. You might want to look up certain information, for ex-
ample. You might know that you can find the email address on the page
by looking for E (in this very particular case). Since content is a string
object, you can apply the find method to look for E :[52]

In [26]: index = [Link](' E ')

index

Out[26]: 2071

Equipped with the index value for the information you are looking for,
you can inspect the subsequent characters of the object:

In [27]: content[index:index + 29]

Out[27]: ' E contact [at] dyjh [dot] de'

Once you are finished, you should again close the connection to the
server:

In [28]: [Link]()

urllib

There is another Python library that supports the use of different web
protocols. It is called urllib . There is also a related library called
urllib2 . Both libraries are designed to work with arbitrary web re-
sources, in the spirit of the “uniform” in URL (uniform resource locator).
[53] A standard use case, for example, is to retrieve files, like CSV data
files, via the Web. Begin by importing urllib :

In [29]: import urllib

The application of the library’s functions resembles that of both ftplib

and httplib . Of course, we need a URL representing the web resource
of interest ( HTTP or FTP server, in general). For this example, we use the
URL of Yahoo! Finance to retrieve stock price information in CSV format:

In [30]: url = '[Link]

url += '&s=YHOO&a=01&b=1&c=2014&d=02&e=6&f=2014'

Next, one has to establish a connection to the resource:

In [31]: connect = [Link](url)

With the connection established, read out the content by calling the read
method on the connection object:

In [32]: data = [Link]()

The result in this case is historical stock price information for Yahoo!
itself:

In [33]: print data

Out[33]: Date,Open,High,Low,Close,Volume,Adj Close

2014-03-06,39.60,39.98,39.50,39.66,10626700,39.66
2014-03-05,39.83,40.15,39.19,39.50,12536800,39.50
2014-03-04,38.76,39.79,38.68,39.63,16139400,39.63
2014-03-03,37.65,38.66,37.43,38.25,14714700,38.25
2014-02-28,38.55,39.38,38.22,38.67,16957100,38.67
2014-02-27,37.80,38.48,37.74,38.47,15489400,38.47
2014-02-26,37.35,38.10,37.34,37.62,15778900,37.62
2014-02-25,37.48,37.58,37.02,37.26,9756900,37.26
2014-02-24,37.23,37.71,36.82,37.42,15738900,37.42
 2014-02-21,37.90,37.96,37.22,37.29,12351900,37.29
2014-02-20,37.83,38.04,37.30,37.79,11155900,37.79
2014-02-19,38.06,38.33,37.68,37.81,15851900,37.81
2014-02-18,38.31,38.59,38.09,38.31,12096400,38.31
2014-02-14,38.43,38.45,38.11,38.23,9975800,38.23
2014-02-13,37.92,38.69,37.79,38.52,12088100,38.52
2014-02-12,38.60,38.91,38.03,38.11,14088500,38.11
2014-02-11,38.15,38.86,38.09,38.50,18348000,38.50
2014-02-10,38.00,38.13,37.25,37.76,17642900,37.76
2014-02-07,36.65,37.27,36.24,37.23,16178500,37.23
2014-02-06,35.65,36.75,35.61,36.24,14250000,36.24
2014-02-05,35.60,35.94,34.99,35.49,14022900,35.49
2014-02-04,35.11,35.86,34.86,35.66,21082500,35.66
2014-02-03,35.94,36.01,34.66,34.90,22195200,34.90

The library also provides convenience functions to customize URL

strings. For example, you might want to be able to parameterize the sym-
bol to look up and the starting date. To this end, define a new URL string
with a string replacement part where you can insert the parameters:

In [34]: url = '[Link]

url += '&%s' # for replacement with parameters
url += '&d=06&e=30&f=2014'

The function urlencode takes as an argument a Python dictionary with

the parameter names and the values to associate:

In [35]: params = [Link]({'s': 'MSFT', 'a': '05', 'b': 1, 'c': 2014})

As result, there is a string object that can be inserted into the preceding
URL string to complete it:

In [36]: params

Out[36]: 'a=05&s=MSFT&b=1&c=2014'

In [37]: url % params

 Out[37]: '[Link]
b=1&c=2014&d=06&e=30&f=2014'

Equipped with this new URL string, establish a connection and read the
data from the connection:

In [38]: connect = [Link](url % params)

In [39]: data = [Link]()

The result again is stock price data, this time for more dates and for
Microsoft:

In [40]: print data

Out[40]: Date,Open,High,Low,Close,Volume,Adj Close

2014-07-30,44.07,44.10,43.29,43.58,31921400,43.31
2014-07-29,43.91,44.09,43.64,43.89,27763100,43.62
2014-07-28,44.36,44.51,43.93,43.97,29684200,43.70
2014-07-25,44.30,44.66,44.30,44.50,26737700,44.22
2014-07-24,44.93,45.00,44.32,44.40,30725300,44.12
2014-07-23,45.45,45.45,44.62,44.87,52362900,44.59
2014-07-22,45.00,45.15,44.59,44.83,43095800,44.55
2014-07-21,44.56,45.16,44.22,44.84,37604400,44.56
2014-07-18,44.65,44.84,44.25,44.69,43407500,44.41
2014-07-17,45.45,45.71,44.25,44.53,82180300,44.25
2014-07-16,42.51,44.31,42.48,44.08,63318000,43.81
2014-07-15,42.33,42.47,42.03,42.45,28748700,42.19
2014-07-14,42.22,42.45,42.04,42.14,21881100,41.88
2014-07-11,41.70,42.09,41.48,42.09,24083000,41.83
2014-07-10,41.37,42.00,41.05,41.69,21854700,41.43
2014-07-09,41.98,41.99,41.53,41.67,18445900,41.41
2014-07-08,41.87,42.00,41.61,41.78,31218200,41.52
2014-07-07,41.75,42.12,41.71,41.99,21952400,41.73
2014-07-03,41.91,41.99,41.56,41.80,15969300,41.54
2014-07-02,41.73,41.90,41.53,41.90,20208100,41.64
2014-07-01,41.86,42.15,41.69,41.87,26917000,41.61
2014-06-30,42.17,42.21,41.70,41.70,30805500,41.44
2014-06-27,41.61,42.29,41.51,42.25,74640000,41.99
2014-06-26,41.93,41.94,41.43,41.72,23604400,41.46
 2014-06-25,41.70,42.05,41.46,42.03,20049100,41.77
2014-06-24,41.83,41.94,41.56,41.75,26509100,41.49
2014-06-23,41.73,42.00,41.69,41.99,18743900,41.73
2014-06-20,41.45,41.83,41.38,41.68,47764900,41.42
2014-06-19,41.57,41.77,41.33,41.51,19828200,41.25
2014-06-18,41.61,41.74,41.18,41.65,27097000,41.39
2014-06-17,41.29,41.91,40.34,41.68,22518600,41.42
2014-06-16,41.04,41.61,41.04,41.50,24205300,41.24
2014-06-13,41.10,41.57,40.86,41.23,26310000,40.97
2014-06-12,40.81,40.88,40.29,40.58,29818900,40.33
2014-06-11,40.93,41.07,40.77,40.86,18040000,40.61
2014-06-10,41.03,41.16,40.86,41.11,15117700,40.85
2014-06-09,41.39,41.48,41.02,41.27,15019200,41.01
2014-06-06,41.48,41.66,41.24,41.48,24060500,41.22
2014-06-05,40.59,41.25,40.40,41.21,31865200,40.95
2014-06-04,40.21,40.37,39.86,40.32,23209000,40.07
2014-06-03,40.60,40.68,40.25,40.29,18068900,40.04
2014-06-02,40.95,41.09,40.68,40.79,18504300,40.54

The function urlretrieve allows us to retrieve content and save it to

disk in a single step, which is quite convenient in many circumstances:

In [41]: [Link](url % params, './data/[Link]')

Out[41]: ('./data/[Link]', <[Link] instance at 0x7f92ca59afc8>)

A brief inspection of the content of the saved file shows that we have in-
deed retrieved and saved the same content as before:

In [42]: csv = open('./data/[Link]', 'r')

[Link]()[:5]

Out[42]: ['Date,Open,High,Low,Close,Volume,Adj Close\n',

'2014-07-30,44.07,44.10,43.29,43.58,31921400,43.31\n',
'2014-07-29,43.91,44.09,43.64,43.89,27763100,43.62\n',
'2014-07-28,44.36,44.51,43.93,43.97,29684200,43.70\n',
'2014-07-25,44.30,44.66,44.30,44.50,26737700,44.22\n']

In [43]: !rm -f ./data/*

Web Plotting

Chapter 5 introduces matplotlib , the most popular plotting library for

Python . However, as powerful as it might be for 2D and 3D plotting, its
strength lies in static plotting. In fact, matplotlib is also able to generate
interactive plots, e.g., with sliders for variables. But it is safe to say that
this is not one of its strengths.[54]

This section starts with generating static plots, then proceeds to interac-
tive plots to finally arrive at real-time plotting.

Static Plots

First, a brief benchmark example using the pandas library based on a fi-
nancial time series from the Yahoo! Finance API, as used in the previous
section:

In [44]: import numpy as np

import pandas as pd
%matplotlib inline

As shown in Chapter 6, using pandas makes data retrieval from the Web
in general quite convenient. We do not even have to use additional li-
braries, such as urllib —almost everything happens under the hood.
The following retrieves historical stock price quotes for Microsoft Inc.
and stores the data in a DataFrame object:

In [45]: url = '[Link]

data = pd.read_csv(url, parse_dates=['Date'])

pandas accepts column names as parameter values for the x and y co-
ordinates. The result is shown in Figure 14-1:

In [46]: [Link](x='Date', y='Close')

 Figure 14-1. Historical stock prices for Microsoft since January 2009 (matplotlib)

Graphics and plots like Figure 14-1 can of course also be used in a web
context. For example, it is straightforward to save plots generated with
matplotlib as files in the PNG ( Portable Network Graphics ) format
and to include such files in a website. However, recent web technologies
typically also provide interactivity, like panning or zooming.

Bokeh is a library that explicitly aims at providing modern, interactive

web-based plots to Python . According to its website:

Bokeh is a Python interactive visualization library for large data

sets that natively uses the latest web technologies. Its goal is to
provide elegant, concise construction of novel graphics in the style
of Protovis/D3, while delivering high-performance interactivity
over large data to thin clients.

Three elements of this description are noteworthy:

Large data sets

It is a “plotting problem” in itself to plot large data sets. Just imagine a scatter
plot with 1,000,000 points—in general, large parts of the information get lost;
Bokeh provides built-in help in this regard.
Latest web technologies
In general, JavaScript is the language of choice as of today when it comes
to web development and visualization; it underlies libraries such as D3
( Data-Driven Documents ) and also Bokeh .
High-performance interactivity
On the Web, people are used to real-time interactivity (think modern browser
games), which can become an issue when visualizing and interacting with large
data sets; Bokeh also provides built-in capabilities to reach this goal.
On a fundamental level, working with Bokeh is not that different from
working with matplotlib . However, the default output generally is not a
standard window or, for example, an IPython Notebook (which is also
an option). It is a separate HTML file:

In [47]: import [Link] as bp

In [48]: bp.output_file("../images/msft_1.html", title="Bokeh Example (Static)")

# use: bp.output_notebook("default")
# for output within an IPython Notebook

In terms of plotting, Bokeh provides a wealth of different plotting styles

that are continuously enhanced. To start with the simplest one, consider
the following code that generates a line plot similar to our
pandas/matplotlib benchmark plot. The result is shown as Figure 14-2.
Apart from the x and y coordinates, all other parameters are optional:

In [49]: [Link](
data['Date'],
# x coordinates
data['Close'],
# y coordinates
color='#0066cc',
# set a color for the line
legend='MSFT',
# attach a legend label
title='Historical Stock Quotes',
# plot title
x_axis_type='datetime',
# datetime information on x-axis
tools = ''
)
[Link]()

In the tradition of matplotlib , Bokeh also has a gallery showcasing dif-

ferent plot styles.
 Figure 14-2. Screenshot of HTML-based Bokeh plot

Interactive Plots

The next step is to add interactivity to the web-based plot. Available inter-
activity elements (“tools”) include:

pan
Supports panning of the plot (like panning with a movie camera); i.e., moving
the plot (including x and y coordinates) relative to the fixed plotting frame
wheel_zoom
Enables zooming into the plot by using the mouse wheel
box_zoom
Enables zooming into the plot by marking a box with the mouse
reset
 Resets the original/default view of the plot
previewsave
Generates a static (bitmap) version of the plot that can be saved in PNG format

The following code demonstrates adding these tools:

In [50]: bp.output_file("../images/msft_2.html",
title="Bokeh Example (Interactive)")
[Link](
data['Date'],
data['Close'],
color='#0066cc',
legend='MSFT',
title='Historical Stock Quotes',
x_axis_type = "datetime",
tools = 'pan, wheel_zoom, box_zoom, reset, previewsave'
# adding a list of interactive tools
)
[Link]()

The output of this code is shown as Figure 14-3, where the panning func-
tion is used to move the plot within the plotting frame (compare this with
Figure 14-2).

In principle, all the features shown so far can also be implemented by us-
ing matplotlib . In fact, the interactive tools shown for Bokeh are avail-
able by default with matplotlib when you plot into a separate window.
Figure 14-4 shows a zoomed and panned version of the pandas plot in
Figure 14-1 in a separate ( Python -controlled) window. However, in con-
trast to Bokeh , matplotlib cannot “export” this functionality to be in-
cluded in a separate, standalone graphics file.[55]
Figure 14-3. Screenshot of HTML-based Bokeh plot with interactive elements
 Figure 14-4. Screenshot of pandas/matplotlib-based plot with interactive elements

Real-Time Plots

The previous subsection shows how easy it is to generate interactive,

web-based plots with Bokeh . However, Bokeh shines when it comes to
real-time visualization of, for example, high-frequency financial data.
Therefore, this subsection contains examples for two different real-time
APIs, one for FX (foreign exchange) data in JSON ( JavaScript Object
Notation ) format and one for intraday tick data for stock prices deliv-
ered in CSV text file format. Apart from the visualization aspect, how to
read out data from such APIs is also of interest.

Real-time FX data
Our first example is based on a JSON API for, among others, FX rates.
Some imports first:

In [51]: import time

import pandas as pd
import datetime as dt
import requests

The API we use is from OANDA, an FX online broker. This broker offers
an API sandbox that provides random/dummy data that resembles real
exchange rates. Our example is based on the EUR–USD exchange rate (cf.
the API guide):

In [52]: url = '[Link]

# real-time FX (dummy!) data from JSON API

To connect to the API we use the requests library whose aim is to im-
prove the interface for “humans” when interacting with web resources:

In [53]: instrument = 'EUR_USD'

api = [Link](url % instrument)

With the open connection, data in JSON format is simply read by calling
the method json on the connection object:

In [54]: data = [Link]()

data

Out[54]: {u'prices': [{u'ask': 1.25829,

u'bid': 1.2582,
u'instrument': u'EUR_USD',
u'time': u'2014-09-29T06:14:34.749878Z'}]}

Unfortunately, the data is not yet completely in the format we would like
it to have. Therefore, we transform it a bit. The following code takes only
the first element of the list object stored under the key “prices.” The re-
sulting object is a standard dict object:
 In [55]: data = data['prices'][0]
data

Out[55]: {u'ask': 1.25829,

u'bid': 1.2582,
u'instrument': u'EUR_USD',
u'time': u'2014-09-29T06:14:34.749878Z'}

Since we collect such small data sets at a high frequency, we use a

DataFrame object to store all the data. The following code initializes an
appropriate DataFrame object:

In [56]: ticks = [Link]({'bid': data['bid'],

'ask': data['ask'],
'instrument': data['instrument'],
'time': [Link](data['time'])},
index=[[Link](data['time']),])
# initialization of ticks DataFrame

In [57]: ticks[['ask', 'bid', 'instrument']]

Out[57]: ask bid instrument

2014-09-29 06:14:34.749878+00:00 1.25829 1.2582 EUR_USD

Implementing a real-time plot requires two things: real-time data collec-

tion and real-time updates of the plot. With Bokeh , this is accomplished
by using the Bokeh server, which handles real-time updates of a plot
given new data. It has to be started via the shell or command-line inter-
face as follows:

$ bokeh-server

With the server running in the background, let us implement the real-
time data update routine:

In [58]: import [Link] as bp

from [Link] import Glyph
Before any updating takes place, there needs to be an object to be up-
dated. This again is a line plot—if only with very little data at first. The
output is directed to the IPython Notebook the code is executed in.
However, in fact it is redirected again to the server, which in this case can
be accessed locally via [Link]

In [59]: bp.output_notebook("default")
[Link](ticks['time'], ticks['bid'],
x_axis_type='datetime', legend=instrument)

Out[59]: Using saved session configuration for [Link]

To override, pass 'load_from_config=False' to Session

<[Link] at 0x7fdb7e1b2e10>

We need to get access to our current plot (i.e., the most recently generated
plot). Calling the function curplot returns the object we are looking for:

In [60]: [Link]()

Out[60]: <[Link] at 0x7fdb7e1b2e10>

Such a Plot object consists of a number of rendering objects that accom-

plish different plotting tasks, like plotting a Grid or plotting the line (=
Glyph ) representing the financial data. All rendering objects are stored
in a list attribute called renderers :

In [61]: [Link]().renderers

Out[61]: [<[Link] at 0x7fdbaece6b50>,

<[Link] at 0x7fdb7e161190>,
<[Link] at 0x7fdb7e161090>,
<[Link] at 0x7fdb7e1614d0>,
<[Link] at 0x7fdb7e161490>,
<[Link] at 0x7fdb7e161550>,
 <[Link] at 0x7fdb7e161650>,
<[Link] at 0x7fdb7e161610>]

The following list comprehension returns the first rendering object of

type Glyph :

In [62]: renderer = [r for r in [Link]().renderers

if isinstance(r, Glyph)][0]

The glyph attribute of the object contains the type of the Glyph object—
in this case, as expected, a Line object:

In [63]: [Link]

Out[63]: <[Link] at 0x7fdb7e161590>

With the rendering object, we can access its data source directly:

In [64]: renderer.data_source

Out[64]: <[Link] at 0x7fdb7e1b2ed0>

In [65]: renderer.data_source.data

Out[65]: {'x': 2014-09-29 06:14:34.749878+00:00 2014-09-29 06:14:34.749878+00

:00
Name: time, dtype: object, 'y': 2014-09-29 06:14:34.749878+00:00 1
2582
Name: bid, dtype: float64}

In [66]: ds = renderer.data_source

This is the object that we will work with and that is to be updated when-
ever new data arrives. The following while loop runs for a predeter-
mined period of time only. During the loop, a new request object is gener-
ated and the JSON data is read. The new data is appended to the existing
DataFrame object. The x and y coordinates of the rendering object are
updated and then stored to the current session:

In [67]: start = [Link]()

# run for 60 seconds
while ([Link]() - start) < 60:
data = [Link](url % instrument).json()
# connect and read data
data = dict(data['prices'][0])
# transform data to dict object
ticks = [Link]([Link]({'bid': data['bid'],
'ask': data['ask'],
'instrument': data['instrument'],
'time': [Link](data['time'])},
index=[[Link](data['time']),]))
# append DataFrame object with new data to existing object
[Link]['x'] = ticks['time']
# update x coordinates in rendering object
[Link]['y'] = ticks['bid']
# update y coordinates in rendering object
[Link]().store_objects(ds)
# store data objects
[Link](0.1)
# wait for a bit

Figure 14-5 shows the output of the plotting exercise—i.e., a static snap-
shot of a real-time plot. This approach and the underlying technology of
course have many interesting application areas, both in finance, with its
focus today on real-time, high-frequency data, and far beyond.
 Figure 14-5. Screenshot of real-time Bokeh plot via Bokeh Server (exchange rate)

Real-time stock price quotes

The second example uses real-time, high-frequency stock price data. First,
make sure to correctly direct the output (i.e., in this case to the Bokeh
server for the real-time plot):

In [68]: bp.output_notebook("default")

Out[68]: Using saved session configuration for [Link]

To override, pass 'load_from_config=False' to Session

Chapter 6 provides an example based on the data source and API that we
use in what follows. It is the stock price API for intraday real-time data
provided by Netfonds, a Norwegian online broker. The API and web ser-
vice, respectively, have the following basic URL format:
 In [69]: url1 = '[Link]
url2 = 'date=%s%s%s&paper=%s.O&csv_format=csv'
url = url1 + url2

This URL is to be customized by providing date information and the sym-

bol one is interested in:

In [70]: today = [Link]()

y = '%d' % [Link]
# current year
m = '%02d' % [Link]
# current month, add leading zero if needed
d = '%02d' % ([Link])
# current day, add leading zero if needed
sym = 'AAPL'
# Apple Inc. stocks

In [71]: y, m, d, sym

Out[71]: ('2014', '09', '29', 'AAPL')

In [72]: urlreq = url % (y, m, d, sym)

urlreq

Out[72]: '[Link]
rmat=csv'

Equipped with the right URL string, retrieving data is only one line of
code away:

In [73]: data = pd.read_csv(urlreq, parse_dates=['time'])

# initialize DataFrame object

The details of what follows are known from the previous example. First,
the initial plot:
 In [74]: [Link](data['time'], data['bid'],
x_axis_type='datetime', legend=sym)
# intial plot

Out[74]: <[Link] at 0x7f92bedc8dd0>

Second, selection of the rendering object:

In [75]: renderer = [r for r in [Link]().renderers

if isinstance(r, Glyph)][0]
ds = renderer.data_source

Third, the while loop updating the financial data and the plot per loop:

In [76]: start = [Link]()

while ([Link]() - start) < 60:
data = pd.read_csv(urlreq, parse_dates=['time'])
data = data[data['time'] > [Link](int(y), int(m), int(d),
10, 0, 0)]
# only data from trading start at 10am
[Link]['x'] = data['time']
[Link]['y'] = data['bid']
ds._dirty = True
[Link]().store_objects(ds)
[Link](0.5)

Figure 14-6 shows the resulting output—again, unfortunately, only a static

snapshot of a real-time plot.
 Figure 14-6. Screenshot of real-time Bokeh plot via Bokeh Server (stock quotes)

Rapid Web Applications

If the Python world were to be divided into continents, there might be,
among others, the science and finance content, the system administration
continent, and for sure the web development continent. Although not re-
ally transparent, it is highly probable that the web development conti-
nent, to stay with this concept, might be one of the largest when it comes
to people (developers) populating it and houses (applications) built on it.

One of the major reasons for Python being strong in web development is
the availability of different high-level, full-stack frameworks. As the
Python web page states:
 A web application may use a combination of a base HTTP applica-
tion server, a storage mechanism such as a database, a template
engine, a request dispatcher, an authentication module and an
AJAX toolkit. These can be individual components or be provided
together in a high-level framework.

Among the most popular frameworks are:

Django
Flask
Pyramid/Pylons
TurboGears
Zope

It is safe to say that there is not a single framework that is best suited for
everybody and every different application type.[56] All have their
strengths (and sometimes weaknesses), and often it is more a matter of
taste (regarding architecture, style, syntax, APIs, etc.) what framework is
chosen.

One framework that has recently gained popularity quite rapidly is

Flask . It is the framework we use here, mainly for the following reasons:

Pythonic
Application development with Flask is really Pythonic , with a lot of the
web-related details being taken care of behind the scenes.
Compactness
It is not too complex and can therefore be learned quite rapidly; it is based
mainly on standard components and libraries widely used elsewhere.
Documentation
It is well documented, with both an online HTML version and a PDF with
around 300 pages available at the time of this writing.[57]

The two main libraries that Flask relies on are:

Jinja2 , a web templating language/engine for Python

Werkzeug , a WSGI ( Web Server Gateway Interface ) toolkit for
Python

Traders’ Chat Room

We will now dive into the example application called Tradechat for a
traders’ chat room, which basically relies on the example used in the tuto-
rial of the Flask documentation but includes a couple of changes and
adds some further functionality.[58]

The basic idea is to build a web-based application for which traders can
register that provides one central chat room to exchange ideas and talk
markets. The main screen shall allow a user who is logged in to type in
text that is, after pushing a button, added to the timeline, indicating who
added the comment and when this happened. The main screen also
shows all the historical entries in descending order (from newest to
oldest).

Data Modeling

We start by generating the needed directories. tradechat shall be the

main directory. In addition, at a minimum, we need the two subdirecto-
ries static and templates (by Flask convention):

$ mkdir tradechat
$ mkdir tradechat/static
$ mkdir tradechat/templates

To store data—both for registered users and for comments made in the
chat room—we use SQLite3 (cf. [Link] and
[Link] as a database. Two different
tables are needed that can be generated by the SQL schema presented in
Example 14-1, the details of which we do not discuss here. You should
store this under the filename [Link] in the main directory of the ap-
plication, tradechat .

Example 14-1. SQL schema to generate tables in SQLite3

drop table if exists comments;

create table comments (
id integer primary key autoincrement,
comment text not null,
user text not null,
time text not null
);
 drop table if exists users;
create table users (
id integer primary key autoincrement,
name text not null,
password text not null
);

The Python Code

The SQL schema is a main input for the Python/Flask application to fol-
low. We will go through the single elements step by step to finally arrive
at the complete Python script to be stored under [Link] in the
main directory, tradechat .

Imports and database preliminaries

At the beginning we need to import a couple of libraries and also some

main functions from Flask . We import the functions directly to shorten
the code throughout and increase readability somewhat:

# Tradechat
#
# A simple example for a web-based chat room
# based on Flask and SQLite3.
#

import os
import datetime as dt
from sqlite3 import dbapi2 as sqlite3
from flask import Flask, request, session, g, redirect, url_for, abort, \
render_template, flash

The whole application hinges on a Flask object, an instance of the main

class of the framework. Instantiating the class with name lets the object
inherit the application name (i.e., main ) when the script is executed, for
example, from a shell:

# the application object from the main Flask class

app = Flask(__name__)
The next step is to do some configuration for the new application object.
In particular, we need to provide a database filename:

# override config from environment variable

[Link](dict(
DATABASE=[Link](app.root_path, '[Link]'),
# the SQLite3 database file ("TC database")
DEBUG=True,
SECRET_KEY='secret_key',
# use secure key here for real applications
))
[Link].from_envvar('TC_SETTINGS', silent=True)
# do not complain if no config file exists

Having provided the path and filename of the database, the function
connect_db connects to the database and returns the connection object:

def connect_db():
''' Connects to the TC database.'''
rv = [Link]([Link]['DATABASE'])
rv.row_factory = [Link]
return rv

Flask uses an object called g to store global data and other objects. For
example, web applications serving large numbers of users make it neces-
sary to connect regularly to databases. It would be inefficient to instanti-
ate a connection object every time a database operation has to be exe-
cuted. One can rather store such a connection object in the attribute
sqlite_db of the g object. The function get_db makes use of this ap-
proach in that a new database connection is opened only when there is no
connection object stored in the g object already:

def get_db():
''' Opens a new connection to the TC database. '''
if not hasattr(g, 'sqlite_db'):
# open only if none exists yet
g.sqlite_db = connect_db()
return g.sqlite_db
At least once, we need to create the tables in the database. Calling the
function init_db for a second time will delete all information previously
stored in the database (according to the SQL schema used):

def init_db():
''' Creates the TC database tables.'''
with app.app_context():
db = get_db()
with app.open_resource('[Link]', mode='r') as f:
[Link]().executescript([Link]())
# creates entries and users tables
[Link]()

The function close_db closes the database connection if one exists in the
g object. For the first time (and for sure not the last time), we encounter
a Flask function decorator, i.e., @app.teardown_appcontext . This deco-
rator ensures that the respective function is called whenever the applica-
tion context tears down—that is, roughly speaking, when the execution of
the application is terminated by the user or by an error/exception:

@app.teardown_appcontext
def close_db(error):
''' Closes the TC database at the end of the request. '''
if hasattr(g, 'sqlite_db'):
g.sqlite_db.close()

Core functionality

Building on the database infrastructure, we can now proceed and imple-

ment the core functionality for the application. First, we have to define
what happens when we connect to the main/home page of the applica-
tion. To this end, we use the Flask function decorator @[Link]("/") .
The function decorated in that way will be called whenever a connection
is established to the main page. The function show_entries basically es-
tablishes a database connection, retrieves all comments posted so far
(maybe none, maybe many), and sends them to a template-based render-
ing engine to return an HTML document based on the template and the
data provided (more on the templating part soon):
 @[Link]('/')
def show_entries():
''' Renders all entries of the TC database. '''
db = get_db()
query = 'select comment, user, time from comments order by id desc'
cursor = [Link](query)
comments = [Link]()
return render_template('show_entries.html', comments=comments)

We only want to allow registered users to post comments in the chat

room. Therefore, we must provide functionality for a user to register. To
this end, technically, we must allow use of the POST method for the re-
spective HTML to be rendered by the application and to be accessed by the
user. To register, a user must provide a username and a password.
Otherwise, an error is reported. The function register should be consid-
ered a simple illustration only. It is missing a number of ingredients im-
portant for real-world applications, like checking whether a username al-
ready exists and encryption of the passwords (they are stored as plain
text). Once users have successfully registered, their status is automati-
cally changed to logged_in and they are redirected to the main page via
redirect(url_for("show_entries")) :

@[Link]('/register', methods=['GET', 'POST'])

def register():
''' Registers a new user in the TC database. '''
error = None
if [Link] == 'POST':
db = get_db()
if [Link]['username'] == '' or [Link]['password'] == '':
error = 'Provide both a username and a password.'
# both fields have to be nonempty
else:
[Link]('insert into users (name, password) values (?, ?)',
[[Link]['username'], [Link]['password']])
[Link]()
session['logged_in'] = True
# directly log in new user
flash('You were sucessfully registered.')
[Link](dict(USERNAME=[Link]['username']))
return redirect(url_for('show_entries'))
return render_template('[Link]', error=error)
For such a web application, there are probably returning users that do
not need or want to reregister anew. We therefore need to provide a form
to log in with an existing account. This is what the function login does.
The functionality is similar to that provided by register :

@[Link]('/login', methods=['GET', 'POST'])

def login():
''' Logs in a user. '''
error = None
if [Link] == 'POST':
db = get_db()
try:
query = 'select id from users where name = ? and password = ?'
id = [Link](query, ([Link]['username'],
[Link]['password'])).fetchone()[0]
# fails if record with provided username and password
# is not found
session['logged_in'] = True
flash('You are now logged in.')
[Link](dict(USERNAME=[Link]['username']))
return redirect(url_for('show_entries'))
except:
error = 'User not found or wrong password.'
return render_template('[Link]', error=error)

Once users have registered or logged in again, they should be able to add
comments in the chat room. The function add_entry stores the comment
text, the username of the user who commented, and the exact time (to the
second) of the posting. The function also checks whether the user is
logged in or not:

@[Link]('/add', methods=['POST'])
def add_entry():
''' Adds entry to the TC database. '''
if not [Link]('logged_in'):
abort(401)
db = get_db()
now = [Link]()
[Link]('insert into comments (comment, user, time) values (?, ?, ?)',
[[Link]['text'], [Link]['USERNAME'], str(now)[:-7]])
[Link]()
 flash('Your comment was successfully added.')
return redirect(url_for('show_entries'))

Finally, to end the session, the user must log out. This is what the function
logout supports:

@[Link]('/logout')
def logout():
''' Logs out the current user. '''
[Link]('logged_in', None)
flash('You were logged out')
return redirect(url_for('show_entries'))

If we want to run the Python script as a standalone application we

should add the following lines, which make sure that a server is fired up
and that the application is served:

# main routine
if __name__ == '__main__':
init_db() # comment out if data in current
# TC database is to be kept
[Link]()

Putting all these pieces together, we end up with the Python script
shown as Example 14-2.

Example 14-2. Python script embodying the core of the Tradechat application

# Tradechat
#
# A simple example for a web-based chat room
# based on Flask and SQLite3.
#

import os
import datetime as dt
from sqlite3 import dbapi2 as sqlite3
from flask import Flask, request, session, g, redirect, url_for, abort, \
render_template, flash
# the application object from the main Flask class
app = Flask(__name__)

# override config from environment variable

[Link](dict(
DATABASE=[Link](app.root_path, '[Link]'),
# the SQLite3 database file ("TC database")
DEBUG=True,
SECRET_KEY='secret_key',
# use secure key here for real applications
))
[Link].from_envvar('TC_SETTINGS', silent=True)
# do not complain if no config file exists

def connect_db():
''' Connects to the TC database.'''
rv = [Link]([Link]['DATABASE'])
rv.row_factory = [Link]
return rv

def get_db():
''' Opens a new connection to the TC database. '''
if not hasattr(g, 'sqlite_db'):
# open only if none exists yet
g.sqlite_db = connect_db()
return g.sqlite_db

def init_db():
''' Creates the TC database tables.'''
with app.app_context():
db = get_db()
with app.open_resource('[Link]', mode='r') as f:
[Link]().executescript([Link]())
# creates entries and users tables
[Link]()

@app.teardown_appcontext
def close_db(error):
''' Closes the TC database at the end of the request. '''
if hasattr(g, 'sqlite_db'):
g.sqlite_db.close()
@[Link]('/')
def show_entries():
''' Renders all entries of the TC database. '''
db = get_db()
query = 'select comment, user, time from comments order by id desc'
cursor = [Link](query)
comments = [Link]()
return render_template('show_entries.html', comments=comments)

@[Link]('/register', methods=['GET', 'POST'])

def register():
''' Registers a new user in the TC database. '''
error = None
if [Link] == 'POST':
db = get_db()
if [Link]['username'] == '' or [Link]['password'] == '':
error = 'Provide both a username and a password.'
# both fields have to be nonempty
else:
[Link]('insert into users (name, password) values (?, ?)',
[[Link]['username'], [Link]['password']])
[Link]()
session['logged_in'] = True
# directly log in new user
flash('You were sucessfully registered.')
[Link](dict(USERNAME=[Link]['username']))
return redirect(url_for('show_entries'))
return render_template('[Link]', error=error)

@[Link]('/login', methods=['GET', 'POST'])

def login():
''' Logs in a user. '''
error = None
if [Link] == 'POST':
db = get_db()
try:
query = 'select id from users where name = ? and password = ?'
id = [Link](query, ([Link]['username'],
[Link]['password'])).fetchone()[0]
# fails if record with provided username and password
# is not found
session['logged_in'] = True
 flash('You are now logged in.')
[Link](dict(USERNAME=[Link]['username']))
return redirect(url_for('show_entries'))
except:
error = 'User not found or wrong password.'
return render_template('[Link]', error=error)

@[Link]('/add', methods=['POST'])
def add_entry():
''' Adds entry to the TC database. '''
if not [Link]('logged_in'):
abort(401)
db = get_db()
now = [Link]()
[Link]('insert into comments (comment, user, time) values (?, ?, ?)',
[[Link]['text'], [Link]['USERNAME'], str(now)[:-7]])
[Link]()
flash('Your comment was successfully added.')
return redirect(url_for('show_entries'))

@[Link]('/logout')
def logout():
''' Logs out the current user. '''
[Link]('logged_in', None)
flash('You were logged out')
return redirect(url_for('show_entries'))

# main routine
if __name__ == '__main__':
init_db() # comment out if data in current
# TC database is to be kept
[Link]()

SECURITY

Although the example in this section illustrates the basic design of a web applica-
tion in Python with Flask , it barely addresses security issues, which are of para-
mount importance when it comes to web applications. However, Flask and other
web frameworks provide complete tool sets to tackle typical security issues (e.g.,
encryption) with due diligence.
Templating

Basically, templating with Flask ( Jinja2 ) works similarly to simple

string replacements in Python : you have a basic string indicating
where to replace what and some data to be inserted into the string ob-
ject. Consider the following examples:

In [77]: '%d, %d, %d' % (1, 2, 3)

Out[77]: '1, 2, 3'

In [78]: '{}, {}, {}'.format(1, 2, 3)

Out[78]: '1, 2, 3'

In [79]: '{}, {}, {}'.format(*'123')

Out[79]: '1, 2, 3'

Templating to generate HTML pages works pretty similarly. The major dif-
ference is that the string object “resembles” an HTML document (or a
part thereof) and has commands for replacements and also, for example,
ways of controlling the flow when rendering the template (e.g., the for
loop). Missing information is added during the rendering procedure, as
we added the integers to the string object in the previous examples.
Consider now the following string object, containing partly standard
HTML code and some template-specific code:

In [80]: templ = '''<!doctype html>

Just print out <b>numbers</b> provided to the template.
<br><br>
{% for number in numbers %}
{{ number }}
{% endfor %}
'''
So far, this is a string object only. We have to generate a Jinja2
Template object out of it before proceeding:

In [81]: from jinja2 import Template

In [82]: t = Template(templ)

This Template object has a method called render to make valid HTML
code out of the template and some input values—in this case, some num-
bers via the parameter numbers :

In [83]: html = [Link](numbers=range(5))

The code is again a string object:

In [84]: html

Out[84]: u'<!doctype html>\n Just print out <b>numbers</b> provided to the temp
late.\n <br><br>\n \n 0\n \n 1\n \n 2\n \n 3\n \n
4\n '

Such an object containing HTML code can be rendered in IPython

Notebook as follows:

In [85]: from [Link] import HTML

HTML(html)

Out[85]: <[Link] at 0x7fdb7e1eb890>

Of course, templating involves much more than this simple example can
illustrate (e.g., inheritance). More details can be found at
[Link] However, the templates for the Tradechat applica-
tion already include a number of important aspects. Specifically, we need
the following templates:
 [Link]
Defines the basic layout from which the other templates inherit
[Link]
The template for the user registration page
[Link]
The corresponding template for the user login
show_entries.html
The main page showing the comments in the chat room and, if the user is
logged in, the text field for writing and posting comments

These files have to be stored in templates , the default (sub)directory for

templates when using Flask .

Example 14-3 shows the template containing the basic layout and some
meta-information (like the site title). This is the template all other tem-
plates inherit from.

Example 14-3. Template for basic layout of Tradechat application

<!doctype html>
<title>Tradechat</title>
<link rel=stylesheet type=text/css
href="{{ url_for('static', filename='[Link]') }}">
<div class=page>
<h1>Tradechat</h1>
<div class=metanav>
{% if not session.logged_in %}
<a href="{{ url_for('login') }}">log in</a><br>
<a href="{{ url_for('register') }}">register</a>
{% else %}
<a href="{{ url_for('logout') }}">log out</a>
{% endif %}
</div>
{% for message in get_flashed_messages() %}
<div class=flash>{{ message }}</div>
{% endfor %}
{% block body %}{% endblock %}
</div>

Figure 14-7 shows a screenshot of the main page after starting the appli-
cation for the first time. No users are registered (or logged in, of course).
No comments have been posted yet.
 Figure 14-7. Screenshot of “empty” home page of Tradechat

Example 14-4 provides the templating code for the user registration page.
Here, forms are used to allow users to provide information to the page
via the POST method.

Example 14-4. Template for Tradechat user registration

{% extends "[Link]" %}
{% block body %}
<h2>Register</h2>
{% if error %}<p class=error><strong>Error:</strong> {{ error }}{% endif %}
<form action="{{ url_for('register') }}" method=post>
<dl>
<dd><font size="-1">Username</font>
<dd><input type=text name=username>
<dd><font size="-1">Password</font>
<dd><input type=password name=password>
<dd><input type=submit value=Register>
</dl>
</form>
{% endblock %}

Figure 14-8 shows a screenshot of the registration page.

 Figure 14-8. Screenshot of Tradechat registration page

The templating code for the login page, as shown in Example 14-5, is
pretty similar to the code for the registration page. Again, the user can
provide login information via a form.

Example 14-5. Template for Tradechat user login

{% extends "[Link]" %}
{% block body %}
<h2>Login</h2>
{% if error %}<p class=error><strong>Error:</strong> {{ error }}{% endif %}
<form action="{{ url_for('login') }}" method=post>
<dl>
<dd><font size="-1">Username</font>
<dd><input type=text name=username>
<dd><font size="-1">Password</font>
<dd><input type=password name=password>
<dd><input type=submit value=Login>
</dl>
</form>
{% endblock %}

The login page, as shown in Figure 14-9, not only looks pretty similar to
the registration page but also provides mainly the same functionality.
 Figure 14-9. Screenshot of Tradechat login page

Finally, Example 14-6 provides the templating code for the main page.
This template does mainly two things:

Enables commenting
If the user is logged in, a text field and a Post button are shown to allow the
user to post comments.
Displays comments
All comments found in the database are displayed in reverse chronological
order (newest first, oldest last).

Example 14-6. Template for Tradechat main page with chat room comments

{% extends "[Link]" %}
{% block body %}
{% if session.logged_in %}
<form action="{{ url_for('add_entry') }}" method=post class=add-comment>
<dl>
<dd>What's up?
<dd><textarea name=text rows=3 cols=40></textarea>
<dd><input type=submit value=Post>
</dl>
</form>
{% endif %}
<ul class=comments>
{% for comment in comments %}
 <li>{{ [Link]|safe }}
<font size="-2">({{ [Link] }} @ {{ [Link] }})</font>
{% else %}
<li><em>No comments so far.</em>
{% endfor %}
</ul>
{% endblock %}

Once a user is logged in and has posted some comments, the main page
shows the text field and the Post button as well as all comments stored
in the database (cf. Figure 14-10).

Just showing the screenshots in combination with the templates is cheat-

ing, in a sense. What is missing in the mix is the styling information.

Figure 14-10. Screenshot of Tradechat main page

Styling

Today’s standard when it comes to the styling of web pages and web-
based applications is CSS ( Cascading Style Sheets ). If you take a
closer look at the single templates, you will find in many places parame-
terizations like class=comments or class=add-comment . Without a cor-
responding CSS file, these parameterizations are essentially
meaningless.

Therefore, let us have a look at the file [Link] , stored in the

(sub)directory static and shown in Example 14-7. Here you find the
aforementioned parameters ( comments , add-comment ) again. You also
find references to standard HTML tags, like h1 for the highest-ranking
header. All information provided after a custom class name, like com‐
ments , or a standard tag, like h1 , defines or changes certain style ele-
ments (e.g., font type and/or size) of the relevant object.

This style information is the final ingredient defining the look of the
Tradechat application and explaining why, for example, the “Tradechat”
heading is displayed in blue (namely, due to the line a, h1, h2 { color:
#0066cc; } ).

Example 14-7. CSS stylesheet for Tradechat application

body { font-family: sans-serif; background: #eee; }

a, h1, h2 { color: #0066cc; }
h1, h2 { font-family: 'Helvetica', sans-serif; margin: 0; }
h1 { font-size: 1.4em; border-bottom: 2px solid #eee; }
h2 { font-size: 1.0em; }

.page { margin: 2em auto; width: 35em; border: 1px solid #ccc;
padding: 0.8em; background: white; }
.comments { list-style: none; margin: 0; padding: 0; }
.comments li { margin: 0.8em 1.2em; }
.comments li h2 { margin-left: -1em; }
.add-comment { color: #0066cc; font-size: 0.7em; border-bottom: 1px solid #ccc;
.add-comment dl { font-weight: bold; }
.metanav { text-align: right; font-size: 0.8em; padding: 0.3em;
margin-bottom: 1em; background: #fafafa; }
.flash { color: #b9b9b9; font-size: 0.7em; }
.error { color: #ff4629; font-size: 0.7em; padding: 0.5em; }

If you have followed every step, your tradechat directory should now
contain the same files listed here:

In [86]: import os
for path, dirs, files in [Link]('../python/tradechat'):
 print path
for f in files:
print f

Out[86]: ../python/tradechat
[Link]
[Link]
[Link]
../python/tradechat/static
[Link]
../python/tradechat/templates
[Link]
[Link]
[Link]
show_entries.html

You can now run the main script from the shell as follows and start the
application:

$ python [Link]

You can then access the application via your web browser at
[Link] Click on register to register as a user, and after
having provided a username and a password you will be able to post
your comments.

Web Services

The last topic in this chapter—and a very interesting and important one—
is web services. Web services provide a simple and efficient means to ac-
cess server-based functionality via web protocols. For example, one of the
web services with the highest traffic is the Google search functionality.
We are used to visiting [Link] and typing some words of
interest into the search/text input field provided on the website.
However, what happens after you press the Return key or push the
Search button is that the page translates all the information it has (from
the search field and maybe your personal preferences) into a more or less
complex URL .
Such a URL could, for example, take on the form
[Link] When you click this
link or copy it into your web browser, Google Search returns those five
search results ( num=5 ) that the engine considers the best matches given
the words provided ( q=Yves+Python ). Your web browser then displays
something similar to Figure 14-11.

Using web services, any kind of data- and transaction-oriented financial

service can be provided via web technologies. For instance, Yahoo!
Finance and Google Finance offer historical stock price information via
such a web service approach. More complex services such as derivatives
pricing and risk analytics are also available via such services (for exam-
ple, the web-based analytics solution DEXISION; cf. [Link]
[Link]). The following example illustrates the implementation of
such a service in the context of option pricing.

Figure 14-11. Screenshot of Google search results via web service

The Financial Model

In this section, we are going to implement a web service that allows us to

value volatility options (e.g., on a volatility index). The model we use is
the one of Gruenbichler and Longstaff (1996). They model the volatility
process (e.g., the process of a volatility index) in direct fashion by a
square-root diffusion, provided in Equation 14-1. This process is known to
exhibit convenient features for volatility modeling, like positivity and
mean reversion.[59]

Equation 14-1. Square-root diffusion for volatility modeling

The variables and parameters in Equation 14-1 have the following

meanings:

Vt
The time t value of the volatility index (for example, the VSTOXX)
θV
The long-run mean of the volatility index
κV
The rate at which Vt reverts to 𝜃
ΣV
The volatility of the volatility (“vol-vol”)
θV, κV, and ΣV
Assumed to be constant and positive
Zt
A standard Brownian motion

Based on this model, Gruenbichler and Longstaff (1996) derive the for-
mula provided in Equation 14-2 for the value of a European call option. In
the formula, D(T) is the appropriate discount factor. The parameter 𝜁 de-

notes the expected premium for volatility risk, while is the com-
plementary noncentral 𝜒2 distribution.

Equation 14-2. Call option formula of Gruenbichler and Longstaff (1996)

The Implementation

The translation of the formula as presented in Equation 14-2 to Python

is, as usual, quite straightforward. Example 14-8 shows the code of a
Python module with such a valuation function. We call the script
vol_pricing_formula.py and store it in a sub-directory, volservice .

Example 14-8. Python script for volatility option valuation

#
# Valuation of European volatility call options
# in Gruenbichler-Longstaff (1996) model
# square-root diffusion framework
# -- semianalytical formula
#
from [Link] import ncx2
import numpy as np

# Semianalytical option pricing formula of GL96

def calculate_option_value(V0, kappa, theta, sigma, zeta, T, r, K):

''' Calculation of European call option price in GL96 model.

Parameters
==========
V0 : float
current volatility level
kappa : float
 mean reversion factor
theta : float
long-run mean of volatility
sigma : float
volatility of volatility
zeta :
volatility risk premium
T : float
time-to-maturity
r : float
risk-free short rate
K : float
strike price of the option

Returns
=======
value : float
net present value of volatility call option
'''
D = [Link](-r * T) # discount factor

# variables
alpha = kappa * theta
beta = kappa + zeta
gamma = 4 * beta / (sigma ** 2 * (1 - [Link](-beta * T)))
nu = 4 * alpha / sigma ** 2
lamb = gamma * [Link](-beta * T) * V0
cx1 = 1 - [Link](gamma * K, nu + 4, lamb)
cx2 = 1 - [Link](gamma * K, nu + 2, lamb)
cx3 = 1 - [Link](gamma * K, nu, lamb)

# formula for European call price

value = (D * [Link](-beta * T) * V0 * cx1
+ D * (alpha / beta) * (1 - [Link](-beta * T))
* cx2 - D * K * cx3)
return value

To simplify the implementation of the web service we write a conve-

nience function, get_option_value , which will check for the provision
of all needed parameters to calculate a call option value. The function is
stored in a Python module called vol_pricing_service.py , the code of
which is shown in Example 14-9. This script also contains a dictionary
with all the necessary parameters and brief descriptions of these parame-
ters. The function will return an error message detailing what is missing
whenever one or more parameters are missing. If all necessary parame-
ters are provided during the web service call, the function calls the pric-
ing function calculate_option_value from the
vol_pricing_formula.py script.

Example 14-9. Python script for volatility option valuation and web service helper
function

#
# Valuation of European volatility options
# in Gruenbichler-Longstaff (1996) model
# square-root diffusion framework
# -- parameter dictionary & web service function
#
from vol_pricing_formula import calculate_option_value

# model parameters

PARAMS={
'V0' : 'current volatility level',
'kappa' : 'mean reversion factor',
'theta' : 'long-run mean of volatility',
'sigma' : 'volatility of volatility',
'zeta' : 'factor of the expected volatility risk premium',
'T' : 'time horizon in years',
'r' : 'risk-free interest rate',
'K' : 'strike'
}

# function for web service

def get_option_value(data):
''' A helper function for web service. '''
errorline = 'Missing parameter %s (%s)\n'
errormsg = ''
for para in PARAMS:
if not data.has_key(para):
# check if all parameters are provided
errormsg += errorline % (para, PARAMS[para])
if errormsg != '':
return errormsg
else:
 result = calculate_option_value(
float(data['V0']),
float(data['kappa']),
float(data['theta']),
float(data['sigma']),
float(data['zeta']),
float(data['T']),
float(data['r']),
float(data['K'])
)
return str(result)

To begin with, we add the path of the aforementioned Python scripts:

In [87]: import sys

[Link]("../python/volservice")
# adjust if necessary to your path

We use the library Werkzeug to handle our WSGI application-based web

service (recall that Werkzeug is an integral part of Flask ). To this end,
we need to import some functions from Werkzeug sublibraries:

In [88]: from [Link] import Request, Response

Furthermore, for our core WSGI application to follow, we need the func-
tion get_option_value that we defined earlier:

In [89]: from vol_pricing_service import get_option_value

The only thing that remains is to implement the WSGI application (func-
tion) itself. This function might in our case look as follows:

In [90]: def application(environ, start_response):

request = Request(environ)
# wrap environ in new object
text = get_option_value([Link])
# provide all parameters of call to function
# get back either error message or option value
response = Response(text, mimetype='text/html')
 # generate response object based on the returned text
return response(environ, start_response)

Here, environ is a dictionary containing all incoming information. The

Request function wraps all information in a manner that makes access-
ing the environ information a bit more convenient. start_response is
usually used to indicate the start of a response. However, with Werkzeug
you have the Response function, which takes care of the response.

All parameters provided to the web service are found in the

[Link] attribute, and this is what we provide to the
get_option_value function. This function returns either an error mes-
sage in text form or the calculated option value in text form.

To be better able to serve this function (e.g., via a local web server), we
put the function into a separate WSGI script and add the serving function-
ality to it. Example 14-10 shows the code of this script, called
vol_pricing.py .

Example 14-10. Python script for volatility option valuation and web service helper
function

#
# Valuation of European volatility options
# in Gruenbichler-Longstaff (1996) model
# square-root diffusion framework
# -- WSGI application for web service
#
from vol_pricing_service import get_option_value
from [Link] import Request, Response
from [Link] import run_simple

def application(environ, start_response):

request = Request(environ)
# wrap environ in new object
text = get_option_value([Link])
# provide all parameters of call to function
# get back either error message or option value
response = Response(text, mimetype='text/html')
# generate response object based on the returned text
return response(environ, start_response)
 if __name__=='__main__':
run_simple('localhost', 4000, application)

Being in the right subdirectory ( volservice ), you can now start the ap-
plication by executing the following command via the shell or command-
line interface:

$ python vol_pricing.py
* Running on [Link]

This fires up a separate Python process that serves the WSGI application.
Using urllib , we can now access the “full power” of the web service.
Copying the URL in your web browser and pressing the Return key yields
something like the result shown in Figure 14-12.

Figure 14-12. Screenshot of the error message of the web service

However, usually you want to use a web service quite a bit differently—
for example, from a scripting environment like IPython . To this end, we
can use the functionality the urllib library provides:

In [91]: import numpy as np

import urllib
url = '[Link]
A simple call to the web service without providing any parameters re-
turns the following error message, which (apart from formatting issues)
is the same as in the screenshot in Figure 14-12:

In [92]: print [Link](url).read()

Out[92]: Missing parameter V0 (current volatility level)

Missing parameter r (risk-free interest rate)
Missing parameter kappa (mean-reversion factor)
Missing parameter T (time horizon in years)
Missing parameter theta (long-run mean of volatility)
Missing parameter zeta (factor of the expected volatility risk premium)
Missing parameter sigma (volatility of volatility)
Missing parameter K (strike)

Of course, we need to provide a number of parameters. Therefore, we

first build a URL string object in which we can replace specific parameter
values during later calls:

In [93]: urlpara = url + 'application?V0=%s&kappa=%s&theta=%s&sigma=%s&zeta=%s'

urlpara += '&T=%s&r=%s&K=%s'

A possible parameterization might be the following one:

In [94]: urlval = urlpara % (25, 2.0, 20, 1.0, 0.0, 1.5, 0.02, 22.5)
urlval

Out[94]: '[Link]
eta=0.0&T=1.5&r=0.02&K=22.5'

Using this particular URL string returns an option value, as desired:

In [95]: print [Link](urlval).read()

Out[95]: 0.202937705934
With such a web service, you can of course do multiple calls to calculate
multiple option values quite easily:

In [96]: %%time
urlpara = '[Link]
urlpara += '&theta=25&sigma=1.0&zeta=0.0&T=1&r=0.02&K=%s'
strikes = [Link](20, 30, 50)
results = []
for K in strikes:
[Link](float([Link](urlpara % K).read()))
results = [Link](results)

Out[96]: CPU times: user 64 ms, sys: 20 ms, total: 84 ms

Wall time: 196 ms

In [97]: results

Out[97]: array([ 4.91296701, 4.71661296, 4.52120153, 4.32692516, 4.1339945 ,

3.94264561, 3.75313813, 3.56575972, 3.38079846, 3.19858765,
3.01946028, 2.8437621 , 2.67184576, 2.50406508, 2.34078693,
2.18230495, 2.02898213, 1.88111287, 1.738968 , 1.60280064,
1.47281111, 1.34917004, 1.23204859, 1.12141092, 1.01739405,
0.9199686 , 0.82907686, 0.74462353, 0.66647327, 0.59445387,
0.52843174, 0.46798166, 0.41300694, 0.36319553, 0.31824647,
0.27785656, 0.24171678, 0.20951651, 0.18094732, 0.1557064 ,
0.1334996 , 0.11414975, 0.09710449, 0.08234678, 0.06958767,
0.05859317, 0.04915788, 0.04109348, 0.03422854, 0.02840802]

One advantage of this approach is that you do not use your local re-
sources to get the results, but rather the resources of a web server—
which might also use, for example, parallelization techniques. Of course,
in our example all is local and the web service uses the local computing
resources. Figure 14-13 shows the valuation results graphically, conclud-
ing this section:

In [98]: import [Link] as plt

%matplotlib inline
[Link](strikes, results, 'b')
[Link](strikes, results, 'ro')
 [Link](True)
[Link]('strike')
[Link]('European call option value')

Figure 14-13. Value of European volatility call option for different strikes

W E B S E RV I C E S A R C H I T E C T U R E

The web services architecture is often a powerful and efficient alternative to the
provision of Python -based analytical functionality, or even whole applications.
This holds true for the Internet as well as for models where private networks are
used. This architecture also simplifies updates and maintenance, since such ser-
vices are generally provided in a centralized fashion.

Conclusions

Nowadays, web technologies are an integral part of almost any applica-

tion architecture. They are not only beneficial for communicating with
the outside world and providing simple to sophisticated web services to
external entities, but also within (financial) organizations.

This chapter first illustrates some basic techniques with regard to the
most common communication protocols (mainly FTP and HTTP ). It also
shows how to implement interactive web plotting, how to interface in real
time with web-based financial data APIs (e.g., JSON -based) and how to vi-
sualize such high frequency data in real time with Bokeh . These basic
tools and techniques are helpful in almost any context.
However, the Python ecosystem also provides a number of powerful,
high level frameworks to develop even complex web applications in rapid
fashion. We use Flask , a framework which has gained some popularity
recently, to implement a simple chat room for traders with simple user
administration (registration and login). All elements of a typical web ap-
plication—core functionality in Python , templating with Jinja2 , and
styling with CSS —are illustrated.

Finally, the last section in this chapter addresses the important topic of
web services. Using the Werkzeug library for a somewhat simplified han-
dling of WSGI applications, we implement a web-based pricing service for
volatility options based on the model and formula of Gruenbichler and
Longstaff (1996).

Further Reading

The following web resources are helpful with regard to the topics covered
in this chapter:

The Python documentation should be a starting point for the basic

tools and techniques shown in this chapter: [Link]
see also this overview page:
[Link]
You should consult the home page of Bokeh for more on this webfo-
cused plotting library: [Link]
For more on Flask , start with the home page of the framework:
[Link] also, download the PDF documentation:
[Link]
Apart from the Python documentation itself, consult the home
page of the Werkzeug library for more on web services:
[Link]

For a Flask reference in book form, see the following:

Grinberg, Miguel (2014): Flask Web Development—Developing Web

Applications with Python. O’Reilly, Sebastopol, CA.

Finally, here is the research paper about the valuation of volatility

options:
 Gruenbichler, Andreas and Francis Longstaff (1996): “Valuing
Futures and Options on Volatility.” Journal of Banking and Finance,
Vol. 20, pp. 985–1001.

[50] For details and background refer to [Link]

[51] For details and background refer to [Link]

[52] This example is for illustration purposes only. In general, you would want to
use specialized libraries such as lxml or Beautiful Soup .

[53]
There are alternatives to these libraries, like Requests , that come with a more
modern API.

[54] For more information on interactive plots with matplotlib , refer to the
library’s home page.

[55] The majority of graphics formats matplotlib can export to are static by na-
ture (i.e., bitmaps). A counterexample is graphics in SVG ( Scalable Vector
Graphics ) format, which can be programmed in JavaScript/ECMAScript . The
library’s website provides some examples of how to do this.

[56] See [Link] for further information on

Python web frameworks. See
[Link] for an overview of con-
tent management systems (CMSs) for Python .
[57] Although the framework is still quite recent (it all started in 2010), there are
already books about Flask available. Cf. Grinberg (2014).

[58] The example application is called Flaskr and represents a microblog applica-
tion. Our example is, more or less, a mixture between Flaskr and Minitwit , an-
other Flask example application resembling a simple Twitter clone.

[59]
See also the larger case study about volatility options presented in Chapter 19.

Support Sign Out

©2022 O'REILLY MEDIA, INC. TERMS OF SERVICE PRIVACY POLICY