Documentation
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Adam Spannbauer
Machine Learning Engineer at Eastman
Documentation in Python
Comments
# Square the number x
Docstrings
"""Square the number x
:param x: number to square
:return: x squared
>>> square(2)
4
"""
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Comments
# This is a valid comment
x = 2
y = 3 # This is also a valid comment
# You can't see me unless you look at the source code
# Hi future collaborators!!
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Effective comments
Commenting 'what'
# Define people as 5
people = 5
# Multiply people by 3
people * 3
Commenting 'why'
# There will be 5 people attending the party
people = 5
# We need 3 pieces of pizza per person
people * 3
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Docstrings
def function(x):
"""High level description of function
Additional details on function
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Docstrings
def function(x):
"""High level description of function
Additional details on function
:param x: description of parameter x
:return: description of return value
Example webpage generated from a docstring in the Flask package.
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Docstrings
def function(x):
"""High level description of function
Additional details on function
:param x: description of parameter x
:return: description of return value
>>> # Example function usage
Expected output of example function usage
"""
# function code
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Example docstring
def square(x):
"""Square the number x
:param x: number to square
:return: x squared
>>> square(2)
4
"""
# `x * x` is faster than `x ** 2`
# reference: [Link]
return x * x
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Example docstring output
help(square)
square(x)
Square the number x
:param x: number to square
:return: x squared
>>> square(2)
4
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Let's Practice
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Readability counts
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Adam Spannbauer
Machine Learning Engineer
The Zen of Python
import this
The Zen of Python, by Tim Peters (abridged)
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
The complex is better than complicated.
Readability counts.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Descriptive naming
Poor naming
def check(x, y=100):
return x >= y
Descriptive naming
def is_boiling(temp, boiling_point=100):
return temp >= boiling_point
Going overboard
def check_if_temperature_is_above_boiling_point(
temperature_to_check,
celsius_water_boiling_point=100):
return temperature_to_check >= celsius_water_boiling_point
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Keep it simple
The Zen of Python, by Tim Peters (abridged)
Simple is better than complex.
Complex is better than complicated.
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Making a pizza - complex
def make_pizza(ingredients):
# Make dough
dough = mix(ingredients['yeast'],
ingredients['flour'],
ingredients['water'],
ingredients['salt'],
ingredients['shortening'])
kneaded_dough = knead(dough)
risen_dough = prove(kneaded_dough)
# Make sauce
sauce_base = sautee(ingredients['onion'],
ingredients['garlic'],
ingredients['olive oil'])
sauce_mixture = combine(sauce_base,
ingredients['tomato_paste'],
ingredients['water'],
ingredients['spices'])
sauce = simmer(sauce_mixture)
...
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Making a pizza - simple
def make_pizza(ingredients):
dough = make_dough(ingredients)
sauce = make_sauce(ingredients)
assembled_pizza = assemble_pizza(dough, sauce, ingredients)
return bake(assembled_pizza)
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
When to refactor
Poor naming
def check(x, y=100):
return x >= y
Descriptive naming
def is_boiling(temp, boiling_point=100):
return temp >= boiling_point
Going overboard
def check_if_temperature_is_above_boiling_point(
temperature_to_check,
celsius_water_boiling_point=100):
return temperature_to_check >= celsius_water_boiling_point
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Let's Practice
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Unit testing
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Adam Spannbauer
Machine Learning Engineer at Eastman
Why testing?
Con rm code is working as intended
Ensure changes in one function don't break another
Protect against changes in a dependency
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Testing in Python
doctest
pytest
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Using doctest
def square(x):
"""Square the number x
:param x: number to square
:return: x squared
>>> square(3)
9
"""
return x ** x
import doctest
[Link]()
Failed example:
square(3)
Expected:
9
Got:
27
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
pytest structure
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
pytest structure
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Writing unit tests
working in workdir/tests/test_document.py
from text_analyzer import Document
# Test tokens attribute on Document object
def test_document_tokens():
doc = Document('a e i o u')
assert [Link] == ['a', 'e', 'i', 'o', 'u']
# Test edge case of blank document
def test_document_empty():
doc = Document('')
assert [Link] == []
assert doc.word_counts == Counter()
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Writing unit tests
# Create 2 identical Document objects
doc_a = Document('a e i o u')
doc_b = Document('a e i o u')
# Check if objects are ==
print(doc_a == doc_b)
# Check if attributes are ==
print(doc_a.tokens == doc_b.tokens)
print(doc_a.word_counts == doc_b.word_counts)
False
True
True
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Running pytest
working with terminal
datacamp@server:~/work_dir $ pytest
collected 2 items
tests/test_document.py .. [100%]
========== 2 passed in 0.61 seconds ==========
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Running pytest
working with terminal
datacamp@server:~/work_dir $ pytest tests/test_document.py
collected 2 items
tests/test_document.py .. [100%]
========== 2 passed in 0.61 seconds ==========
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Failing tests
working with terminal
datacamp@server:~/work_dir $ pytest
collected 2 items
tests/test_document.py F.
============== FAILURES ==============
________ test_document_tokens ________
def test_document_tokens(): doc = Document('a e i o u')
assert [Link] == ['a', 'e', 'i', 'o']
E AssertionError: assert ['a', 'e', 'i', 'o', 'u'] == ['a', 'e', 'i', 'o']
E Left contains more items, first extra item: 'u'
E Use -v to get the full diff
tests/test_document.py:7: AssertionError
====== 1 failed in 0.57 seconds ======
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Let's Practice
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Documentation &
testing in practice
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Adam Spannbauer
Machine Learning Engineer at Eastman
Documenting projects with Sphinx
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Documenting classes
class Document:
"""Analyze text data
:param text: text to analyze
:ivar text: text originally passed to the instance on creation
:ivar tokens: Parsed list of words from text
:ivar word_counts: Counter containing counts of hashtags used in text
"""
def __init__(self, text):
...
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Continuous integration testing
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Continuous integration testing
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Links and additional tools
Sphinx - Generate beautiful documentation
Travis CI - Continuously test your code
GitHub & GitLab - Host your projects with git
Codecov - Discover where to improve your projects tests
Code Climate - Analyze your code for improvements in readability
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Let's Practice
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Final Thoughts
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N
Adam Spannbauer
Machine Learning Engineer at Eastman
Looking Back
Modularity
def function()
...
class Class:
...
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Looking Back
Modularity
"""docstrings"""
Documentation
# Comments
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Looking Back
Modularity
Documentation
Automated testing
def f(x):
"""
>>> f(x)
expected output
"""
...
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Data Science & Software Engineering
SOFTWARE ENGINEERING FOR DATA SCIENTISTS IN PYTHON
Good Luck!
S O F T WA R E E N G I N E E R I N G F O R D ATA S C I E N T I S T S I N P Y T H O N