Download Python Crash Course – Programming tools

Python Crash Course – Programming tools
Python Crash Course – Programming tools
Give me six hours to chop down a tree
and I will spend the first four sharpening the axe.
Abraham Lincoln
Steffen Brinkmann
Max-Planck-Institut für Astronomie, Heidelberg
IMPRESS, 2016
1 / 60
Python Crash Course – Programming tools
The ingredients
I
A working Python installation
I
Internet connection
I
Passion for Python
If anything of the above is missing, please say so now!
2 / 60
Python Crash Course – Programming tools
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
3 / 60
Python Crash Course – Programming tools
The actual problem
Who you are
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
4 / 60
Python Crash Course – Programming tools
The actual problem
Who you are
5 / 60
Python Crash Course – Programming tools
The actual problem
Who you are
6 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
7 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
The problem
I
A computer scientist is the analyst, designer, programmer,
maintainer, tester, support and customer.
I
You cannot do all of this and do science and publish.
I
But you can do much more than you do now with the help of
tools.
All of the problems in the comics are communication problems.
Either with your (future) self or with your collaborators.
8 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
What you do now
The (modified) Joel test
Do you manage your source code, version branches and
release cycles using a version control system?
Examples for version control systems are cvs, svn, git and
mercurium.
Can you make a build in one step?
A build means getting from source code to a distributable package
or executable. One step means one command on the command
line or one click of a button.
http://www.joelonsoftware.com/articles/fog0000000043.html
9 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
What you do now
The (modified) Joel test
Do you use a continuous integration and automated testing
platform?
Examples for CI and automated testing software are Jenkins and
Hudson
Do you make daily builds?
Imagine a colleague asks for your code as you arrive at the office.
Could you pass him your code within a minute and be sure she
gets it to work (and it works as expected) without further
interaction from your side.
http://www.joelonsoftware.com/articles/fog0000000043.html
10 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
What you do now
The (modified) Joel test
Do you have a bug database?
Do you document bugs and feature requests at some well
accessible place?
Do you fix bugs before writing new code?
Is fixing bugs the first priority?
http://www.joelonsoftware.com/articles/fog0000000043.html
11 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
What you do now
The (modified) Joel test
Are input and output formats standardized? Do you use
common file formats, such as csv, fits, png or do you have a written
document that specifies the input and output comprehensively?
Do you provide comprehensive source and usage
documentation? Is every function, method, class module and file
of your code documented in-code, e.g. with robodoc or doxygen?
Do you provide README files and/or pdf documents as a manual
on how to install and use your program?
http://www.joelonsoftware.com/articles/fog0000000043.html
12 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
What you do now
The (modified) Joel test
How many questions did you answer with Yes?
0-3 Look no further. This course is for you.
over 3 You’re on the right track. There is always room for
enhancements.
http://www.joelonsoftware.com/articles/fog0000000043.html
13 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
What you want to do from now on
I
Defensive programming
I
Use a version control system
I
Document your code
I
Write tests
I
Automate the build process
I
Automate the tests
14 / 60
Python Crash Course – Programming tools
The actual problem
Who you can be
how you want to do it from now on
I
Defensive programming → programming skills
I
Use a version control system → git
I
Document your code → doxygen
I
Write tests → doctests, unittest
I
Automate the build process → bash, configure, make
I
Automate the tests → Jenkins
15 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
16 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Defensive programming
I
Write comprehensible code
I
Check input and make sure, the program reacts in a
predictable manner
I
Throw and catch exceptions
I
Fix bugs before you implement the next feature
17 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Comprehensive code
I
Beautiful is better than ugly.
I
Explicit is better than implicit.
I
Simple is better than complex.
I
Complex is better than complicated.
I
Flat is better than nested.
I
https://www.python.org/dev/peps/pep-0020/
18 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Check input and intermediate results
I
Check for type of function arguments
I
Expect user input to be bogus
I
Check for empty strings, negative values etc.
I
Use assert <expression>[, <description>] statements.
It will throw an AssertionError if <expression> is False
I
In case of an error, print error message to stderr and return or
exit with a well defined and documented return value
I
Include these cases in your testing routines
19 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Throw and catch expressions
def checkedDivide (a , b ):
if b == 0:
raise Exception ( " Do not divide by zero !!! " )
return a / b
def t e s t C h e c k e d D i v i d e (a , b ):
try :
result = checkedDivide ( a , b )
except Exception as e :
sys . stderr . write ( " Division Error : % s \ n " % str ( e ))
return math . nan
else :
sys . stderr . write ( " successful division \ n " )
finally :
sys . stderr . write ( " checked division performed .\ n " )
return result
https://docs.python.org/3/tutorial/errors.html
20 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Throw and catch expressions
def checkedDivide (a , b ):
if b == 0:
raise Exception ( " Do not divide by zero !!! " )
return a / b
def t e s t C h e c k e d D i v i d e (a , b ):
try :
result = checkedDivide ( a , b )
except Exception as e :
logging . error ( " Division Error : % s \ n " % str ( e ))
return math . nan
else :
logging . info ( " successful division \ n " )
finally :
logging . debug ( " checked division performed .\ n " )
return result
https://docs.python.org/3/howto/logging.html
21 / 60
Python Crash Course – Programming tools
Defensive programming
Some thoughts
Fix bugs
Find bugs : in-code checks, tests, debugging
Document bugs : Mention known bugs in a comment line. A
documented bug is better than an
undocumented one
Evaluate bugs : Does it effect one feature? Many features?
The whole program? Will the program crash?
Will the results be obviously wrong? Will the
results be slightly wrong? Will the user notice?
Fix bugs : Do not let the bugs propagate into future
versions. Fix the bug in every version (branch).
22 / 60
Python Crash Course – Programming tools
Version control
Introduction
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
23 / 60
Python Crash Course – Programming tools
Version control
Introduction
Introduction
I
Version (revision, source) control systems are tools to help
you manage the ever changing source you are working on.
I
A version is identified by a unique number or letter code.
I
A commit is a change to the source (i.e. in one or more files)
added to the current branch, making it the head revision of
that branch.
I
Branches are parallel development lines emerging from the
same commit as the result of forking.
I
Branches can (and mostly should) be merged at some point.
24 / 60
Python Crash Course – Programming tools
Version control
Introduction
Why and why not
Advantages
I
Easy recovery of former (working) versions.
I
Documentation of development progress in the commit
messages.
I
Branching enables you to work on different features at the
same time.
I
Branching enables different programmers to work in an
ordered fashion to work on the same program
I
Separate bug-fixes from adding new features and development
of new versions.
25 / 60
Python Crash Course – Programming tools
Version control
Introduction
Why and why not
Cost
I
Learn how to use a VCS: this course, once in a while half an
hour.
I
Set up the VCS (git): this course, once a few minutes
I
Set up a repository: once every time you start a new project 1
minute
I
Commit your changes: several times a day 30 seconds.
26 / 60
Python Crash Course – Programming tools
Version control
Hello git
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
27 / 60
Python Crash Course – Programming tools
Version control
Hello git
Why git
I
Used in many large projects: Linux kernel, Qt, Gnome, eclipse,
KDE, X11, Perl, Rails,. . .
I
Decentralised: Every local copy is a complete repository. No
need for a central server.
I
Very fast
I
Powerful branching and merging
28 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
One time set up of git
I
Install git
I
Documentation: http://www.git-scm.com/documentation
I
Set up you user name and email address:
$ git config -- global user . name " YOUR NAME "
$ git config -- global user . email " NAME@mpia . de "
29 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
Let’s start a new project
We started an new project:
# !/ bin / env python
from __future__ import print_function
print ( " hello git " )
Let’s set up a repository for it.
30 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
In the new project’s directory
Let’s set up a repository for the new project.
$ ls -A
hello_git . py
$ git init
Initialized empty Git repository in /.../. git /
$ ls -A
. git hello_git . py
31 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
In the new project’s directory
Let’s add the source file to the new repository.
$ git status
On branch master
Initial commit
Untracked files :
( use " git add < file >... " to include in what will be committed )
hello_git . py
nothing added to commit but untracked files present ( use " git add " to track )
$ git add hello_git . py
$ git status
On branch master
Initial commit
Changes to be committed :
( use " git rm -- cached < file >... " to unstage )
new file :
hello_git . py
32 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
In the new project’s directory
Let’s commit the source file to the new repository.
$ git commit -m " initial commit "
[ master ( root - commit ) bb32989 ] initial commit
1 file changed , 5 insertions (+)
create mode 100644 hello_git . py
$ git status
On branch master
nothing to commit , working directory clean
$ git log
commit b b 3 2 9 8 9 2 f 1 e 7 5 e 8 e 2 3 2 2 e 6 6 2 1 4 5 a d e f 7 8 4 3 6 4 6 d 7
Author : Steffen Brinkmann < brinkmann@mpia . de >
Date :
Tue Feb 23 20:49:31 2016 +0100
initial commit
33 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
git background
34 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
In the new project’s directory
Let’s change a source file and add a new one.
$ emacs hello_git . py
$ cat > README . md
hello_git
=========
A very simple program
$ git add hello_git . py README
$ git commit -m ’ added README and in - code documentation ’
[ master 9 ed1ab4 ] added README and in - code documentation
2 files changed , 5 insertions (+)
create mode 100644 README
35 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
going remote
I
I
I
I
Setting up a remote repository serves for backing up and
sharing.
Many free services available: gitlab, github, bitbucket,
sourceforge,. . .
Create the repository online
local set up (once):
$ git remote add origin git@gitlab . com : szs / test . git
$ git push -u origin master
Counting objects : 7 , done .
Delta compression using up to 2 threads .
Compressing objects : 100% (5/5) , done .
Writing objects : 100% (7/7) , 689 bytes | 0 bytes /s , done .
Total 7 ( delta 0) , reused 0 ( delta 0)
To git@gitlab . com : szs / test . git
* [ new branch ]
master -> master
Branch master set up to track remote branch master from origin .
https://en.wikipedia.org/wiki/Comparison of source code hosting facilities#Available version control systems 36 / 60
Python Crash Course – Programming tools
Version control
Hello git
Hello git
Further workflow
$ git pull
Already up - to - date .
$ emacs hello_git . py
$ git add hello_git . py
$ git commit -m ’ some more output ’
[ master edcaeef ] some more output
1 file changed , 1 insertion (+)
$ git push
Counting objects : 3 , done .
Delta compression using up to 2 threads .
Compressing objects : 100% (3/3) , done .
Writing objects : 100% (3/3) , 327 bytes | 0 bytes /s , done .
Total 3 ( delta 1) , reused 0 ( delta 0)
To git@gitlab . com : szs / test . git
9 ed1ab4 .. edcaeef master -> master
37 / 60
Python Crash Course – Programming tools
Version control
Hello git
git with branches
Next steps: branches
I
Branches enable you to work on several fixes/features at the
same time
I
Branches enable you to work with other programmers
I
Branches should be merged often
I
Commits should only be made to feature or fix branches
I
Merge these branches to a central development branch
I
Test the development branch
I
Merge the development branch to the master branch
I
The master branch should only contain versions which are
distributable
38 / 60
Python Crash Course – Programming tools
Version control
Hello git
git with branches
Next steps: branches
http://nvie.com/posts/a-successful-git-branching-model/
39 / 60
Python Crash Course – Programming tools
Version control
Hello git
git with branches
Exercises
Using
$ git help branch
$ git help checkout
$ git help merge
do the following:
I
Create a development branch called ”develop”
I
Create a feature branch called ”features”
I
Change the source and/or add files in two or more commits
I
Merge this branch back to ”development”
I
Merge the ”development” branch back to ”master”
40 / 60
Python Crash Course – Programming tools
Documentation
Documentation levels
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
41 / 60
Python Crash Course – Programming tools
Documentation
Documentation levels
Documentation levels
I
README, CHANGES, INSTALL, LICENSE files
I
in-code (doxygen, robodoc, docstrings)
I
technical documentation
I
usage manual
I
online manuals, cookbooks, examples
42 / 60
Python Crash Course – Programming tools
Documentation
Documentation levels
How much documentation is just enough
I
Code should explain itself
I
”Don’t comment your code, code your comments”
I
Explain what you do and why you do it, not how you do it.
Rules of thumb:
I
I
I
I
I
Every file gets a block comment at the beginning, describing
what its purpose is, and what the interface is
Every class, method and function gets a block comment
describing input, output, side effects (there should be none)
and additional information (e.g. literature)
Variables get a comment line, if they are of major importance
for the code
Code blocks get a comment if they implement a non-trivial
algorithm.
43 / 60
Python Crash Course – Programming tools
Documentation
docstrings in Python
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
44 / 60
Python Crash Course – Programming tools
Documentation
docstrings in Python
docstrings
I
In-code documentation for module, class, function or method
I
Enclosed by ’’’ or """
I
Is read by built-in documentation system
45 / 60
Python Crash Course – Programming tools
Documentation
docstrings in Python
docstrings
def inc (a , b =1):
’’’
inc (a , b =1) -> sum of a and b
inc ( a ) -> a +1
’’’
return a + b
>>> import mymodule
>>> help ( mymodule . inc )
46 / 60
Python Crash Course – Programming tools
Documentation
doxygen
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
47 / 60
Python Crash Course – Programming tools
Documentation
doxygen
Getting started
I
doxygen generates technical documentation from in code
documentation
I
It can generate minimal documentation from undocumented
code...
I
Or generate in-depth documentation from special markers in
your in-code documentation
$ doxygen -g doxygen . cfg
$ emacs doxygen . cfg
$ doxygen doxygen . cfg
48 / 60
Python Crash Course – Programming tools
Testing
Testing strategies
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
49 / 60
Python Crash Course – Programming tools
Testing
Testing strategies
Why test
I
Tests define the proper behaviour of the code.
I
Tests help you detect bugs due to side effects of new features
I
A reasonable strategy: Define tests before you start
programming: Test Driven Development (TDD)
Caution:
I
A passed test is no guarantee for correct code.
I
Correct code is not the same as good code (readability,
modularity, locality, portability, efficiency,. . . )
I
Seduces to program only what is necessary for the next test.
I
Not every requirement can be translated into a test.
50 / 60
Python Crash Course – Programming tools
Testing
Testing strategies
Testing levels
Unit testing : Verify the functionality of a specific section of
code, usually at function or class level.
Integration testing : Test interfaces between modules or similar
components (e.g. of a pipeline)
System testing : Check the whole set of modules, files,
programs, pipelines etc. to verify that it meets
its requirements.
51 / 60
Python Crash Course – Programming tools
Testing
doctests in Python
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
52 / 60
Python Crash Course – Programming tools
Testing
doctests in Python
doctests
I
In Python, unit tests can be defined within the docstrings
I
doctests are executed calling doctest.testmod()
53 / 60
Python Crash Course – Programming tools
Testing
doctests in Python
docstrings
"""
This is the factorial module .
The factorial module supplies one function , factorial ().
For example ,
>>> factorial (5)
120
"""
def factorial ( n ):
""" Return the factorial of n , an exact integer >= 0.
>>> [ factorial ( n ) for n in range (6)]
[1 , 1 , 2 , 6 , 24 , 120]
>>> factorial (30)
265252859812191058636308480000000 L
>>> factorial ( -1)
Traceback ( most recent call last ):
...
ValueError : n must be >= 0
"""
if n < 0: raise ValueError ( ’n must be >= 0 ’)
if n == 0: return 1
return n * factorial (n -1)
if __name__ == " __main__ " :
import doctest
doctest . testmod ()
54 / 60
Python Crash Course – Programming tools
Testing
doctests in Python
docstrings
"""
This is the factorial module .
The factorial module supplies one function , factorial ().
For example ,
>>> factorial (5)
120
"""
def factorial ( n ):
""" Return the factorial of n , an exact integer >= 0.
>>> [ factorial ( n ) for n in range (6)]
[1 , 1 , 2 , 6 , 24 , 120]
>>> factorial (30)
265252859812191058636308480000000 L
>>> factorial ( -1)
Traceback ( most recent call last ):
...
ValueError : n must be >= 0
"""
if n < 0: raise ValueError ( ’n must be >= 0 ’)
if n == 0: return 1
return n * factorial (n -1)
>>> python -m doctest test _doctest s1 . py
55 / 60
Python Crash Course – Programming tools
Testing
Unit tests in Python
Outline
The actual problem
Who you are
Who you can be
Defensive programming
Some thoughts
Version control
Introduction
Hello git
Documentation
Documentation levels
docstrings in Python
doxygen
Testing
Testing strategies
doctests in Python
Unit tests in Python
56 / 60
Python Crash Course – Programming tools
Testing
Unit tests in Python
import unittest
I
More complex unit tests can be done using the module
unittest
I
Derive a class from unittest.TestCase. . .
I
and implement methods starting with test.
These methods can call
I
I
I
I
I
assertEqual() to check for an expected result
assertTrue() or assertFalse() to verify a condition;
assertRaises() to check for raises exceptions
For more complex settings overwrite setUp() and
tearDown()
57 / 60
Python Crash Course – Programming tools
Testing
Unit tests in Python
docstrings
from test_doctests import factorial
import unittest
class TestFactorial ( unittest . TestCase ):
def test_valid ( self ):
self . assertEqual ( factorial (5) , 120)
self . assertEqual ([ factorial ( n ) for n in range (6)] , [1 , 1 , 2 , 6 , 24 , 120])
self . assertEqual ( factorial (30) , 2 6 5 2 5 2 8 5 9 8 1 2 1 9 1 0 5 8 6 3 6 3 0 8 4 8 0 0 0 0 0 0 0 )
def test_invalid ( self ):
self . assertRaises ( ValueError , factorial , -1)
if __name__ == ’ __main__ ’:
unittest . main ()
58 / 60
Python Crash Course – Programming tools
Testing
Unit tests in Python
docstrings
from test_doctests import factorial
import unittest
class TestFactorial ( unittest . TestCase ):
def test_valid ( self ):
self . assertEqual ( factorial (5) , 120)
self . assertEqual ([ factorial ( n ) for n in range (6)] , [1 , 1 , 2 , 6 , 24 , 120])
self . assertEqual ( factorial (30) , 2 6 5 2 5 2 8 5 9 8 1 2 1 9 1 0 5 8 6 3 6 3 0 8 4 8 0 0 0 0 0 0 0 )
def test_invalid ( self ):
self . assertRaises ( ValueError , factorial , -1)
>>> python -m unittest te st_unitt est1 . py
59 / 60
Python Crash Course – Programming tools
Now what?
I
I
Some action is better than no action!
Start slow:
I
I
I
I
I
git init in every new code directory
Only master branch
Get used to write in-code documentation: docstrings
use built scripts (makefiles, bash scripts)
Then, accelerate as you become comfortable.
60 / 60