Download Postprocessing with Python

Postprocessing with Python
Boris Dintrans (CNRS & University of Toulouse)
[email protected]
Collaborator: Thomas Gastine (PhD)
Outline
Outline
•
Introduction
- what’s Python and why using it?
- Installation procedure
Outline
•
Introduction
- what’s Python and why using it?
- Installation procedure
•
-
Python and the Pencil Code
the Python repository and initialization
Migrating from IDL to Python
Some examples & tricks
Parallel Python with Pypar
Doing widgets with PyQt
Outline
•
Introduction
- what’s Python and why using it?
- Installation procedure
•
-
Python and the Pencil Code
the Python repository and initialization
Migrating from IDL to Python
Some examples & tricks
Parallel Python with Pypar
Doing widgets with PyQt
•
Conclusion/Outlook
What’s Python?
What’s Python?
Python was created
in 1991 by Guido van
Rossum (CWI,
Centrum voor
Wiskunde en
Informatica,
Amsterdam)
Benevolent Dictator
for Life (BDFL)
What’s Python?
Python was created
in 1991 by Guido van
Rossum (CWI,
Centrum voor
Wiskunde en
Informatica,
Amsterdam)
Benevolent Dictator
for Life (BDFL)
What’s Python?
Python was created
in 1991 by Guido van
Rossum (CWI,
Centrum voor
Wiskunde en
Informatica,
Amsterdam)
Benevolent Dictator
for Life (BDFL)
The first release in 1991 on alt.sources
DARPA funding proposal “Computer Programming for
Everybody” (1999):
DARPA funding proposal “Computer Programming for
Everybody” (1999):
• an
easy and intuitive language just as powerful as major
competitors
• open source, so anyone can contribute to its development
• code that is as understandable as plain English
• suitability for everyday tasks, allowing for short development
times
DARPA funding proposal “Computer Programming for
Everybody” (1999):
• an
easy and intuitive language just as powerful as major
competitors
• open source, so anyone can contribute to its development
• code that is as understandable as plain English
• suitability for everyday tasks, allowing for short development
times
year
1991
1994
1995
2001
2003
2007
2009
version
0.9
1.0
1.2
2.0
2.2
2.5
3.0
DARPA funding proposal “Computer Programming for
Everybody” (1999):
• an
easy and intuitive language just as powerful as major
competitors
• open source, so anyone can contribute to its development
• code that is as understandable as plain English
• suitability for everyday tasks, allowing for short development
times
• Why
year
1991
1994
1995
2001
2003
2007
2009
version
0.9
1.0
1.2
2.0
2.2
2.5
3.0
Python?
- it’s free! ;-)
- quite easy to use; object-oriented; highly modular, etc...
- much more rapid than IDL and even PARALLEL
The main Python website: www.python.org
The SciPy website: www.scipy.org
How to install Python?
How to install Python?
Required:
• python 2.5: the engine
• numpy: the scientific computing package (arrays, linear
algebra, FFT, random numbers, etc...); [replaces old numarray
and numeric]
• scipy: modules for integrating ODEs, optimizing functions,
etc... [tends to federate all of Python scientific modules]
• matplotlib: MATLAB-inspired mostly-2D plotting modules
How to install Python?
Required:
• python 2.5: the engine
• numpy: the scientific computing package (arrays, linear
algebra, FFT, random numbers, etc...); [replaces old numarray
and numeric]
• scipy: modules for integrating ODEs, optimizing functions,
etc... [tends to federate all of Python scientific modules]
• matplotlib: MATLAB-inspired mostly-2D plotting modules
Optional:
• ipython: convenient shell to develop and run Python
• basemap: map projections
• Pypar: parallel Python (interface with MPI libraries)
• PyQt: to do Qt-like widgets VERY easily under Python
• MayaVi: 3D plotting
http://python.org/download/
From sources or binary packages?
From sources or binary packages?
•
•
For all platforms: everything can be compiled from sources
For Linux, Windows & Mac (at least): binaries are provided
(Linux: yum, apt-get, dpkg; Mac: Fink, MacPorts, dmg)
From sources or binary packages?
•
•
For all platforms: everything can be compiled from sources
For Linux, Windows & Mac (at least): binaries are provided
(Linux: yum, apt-get, dpkg; Mac: Fink, MacPorts, dmg)
Linux (FedoraCore 7)
Mac OS X 10.4 (Tiger)
Python
2.5.12
2.5.2
Numpy
1.0.3
1.0.4
Scipy
0.6.0
0.7.0
Matplotlib
0.90.0
0.90.1
ipython
0.8.1
0.8.2
Typical installation on a Linux box
Python on Mac OSX: www.pythonmac.org/packages/py25-fat
Scipy Superpack for OS X: http://macinscience.org/?page_id=6
Python packages on my Macbook Pro
(SciPy Superpack)
Python in the Pencil Code repository:
f90/pencil-code/numpy commited by Jeff in fall of 2007
revision 1.1
date: 2007-11-16 13:57:04 +0000; author: joishi; state: Exp;
* added python scripts for reading pencil code data. they require only the
numpy package, but matplotlib is useful for plotting. almost all of these
routines are simplified clones of their idl counterparts. i'd love to make a
more OO pencil-code package, but my current occupational constraints make
that unlikely in the near term. NB: the byte ordering in python is C, not
fortran, so these routines return an f array with shape f[nvar,nz,ny,nx]--the
reverse of pencil.
* added nl2python to take advantage of the amazing perl F90Namelist.pm
* modified F90Namelist.pm to output python
* i hope these are moderately useful to people!
The actual Python tree: 3 directories
The reading stuff
numpy/pencil/files
__init__.py
yzaver.py
yaver.py
xyaver.py
npfile.py
dim.py
param.py
grid.py
slices.py
zprof.py
index.py
var.py
ts.py
The actual Python tree: 3 directories
The reading stuff
numpy/pencil/files
__init__.py
yzaver.py
yaver.py
xyaver.py
npfile.py
dim.py
param.py
grid.py
slices.py
zprof.py
index.py
var.py
ts.py
The math stuff
numpy/pencil/math
__init__.py
vector_multiplication.py
derivatives/
numpy/pencil/math/derivatives
__init__.py
der.py
div_grad_curl.py
der_6th_order_w_ghosts.py
...and the initialization of $PYTHONPATH in
f90/pencil-code/sourceme.csh
# Set PYTHON path
if ($?PYTHONPATH) then
setenv PYTHONPATH "${PYTHONPATH}:${PENCIL_HOME}/numpy"
else
setenv PYTHONPATH "${PENCIL_HOME}/numpy"
endif
...and the initialization of $PYTHONPATH in
f90/pencil-code/sourceme.csh
# Set PYTHON path
if ($?PYTHONPATH) then
setenv PYTHONPATH "${PYTHONPATH}:${PENCIL_HOME}/numpy"
else
setenv PYTHONPATH "${PENCIL_HOME}/numpy"
endif
These modules are loaded
when importing the whole
pencil directory due to
the __init__.py file
cat numpy/pencil/
__init__.py
cat numpy/pencil/
__init__.py
In [1]: import pencil as pc
In [2]: pc.read_ts()
An important point: Python’s classes
•
Python is an object-oriented interpreted language:
instead of doing pc.read_ts(),
it is better to do a=pc.read_ts()
An important point: Python’s classes
•
Python is an object-oriented interpreted language:
instead of doing pc.read_ts(),
it is better to do a=pc.read_ts()
... and we can plot the
other variables read in
time_series.dat and
embedded in object ‘a’
Another example when
using pc.read_var()
Another example when
using pc.read_var()
...and we plot the
entropy at the top of
the 32^3 box
Another examples of postprocesing with Python
MayaVi:
3D plots
Basemap: various kind of
map projections
http://code.enthought.com/projects/mayavi/
Be careful: Python’s
arrays are ordered like
f[nvar,mz,my,mx]
i.e. REVERSED ORDER
COMPARED TO PENCILCODE OR IDL!!!
Migrating from IDL to Python: some useful Web Guides
http://www.stsci.edu/resources/software_hardware/numarray/idl2numarray
http://mathesaurus.sourceforge.net/idl-numpy.html
For lazy guys: the i2py converter
http://code.google.com/p/i2py/
Some tricks when using Python...
Some tricks when using Python...
•
plays with ~/.ipython/ipythonrc to load modules by default
(import_all pencil) and thus use ‘read_var’ instead of
‘pc.read_var()’, etc...
Some tricks when using Python...
•
plays with ~/.ipython/ipythonrc to load modules by default
(import_all pencil) and thus use ‘read_var’ instead of
‘pc.read_var()’, etc...
•
import just what you need! (a cleaning is certainly needed in
that respect in the PC tree...)
Some tricks when using Python...
•
plays with ~/.ipython/ipythonrc to load modules by default
(import_all pencil) and thus use ‘read_var’ instead of
‘pc.read_var()’, etc...
•
import just what you need! (a cleaning is certainly needed in
that respect in the PC tree...)
•
launch ipython with the ‘-pylab’ option to call directly plot,
contour, imshow, etc...
Some tricks when using Python...
•
plays with ~/.ipython/ipythonrc to load modules by default
(import_all pencil) and thus use ‘read_var’ instead of
‘pc.read_var()’, etc...
•
import just what you need! (a cleaning is certainly needed in
that respect in the PC tree...)
•
launch ipython with the ‘-pylab’ option to call directly plot,
contour, imshow, etc...
•
accelerate the VAR* reading by passing param, grid, index,
etc... [tricks.py]
Some tricks when using Python...
•
plays with ~/.ipython/ipythonrc to load modules by default
(import_all pencil) and thus use ‘read_var’ instead of
‘pc.read_var()’, etc...
•
import just what you need! (a cleaning is certainly needed in
that respect in the PC tree...)
•
launch ipython with the ‘-pylab’ option to call directly plot,
contour, imshow, etc...
•
accelerate the VAR* reading by passing param, grid, index,
etc... [tricks.py]
•
accelerate the graphics by using an handle [tricks.py]
Some tricks when using Python...
•
plays with ~/.ipython/ipythonrc to load modules by default
(import_all pencil) and thus use ‘read_var’ instead of
‘pc.read_var()’, etc...
•
import just what you need! (a cleaning is certainly needed in
that respect in the PC tree...)
•
launch ipython with the ‘-pylab’ option to call directly plot,
contour, imshow, etc...
•
accelerate the VAR* reading by passing param, grid, index,
etc... [tricks.py]
•
•
accelerate the graphics by using an handle [tricks.py]
take advantage of class and objects (a.shape instead of
shape(a))
Parallel Python using the Pypar module
http://sourceforge.net/projects/pypar
Pypar example 1: compute a vertical profile in parallel
Pypar example 2: write PNG files in parallel for a
movie
Widgets using Qt Designer + PyQt
Conclusion/Outlook
Conclusion/Outlook
•
Python can do a very good job in the Pencil Code postprocessing
• Its using is rapidly increasing in astrophysics (NASA, ESA,
ESO, labs,...)
• More in the Pencil Code philosophy (i.e. under GPL)
compared to IDL
Conclusion/Outlook
•
Python can do a very good job in the Pencil Code postprocessing
• Its using is rapidly increasing in astrophysics (NASA, ESA,
ESO, labs,...)
• More in the Pencil Code philosophy (i.e. under GPL)
compared to IDL
•
the actual Python subroutines must be rewritten in a more
oriented-object form (class inheritance)
•the Python tree shall maybe be re-organized in something
like f90/pencil-code/python or???
• what’s about the calling of Fortran or C subroutines to
increase the speed?