Download Single Source Python 2 3

Single Source Python 2 3
May 21, 2016
1
Single Source Python 2/3
1.1
A Tutorial at PyData Berlin
Dr. Mike Müller Python Academy GmbH & Co. KG
[email protected]
@pyacademy
PyData Berlin 2016
Berlin, Germany
2
Overview
•
•
•
•
•
•
•
3
Motivation
Major differences between Python 2 and 3
Conversion strategies
Single source
Roll your own
Use a library
python-future.org
Motivation
•
•
•
•
Python 3.0 was released 2008
Python 2 (2.7) still widely used
Supporting only major Python version is often not an option
Single source can be a good solution
1
4
Overview of important Python 2/3 differences
•
•
•
•
•
5
Many small changes
You might not encounter many of them
Lot’s of cleanups of “warts”
Many (partially) backported to Python 2.7
Very few real syntax incompatibility between 2 and 3
print is a function
•
•
•
•
likely the most obvious one
but is pretty easy to fix
from future import print function
print(value, ..., sep=’ ’, end=’\n’, file=sys.stdout, flush=False)
In [3]: for x in range(10):
print(x, end=’ ’)
0 1 2 3 4 5 6 7 8 9
In [4]: print(*range(10), sep=’ ’)
0 1 2 3 4 5 6 7 8 9
6
Iterators are everywhere
• zip()
• map()
In [6]: zip(’abc’, ’xyz’)
Out[6]: <zip at 0x1050ae548>
In [7]: list(zip(’abc’, ’xyz’))
Out[7]: [(’a’, ’x’), (’b’, ’y’), (’c’, ’z’)]
In [8]: map(str.upper, ’abc’)
Out[8]: <map at 0x1050ad198>
In [9]: list(map(str.upper, ’abc’))
Out[9]: [’A’, ’B’, ’C’]
7
range() a.k.a. xrange() ++
• range() just like xrange()
• no limit
• slice-able
In [10]: range(10)
Out[10]: range(0, 10)
2
In [13]: range(int(1e100))
Out[13]: range(0, 10000000000000000159028911097599180468360808563945281389781327557747838772170381060813
In [14]: %%python2
xrange(int(1e100))
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
OverflowError: Python int too large to convert to C long
In [15]: r = range(10)
r[2:7]
Out[15]: range(2, 7)
In [16]: %%python2
xr = xrange(10)
xr[2:7]
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
TypeError: sequence index must be integer, not ’slice’
8
Key views
•
•
•
•
dict.keys() returns a view
Python 2: list
views are more efficient
they update
In [28]: d = {’a’: 100, ’b’: 200, ’c’: 300}
k = d.keys()
k
Out[28]: dict keys([’b’, ’a’, ’c’])
In [29]: d[’x’] = 400
k
Out[29]: dict keys([’b’, ’x’, ’a’, ’c’])
9
No apple-orange comparisons
• only defined comparisons
• for many the most important change
In [18]: 1 < ’1’
--------------------------------------------------------------------------TypeError
Traceback (most recent call last)
3
<ipython-input-18-9ee3bb6f2cf3> in <module>()
----> 1 1 < ’1’
TypeError: unorderable types: int() < str()
In [20]: %%python2
print(1 < ’1’)
True
In [21]: 100 < None
--------------------------------------------------------------------------TypeError
Traceback (most recent call last)
<ipython-input-21-ab68f0474b07> in <module>()
----> 1 100 < None
TypeError: unorderable types: int() < NoneType()
In [22]: %%python2
print(100 < None)
False
10
Integers without limits
• long became int
• no big change
• difference has been small in Python 2
In [23]: 10000000000000000000000000
Out[23]: 10000000000000000000000000
In [25]: %%python2
print(repr(10000000000000000000000000))
10000000000000000000000000L
11
Mathematical division
• divide integers just like in math by default
• from future import division
In [30]: 1 / 2
Out[30]: 0.5
4
In [32]: 1 // 2
Out[32]: 0
In [33]: %%python2
print(1 / 2)
print(1 // 2)
0
0
In [34]: %%python2
from __future__ import division
print(1 / 2)
print(1 // 2)
0.5
0
12
Strings are always unicode
• Maybe the biggest change, depending on the use case
• Python 2: weak distinction between bytes and strings
13
Prefixing
• Python 2:
• ’abc’: bytestring
• u’abc’: unicode string
• Python 3:
• ’abc’: string unicode
• b’abc’: bytestring
• both:
• u’abc’: string unicode
• b’abc’: bytestring
14
Source code encoding
• default encodings:
• Python 2 ascii
• Python 3 utf-8
• both, explicit: # -*- coding:
utf-8 -*-
5
15
•
•
•
•
•
•
•
•
•
16
Cleanup - Modern Python 2 can do
no classic classes anymore
no backticks, use repr()
‘<>’ gone
exception instances only with as, old comma syntax is gone
literal octal numbers with 0o prefix, e.g. 0o7
open() is now io.open()
use next() not obj.next()
use functools.reduce() instead of buil-in reduce()
use func(*args) instead of apply(func, args)
Cleanup - Modern Python 2 can do with a little help
• raw input() –> input(), input() –> eval(input())
• importlib.reload() instead of reload()
17
•
•
•
•
18
•
•
•
•
•
•
19
Cleanup - Modern Python 2 can do with help of a library
exec statement turned into a function
metaclasses as keyword argument in class definition, no metaclass anymore
super() without class and self
round() uses a different rule round(2.5) == 2, instead of round(2.5) == 3
New syntax - Python 3 only
keyword-only arguments
nonlocal
extended unpacking of iterables
True, False, and None are keywords, yeah
variables from list comprehensions don’t leak anymore, great
exception instances only available in except block
Re-structuring of the standard library
• Removal of old, unused modules gopherlib
• renaming to comply with PEP8, ConfigParser–> configparser, StringIO moved into io
• grouping of related modules like http or dbm
20
•
•
•
•
•
•
•
Write your own compatibility layer
use modern Python 2 (exceptions, next)
future
version testing
re-defining built-ins
standard lib packports
handling strings and bytes
IO
6
21
Never use classic classes
In [73]: class A(object):
pass
22
•
•
•
•
•
Don’t use ancient features
no backticks, use repr()
never use ‘<>’
prefix literal octal numbers with 0o prefix, e.g. 0o7
use functools.reduce() instead of buil-in reduce()
use func(*args) instead of apply(func, args)
In [136]: %%python2
a = ’a’
print(‘a‘)
’a’
In [79]: a = ’a’
print(‘a‘)
File "<ipython-input-79-2c78c9df6889>", line 2
print(‘a‘)
^
SyntaxError: invalid syntax
In [81]: a = ’a’
print(repr(a))
’a’
23
Exceptions always with as
In [55]: %%python2
try:
1 / 0
except ZeroDivisionError, err:
pass
print(err)
integer division or modulo by zero
In [56]: try:
1 / 0
except ZeroDivisionError, err:
pass
print(err)
7
File "<ipython-input-56-d4e7da7453b9>", line 3
except ZeroDivisionError, err:
^
SyntaxError: invalid syntax
In [82]: try:
1 / 0
except ZeroDivisionError as err:
pass
In [83]: %%python2
try:
1 / 0
except ZeroDivisionError as err:
pass
24
Use io.open
• you can specify an encoding
In [84]: import io
io.open is open
Out[84]: True
In [87]: %%python2
from io import open
In [88]: open(file, mode=’r’, buffering=-1, encoding=None, errors=None, newline=None, closefd=True, open
25
Use next()
In [93]: i = iter(’abc’)
print(next(i))
a
In [92]: %%python2
i = iter(’abc’)
print(next(i))
a
26
Travel to the
future
• add this line at the beginning of all your source files
In [137]: from __future__ import absolute_import, division, print_function
8
27
Unicode litertals can be problematic
In [138]: from __future__ import unicode_literals
•
•
•
•
28
not always recommended
good for 3 –> 2
much less useful for 2 –> 3
e.g., may cause problems with library calls (e.g. paths)
Version Testing
• always test for < 3 or > 2
• never == 2 or == 3
• Python 4 will be comming
29
Re-defining built-ins
In [106]: %%python2
from __future__ import absolute_import, division, print_function
import sys
if sys.version_info.major < 3:
input = raw_input
range = xrange
from io import open
from itertools import izip as zip, imap as map, ifilter as filter
print(filter)
<type ’itertools.ifilter’>
30
Standard library packports
• several libraries are backported
• lru cache: backports.functools lru cache
• pathlib2
In [107]: try:
from functools import lru_cache
except ImportError:
from backports.functools_lru_cache import lru_cache
31
Your own compatibility layer
• use of
all
seems justified
In [ ]: # %load compat.py
from __future__ import absolute_import, division, print_function
9
import sys
__all__ = []
if sys.version_info.major < 3:
input = raw_input
range = xrange
from io import open
from itertools import (izip as zip, imap as map,
ifilter as filter)
__all__ = [’input’, ’filter’, ’map’, ’open’,
’range’, ’zip’]
In [109]: %%python3
from compat import *
print(range)
<class ’range’>
In [110]: %%python2
from compat import *
print(range)
<type ’xrange’>
32
Using help
• six Python 2.5 and up, latest version only 2.6
• future Python 2.6 and up
33
•
•
•
•
34
•
•
•
•
•
35
Using python-future.org
http://python-future.org/
does the work for you
and is much better at it
makes Python 2 behave largely like Python 3
Options
starting from scratch
starting from Python 2
starting from Python 3
automatic conversion
porting philosophy
Starting from scratch
• at the beginng of your source files add:
10
In [112]: from __future__ import (absolute_import, division,
print_function, unicode_literals)
from builtins import *
• write standard Python 3
• without th Python 3 only parts
36
Starting from Python 2
In [ ]: # %load py2sample.py
import StringIO
fobj = StringIO.StringIO("""Hello world
Hallo Welt""")
line1 = fobj.next()
print ’Done’
In [ ]: $ futurize -w py2sample.py
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored py2sample.py
--- py2sample.py
(original)
+++ py2sample.py
(refactored)
@@ -1,10 +1,13 @@
from __future__ import print_function
+from future import standard_library
+standard_library.install_aliases()
+from builtins import next
-import StringIO
+import io
-fobj = StringIO.StringIO("""Hello world
+fobj = io.StringIO("""Hello world
Hallo Welt""")
line1 = next(fobj)
RefactoringTool: Files that were modified:
RefactoringTool: py2sample.py
In [ ]: # %load py2sample.py
from __future__ import print_function
from future import standard_library
standard_library.install_aliases()
import io
11
fobj = io.StringIO("""Hello world
Hallo Welt""")
line1 = next(fobj)
print(’Done’)
37
Backup
py2sample.py.bak
38
•
•
•
•
Two-stage approach
first stage applies only save changes
no imports from future
produces modern, 2.6-compatible code
result may not run with Python 3 yet
In [ ]: $ futurize -w --stage1 py2sample.py
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored py2sample.py
--- py2sample.py
(original)
+++ py2sample.py
(refactored)
@@ -1,3 +1,4 @@
+from __future__ import print_function
import StringIO
@@ -6,6 +7,6 @@
fobj = StringIO.StringIO("""Hello world
Hallo Welt""")
-line1 = fobj.next()
+line1 = next(fobj)
-print ’Done’
+print(’Done’)
RefactoringTool: Files that were modified:
RefactoringTool: py2sample.py
In [ ]: # %load py2sample.py
from __future__ import print_function
import StringIO
fobj = StringIO.StringIO("""Hello world
Hallo Welt""")
12
line1 = next(fobj)
print(’Done’)
In [ ]: $ futurize -w --stage2 py2sample.py
RefactoringTool: Refactored py2sample.py
--- py2sample.py
(original)
+++ py2sample.py
(refactored)
@@ -1,10 +1,13 @@
from __future__ import print_function
+from future import standard_library
+standard_library.install_aliases()
+from builtins import next
-import StringIO
+import io
-fobj = StringIO.StringIO("""Hello world
+fobj = io.StringIO("""Hello world
Hallo Welt""")
line1 = next(fobj)
RefactoringTool: Files that were modified:
RefactoringTool: py2sample.py
In [ ]: # %load py2sample.py
from __future__ import print_function
from future import standard_library
standard_library.install_aliases()
from builtins import next
import io
fobj = io.StringIO("""Hello world
Hallo Welt""")
line1 = next(fobj)
print(’Done’)
39
Going fix-by-fix
• more than fifty different fixes
• futurize --list-fixes
In [ ]: $ futurize -f lib2to3.fixes.fix_next py2sample.py
RefactoringTool: Refactored py2sample.py
--- py2sample.py
(original)
+++ py2sample.py
(refactored)
@@ -4,6 +4,6 @@
13
fobj = StringIO.StringIO("""Hello world
Hallo Welt""")
-line1 = fobj.next()
+line1 = next(fobj)
print ’Done’
RefactoringTool: Files that need to be modified:
RefactoringTool: py2sample.py
In [ ]: futurize -f lib2to3.fixes.fix_has_key py2sample.py
RefactoringTool: No files need to be modified.
40
•
•
•
•
•
41
Strategies
need to have tests (coverage)
create environments for Python 2 and 3 (conda)
version control all steps
either go fix-by-fix (>50) or
go stage-by-stage (2 steps)
Starting from Python 3
In [116]: # %load py3sample.pya
zip(’abc’, range(3))
Out[116]: <zip at 0x1051d3f88>
pasteurize -w py3sample.py
In [ ]: # %load py3sample.py
from __future__ import unicode_literals
from __future__ import print_function
from __future__ import division
from __future__ import absolute_import
from builtins import zip
from builtins import range
from future import standard_library
standard_library.install_aliases()
zip(’abc’, range(3))
old version in py3sample.py.bak
In [ ]:
14