Python 2 and 3 compatibility testing via optional run-time type checking

Transcription

1 Python 2 and 3 compatibility testing via optional run-time type checking Raoul-Gabriel Urma Work carried out during a Google internship & PhD

2

3 Python 2 vs. Python 3

4 Valid in Python 2 def pivot_index(len): return len/2 list = [4, 1, 3, 10, 4] index = pivot_index(len(list)) print(list[index])

5 Error in Python 3 File "integer.py", line 6, in <module> print(list[index]) TypeError: list indices must be integers, not float

6 Valid in Python 2 def concat(l1, l2): return l1 + l2 d = {1:'a', 2:'b'} l = concat(d.keys(), d.values())

7 Error in Python 3 File "dict.py", line 5, in <module> l = concat(d.keys(), d.values()) File "dict.py", line 2, in concat return l1 + l2 TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values'

8 Valid in Python 2 def assertequals(s1, s2): assert(s1 == s2) def slurp(filename, mode): return open(filename, mode).read() data = slurp("data.txt","rb") # "hello" assertequals("hello", data)

9 Error in Python 3 File "open.py", line 8, in <module> assertequals("hello", data) File "open.py", line 2, in assertequals assert(s1 == s2) AssertionError

10 Python 2 to Python 3 complaints On unicode: this will be maybe an impossible task to do because of one feature: unicode strings. We currently extensively use unicode [...]. On using 2to3: the script doesn t work with python 3. I have tried 2to3 script [...] but it still doesn t work. On using 2to3: this is something that 2to3 does not pick up and most likely you will not either. At least in my situation it took me a long time to track down the problem because some other implicit conversion was happening at another place.

11

12 Our idea Before After File "open.py", line 8, in <module> TYPE_ERROR: Function: slurp, returns <class 'bytes'> assertequals("hello", data) File "open.py", line 2, in assertequals but EXPECTED <class 'str'> assert(s1 == s2) AssertionError def assertequals(s1: str, s2: str) -> bool def slurp(filename: str, mode: str) -> str

13 Pytypedecl

14 In a nutshell Type declaration language for Python Optional run-time type checking using type annotations Design constraints: Engineering: integrates naturally into existing development infrastructure (no external tools) Usability: syntax familiar to Python programmers

15 Normal Python code class Logger: def log(messages, buffer):... def setstatus(status):... def find(elements, key):...

16 What we are adding exceptions qualified parametric types class Logger: def log(messages: list<str>, buffer: Readable and Writeable) -> None raises IOException overloading def log(messages: list<str>) -> None def setstatus(status: int or str) interface Readable: union intersection def read interfaces interface Writable: def write parametric polymorphism and type bounds def <T <: UserId> find(elements: list<t>, key<t>) -> T? none-able type

17 Type checking

18 Basic blueprint qualified parametric types container integrity: assume iterator provided to type check elements structure integrity: optional iterator to type check integrity parametric types unification system with inequations resolved online extend iscompatible() with a type parameter table overloading at least one signature valid

19 Implementation

20 Prototype Type declaration language types defined in an external file parser implemented with Python Lex-Yacc Type checking Implemented as a Python library checker.py Project is open source

21 How to use # er.py # er.pytd import sys class er: def MakeAnnouncement(cls, s:list[str]) -> None from pytypedecl import checker def Send (cls, addr:str) -> None class def MakeAnnouncement(cls, s): for addr in s: cls.send (addr) add two lines to opt for type def Send (cls, addr): checker.checkfromfile(sys.modules[ name ], file + "td")

22 Currently in the pipeline Type system formalisation Interaction between parametric polymorphism, overloading and generators Evaluation Case study: porting a Python 2 project to Python 3 using type annotations Performance impact of implementation

23 Questions? Raoul-Gabriel Java 8 in Action: Lambdas, Streams and functional-style programming