Python 2 and 3 compatibility testing via optional run-time type checking Raoul-Gabriel Urma Work carried out during a Google internship & PhD https://github.com/google/pytypedecl
Python 2 vs. Python 3
Valid in Python 2 def pivot_index(len): return len/2 list = [4, 1, 3, 10, 4] index = pivot_index(len(list)) print(list[index])
Error in Python 3 File "integer.py", line 6, in <module> print(list[index]) TypeError: list indices must be integers, not float
Valid in Python 2 def concat(l1, l2): return l1 + l2 d = {1:'a', 2:'b'} l = concat(d.keys(), d.values())
Error in Python 3 File "dict.py", line 5, in <module> l = concat(d.keys(), d.values()) File "dict.py", line 2, in concat return l1 + l2 TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values'
Valid in Python 2 def assertequals(s1, s2): assert(s1 == s2) def slurp(filename, mode): return open(filename, mode).read() data = slurp("data.txt","rb") # "hello" assertequals("hello", data)
Error in Python 3 File "open.py", line 8, in <module> assertequals("hello", data) File "open.py", line 2, in assertequals assert(s1 == s2) AssertionError
Python 2 to Python 3 complaints On unicode: this will be maybe an impossible task to do because of one feature: unicode strings. We currently extensively use unicode [...]. On using 2to3: the script doesn t work with python 3. I have tried 2to3 script [...] but it still doesn t work. On using 2to3: this is something that 2to3 does not pick up and most likely you will not either. At least in my situation it took me a long time to track down the problem because some other implicit conversion was happening at another place.
Our idea Before After File "open.py", line 8, in <module> TYPE_ERROR: Function: slurp, returns <class 'bytes'> assertequals("hello", data) File "open.py", line 2, in assertequals but EXPECTED <class 'str'> assert(s1 == s2) AssertionError def assertequals(s1: str, s2: str) -> bool def slurp(filename: str, mode: str) -> str
Pytypedecl
In a nutshell Type declaration language for Python Optional run-time type checking using type annotations Design constraints: Engineering: integrates naturally into existing development infrastructure (no external tools) Usability: syntax familiar to Python programmers
Normal Python code class Logger: def log(messages, buffer):... def setstatus(status):... def find(elements, key):...
What we are adding exceptions qualified parametric types class Logger: def log(messages: list<str>, buffer: Readable and Writeable) -> None raises IOException overloading def log(messages: list<str>) -> None def setstatus(status: int or str) interface Readable: union intersection def read interfaces interface Writable: def write parametric polymorphism and type bounds def <T <: UserId> find(elements: list<t>, key<t>) -> T? none-able type
Type checking
Basic blueprint qualified parametric types container integrity: assume iterator provided to type check elements structure integrity: optional iterator to type check integrity parametric types unification system with inequations resolved online extend iscompatible() with a type parameter table overloading at least one signature valid
Implementation
Prototype Type declaration language types defined in an external file parser implemented with Python Lex-Yacc Type checking Implemented as a Python library checker.py Project is open source http://www.github.com/google/pytypedecl
How to use # emailer.py # emailer.pytd import sys class Emailer: def MakeAnnouncement(cls, emails:list[str]) -> None from pytypedecl import checker def SendEmail(cls, addr:str) -> None class Emailer(object): @classmethod def MakeAnnouncement(cls, emails): for addr in emails: cls.sendemail(addr) add two lines to opt for type checking @classmethod def SendEmail(cls, addr): checker.checkfromfile(sys.modules[ name ], file + "td")
Currently in the pipeline Type system formalisation Interaction between parametric polymorphism, overloading and generators Evaluation Case study: porting a Python 2 project to Python 3 using type annotations Performance impact of implementation
Questions? Raoul-Gabriel Urma raoul@urma.com @raouluk Java 8 in Action: Lambdas, Streams and functional-style programming