Testing Database Programs using Relational Symbolic Execution

Transcription

1 Testng Database Programs usng Relatonal Symbolc Executon Mchaël Marcozz 1, Wm Vanhoof, Jean-Luc Hanaut Faculty of Computer Scence Unversty of Namur Rue Grandgagnage, Namur, Belgum Abstract Symbolc executon s a technque whch allows to automatcally generate test nputs (and outputs) exercsng a set of executon paths wthn a program to be tested. If the paths cover a suffcent part of the code under test, the test data offer a representatve vew of the program s actual behavour, allowng to detect falures and correct faults. Relatonal databases are ubqutous n software, but symbolc executon of the programs that manpulate them remans a non-trval problem, because of the complex structure of such databases and the complex behavour of the SQL statements. In ths work, we defne a symbolc executon for SQL code and ntegrate t wth a more tradtonal symbolc executon of normal program code. The database tables are represented by relatonal symbols and the SQL statements by relatonal constrants over these symbols and the symbols representng the normal varables of the program. An algorthm based on these prncples s presented for the symbolc executon of smple Java methods, readng and wrtng wth transactonal SQL n a relatonal database, the latter subject to data ntegrty constrants. The algorthm s ntegrated n a test data generaton tool and expermentally evaluated over thousands of program paths ncludng real applcaton code. The target language for the constrants produced by the tool s the SMTv2 standard and the used solver s Mcrosoft Z3. The results show that the proposed approach allow to generate meanngful test data for database programs, ncludng vald database content, n reasonable tme. Keywords: Software Testng, Symbolc Executon, Constrants, Satsfablty Modulo Theores (SMT), Quantfers, Frst-Order Logc, Relatonal Algebra, Structured Query Language (SQL), Database Applcatons Emal addresses: [email protected] (Mchaël Marcozz), [email protected] (Wm Vanhoof), [email protected] (Jean-Luc Hanaut) 1 F.R.S.-FNRS Research Fellow

2 1. Introducton In current software development practce, testng [1, 2] remans the prmary approach to mprove the relablty of software. Ths motvates [3] much research on effcent technques to automate all aspects of the software testng process. Of partcular nterest s the automaton of test data generaton [4] for functonal testng of unts of code, where the dea s to automatcally generate a representatve set of nputs (and outputs) for a program fragment under test (typcally a functon or method). These data can subsequently be compared wth the functon s expected behavour n order to detect falures and correct faults. Moreover, once a sutable set of test data has been generated (and verfed), t can be used as reference data for contnued (regresson) testng of the code. Whle dfferent approaches exst towards the automatc generaton of test data, symbolc executon [5] has been recognsed as a promsng technque for so-called whte-box or glass-box testng [6, 7, 8, 9, 10], where the dea s to generate nput data that n some way cover a suffcently large part of the control-flow graph of the functon under test [4]. The symbolc executon process traverses the control-flow graph of a program or functon by executng the code over symbolc values nstead of concrete values [5]. Each tme a control dependency s encountered, the symbolc executon process proceeds along one of the possble paths, thereby generatng constrans upon the symbolc values such that when the program varables have values that satsfy these constrants, the real executon would proceed along the selected path. The process termnates when a suffcently large (and dverse) number of paths through the control-flow graph have been explored accordng to some coverage crteron [4]. For each path, the constrants that have been collected along are regrouped n a so-called path-constrant whch s subsequently solved, resultng n a set of concrete test values for the program varables that make the real executon proceed along the gven path. Test data generaton based on symbolc executon s now at the core of varous popular open-source and commercal testng tools [9]. Although test data generaton technques are maturng, barely few works [11, 12, 13] have studed how to automate the generaton of representatve test data for programs that nteract wth a database,.e., programs that ntensvely read and wrte nto a large, persstent, and hghly-structured relatonal database [14] usng SQL statements. In ths work, we detal, formalse and evaluate an approach that defnes a symbolc executon for SQL code and ntegrates t wth a more tradtonal symbolc executon of typcal program code. The data generated by our technque consttutes test data that ncludes database content n addton to values for the program varables. Consequently, t can serve to test the program at hand, ncludng the nteracton between the code and the relatonal database. Enablng test data generaton for database ntensve programs s a non-trval extenson of known data generaton technques. In a database program, the database can be seen as a partcular knd of contaner for some of the values manpulated by the program. Durng symbolc executon, these values must thus be represented symbolcally and subsequently constraned to allow the proper generaton of test data, ncludng an nput and output database content. Nevertheless, the symbolc representaton of these values rases dffcultes as the database s a contaner of partcularly complex shape: ts content must obey the so-called database schema, defned usng SQL DDL code [14]. Ths database schema defnes a set of tables where the database content wll be stored. Each table can be seen as representng a mathematcal relaton,.e., a set of tuples wth no lmt on the 2

3 potental number of tuples. The schema typcally also descrbes a set of data ntegrty constrants that must be enforced by the relatons n the tables, lke the prmary key, foregn key or check constrants. These constrants are partcularly complex as they are frst-order logc constrants. For nstance, a prmary key constrant states that for all couples of tuples n the relaton represented by a table, the value of the prmary key felds cannot be all equal. A program typcally nteracts wth the database by usng SQL [14] query statements and DML statements that are embedded n the program s source code. As such, SELECT queres allow to gather values from the database tables n order to copy them nto the program varables. DML statements INSERT, UPDATE and DELETE allow to modfy the content of the database tables, typcally n functon of the value of the program varables. Symbolc executon of such SQL statements rases dffcultes due to ther complex behavor. Frstly, SQL s a declaratve language: SQL statements express the desred acton over the content of the database relatons, but they do not explct the (often complex) control-flow necessary to compute ths acton. In practce, durng executon, these SQL statements are sent by the database program usng a dedcated API to the DataBase Management System (DBMS) [14], an external component responsble for the nterpretaton and executon of SQL code over the database. The DBMS keeps an optmsed and persstent nternal representaton of the database and manages concurrent dstant accesses by allowng the database programs to use transactons [14]. Secondly, the executon of DML INSERT, UPDATE or DELETE statements by the DBMS does not only consst n modfyng the content of the database, but also n checkng that these modfcatons do not let the database n a state where the ntegrty constrants defned n the schema are volated. If an ntegrty constrant volaton s detected, the DML statement executon fals and the database remans unmodfed. As a consequence, durng symbolc executon each INSERT, UPDATE or DELETE statement wll have to be treated as an f-then-else statement wth a partcularly complex condton: f (Program varables and Database are n a state where the SQL statement wll not volate any constrant ) { Execute the SQL statement! } else { Sgnal a constrant volaton! } The technque that we detal n ths work overcomes these dffcultes by modellng every table n the database as a varable typed as a mathematcal relaton over smple domans, lke the ntegers. Each SQL statement n the program can then be modelled as a relatonal operaton over these relatonal varables as well as the tradtonal program varables. By defnng a symbolc executon over ths relatonal verson of the program, we can derve a set of path constrants over the values of both the program s relatonal and tradtonal varables. The generated path constrants wll thus nclude symbols representng smple program values as well as symbols representng the relatons n the database tables. Furthermore, each path constrant must be combned wth the schema constrants n order enforce data ntegrty of the database content. The result s a complex constrant system that mxes tradtonal constrants on the program s varables wth relatonal constrants over the relatonal varables. Each soluton to the combned constrant system descrbes 3

4 test data, ncludng a vald ntal and fnal state for each table n the database, such that when the program s executed wth respect to these data values (ncludng the database), the executon wll follow the path represented by the constrant system. In our work, we use the SMTv2 [15] language logc to express these combned constrant systems, n a smlar way to the scheme proposed n [16] for relatonal constrants modellng. The constrants are solved usng the Z3 [17] solver, whch s at the core of several exstng symbolc executon tools (e.g. [18, 19]). The man contrbuton of ths work s a relatonal symbolc executon algorthm for smple database programs. Ths algorthm ntegrates the technque descrbed n the prevous paragraphs for symbolc executon of SQL code wth a classcal symbolc executon process for basc Java statements. Ths algorthm can be used for symbolc executon of a language composed of smple Java methods nteractng wth a relatonal database usng SQL statements and transactons through JDBC. Gven the SQL DDL code descrbng the database schema, the Java/SQL code of the method and a fnte path n the control flow graph of ths method, the algorthm generates the correspondng constrants n the SMTv2 language. A test generaton tool based on ths algorthm has been coded and used to generate test data for a number of sample Java methods and databases. Ths work extends our prevous work presented n [20, 21]. It notably adds the support of SMTv2 as output constrant language. Ths allows the technque to beneft from the power of SMT solvers for relatonal constrant solvng [16]. It also provdes an extended expermental evaluaton of the technque over thousands of paths n varous peces of code, ncludng real applcaton code. The remander of ths paper s organsed as follows. Our relatonal symbolc executon algorthm s descrbed n secton 2. Frst, we formally defne the part of the Java/SQL syntax that s supported by the algorthm. Then, we systematcally descrbe the constrant generaton rules to be used for the symbolc executon of ths sub-language. A test generaton tool based on the algorthm s descrbed and evaluated expermentally over a set of sample programs n secton 3. Fnally, some conclusons, related and future work are dscussed n secton 4. 4

5 2. A relatonal symbolc executon algorthm for smple Java/SQL programs 2.1. Syntax of the tested Java/SQL programs In ths secton, we defne precsely usng a BNF grammar what subset of the Java/SQL syntax our algorthm can execute symbolcally. Ths subset has been chosen to offer a good compromse between smplcty and expressveness. Wthout loss of generalty, we consder a sngle java method that nteracts wth a relatonal database. The database schema descrbes a set of tables. Each table has a prmary key and the attrbutes can be constraned by foregn key constrants and check constrants. The code of the method can contan f-then-else blocks, whle loops, return statements and local varable assgnments wth typcal operators for lsts and ntegers. The method wll nteract wth the database through SQL base statements, as they are typcally used n onlne transacton processng (OLTP) programs, and through SQL base prmtves for transacton management. The method receves as nput parameters a JDBC connecton to the database, a set of nteger lsts and an nput scanner for ntegers. The lsts model any structured group of nputs transmtted to the code at method call. The scanner models the method s access to smple data from the outsde world, lke user prompt, network access, etc. Fgure 1 provdes an example database program n ths partcular sublanguage. The example descrbes a database wth two tables: one for lbrary shelves and one for the books stored n each of these shelves. The total number of books stored n a shelf s saved for each shelf. The example also descrbes a method manpulatng ths database: t adds a set of new books to the database and updates the shelves books counts. If a book s added to a non-exstent shelf, then the shelf s tself added to the database as well. The books are nserted one by one n solated transactons. If a transacton was successful, the code of the added book s saved n a lst, whch s returned at the end of the method s executon. In the next paragraphs, the chosen notaton for the BNF grammar of the syntax s standard but ncludes some addtonal meta-symbols: {...} (groupng),? (zero or one tmes), * (zero or more tmes) and + (one or more tmes). When a sngle nontermnal appears several tmes n a sngle producton, subscrpt notaton allows to dstngush between the occurrences Database program A database program s composed of the SQL DDL code of the database schema and of the code of the Java method under test. database-program ::= sql-ddl java-method Database schema The relatonal database schema s a lst of table defntons. Ths lst can be empty, n what case the program s a tradtonal program that works ndependently of any database. In the lst, each table s dentfed by ts name, contans at least one attrbute and endorses exactly one prmary key. Foregn keys and addtonal check constrants can be declared for a table. A row n a table cannot be deleted or see ts prmary key value modfed as long as there exsts at least another row n the database that references t (ON DELETE/UPDATE NO ACTION). Semantcs of all the schema creaton prmtves conforms to the classcal SQL DDL specfcaton provded by ISO. 5

6 Fgure 1: SQL DDL and Java/SQL DML code of a database program addng books and updatng shelves books count n a lbrary database. CREATE TABLE shelf ( d INTEGER NOT NULL, numberofbooks INTEGER NOT NULL, CONSTRAINT spk PRIMARY KEY (d), CHECK(numberOfBooks > 0)); CREATE TABLE book ( code INTEGER NOT NULL, shelfid INTEGER NOT NULL, CONSTRAINT bpk PRIMARY KEY (code), CONSTRAINT bfk FOREIGN KEY(shelfId) REFERENCES shelf (d)); 1 Lst<Integer> addbooks (Connecton con, Scanner n, Lst<Integer> newbooks) throws SQLExcepton { 2 nt = 0; Lst<Integer> addedbooks = new ArrayLst<Integer>(); 3 whle (!(newbooks == null) & ( < newbooks.sze()) ) { 4 nt error = 0; nt theshelf = n.nextint (); 5 ResultSet shelves = con.createstatement().executequery( SELECT d FROM shelf WHERE d= +theshelf); 6 f (! shelves.next() ) 7 con.createstatement().execute( INSERT INTO shelf VALUES ( +theshelf+,1) ); 8 else 9 con.createstatement().execute( UPDATE shelf SET numberofbooks=numberofbooks+1 WHERE d = +theshelf); 10 try { 11 con.createstatement().execute( INSERT INTO book VALUES ( +newbooks.get()+, +theshelf+ ) ); 12 } catch (SQLExcepton e) { 13 error = 1; 14 }; 15 f (error==0) { 16 con.commt(); 17 addedbooks.add(newbooks.get()); 18 } else 19 con. rollback (); 20 = + 1; 21 }; 22 return addedbooks; } 6

7 sql-ddl ::= table * table ::= CREATE TABLE d ( att + p-key f-key * chk *); att ::= d INTEGER NOT NULL, p-key ::= CONSTRAINT d cst PRIMARY KEY ( d att ) f-key ::=,CONSTRAINT d cst FOREIGN KEY ( d att ) REFERENCES d tab ( d re f d ) chk ::=,CHECK ( d {< = >} nteger ) d ::= {a... z A... Z}{a... z A... Z }* nteger ::= -? {1... 9}{0... 9} 0} Method sgnature and body We consder smple Java methods manpulatng only nternal varables and parameters. Varables can only be typed as nt, java.utl.lst<java.lang.integer> or java.sql.resultset. The method receves as nput parameters a connexon to the database (typed as java.sql.connecton ), a scanner (typed as java.utl.scanner ) and some lsts of ntegers (typed as java.utl.lst<java.lang.integer> ), where two dstnct lst parameters cannot reference a sngle lst object. Its return type can be ether vod, nt or java.utl.lst<java.lang.integer>. java-method ::= type d ( db-con, np parameters ) throws SQLExcepton { stmt * } type ::= vod nt Lst<Integer> db-con ::= Connecton con np ::= Scanner n parameters ::= {, Lst<Integer> d }* The connecton wth the database s supposed to stay relable and every SQL statement to be processed wthout any techncal problem durng the method s executon. The semantcs of all the Java constructs conforms to the classcal Java specfcaton and documentaton. The semantcs of all SQL statements conforms to the classcal SQL specfcaton provded by ISO. Common statements and lsts management. Common condton, loop and assgnment statements, as well as common nteger expressons and boolean condtons can be used. Lsts can be manpulated usng the add(nt), remove(nt), get(nt) and sze(nt) methods. The java.utl.arraylst<integer> mplementaton of these methods s supposed to be used. A lst varable can be null. stmt ::= f ( cond ) { stmt then *} {else { stmt else *}}? ; whle ( cond ) { stmt * }; {nt Lst<Integer>}? d = expr ; d.add( nt-expr ); d.remove( nt-expr ); return d ; 7

8 cond ::= true false (! cond ) ( cond 1 {& } cond 2 ) ( nt-expr 1 {< == >} nt-expr 2 ) ( d == null) expr ::= nt-expr lst-expr nt-expr ::= d nteger ( nt-expr 1 {+ -} nt-expr 2 ) ( d.get( nt-expr )) ( d.sze()) lst-expr ::= d null new ArrayLst<Integer>() Interactng wth the outsde world. The scanner parameter of the method can be used to get nteger data from the outsde world (user prompt, network access, readng from a fle, etc.). Ths nteracton s supposed to always succeed, wthout any techncal problem. stmt ::= {nt}? d = n.nextint(); Readng data from the database. Data can be read from the database usng smple SQL queres. The obtaned ResultSet can be accessed usng the next() and getint(strng) methods. stmt ::= { ResultSet }? d = con.createstatement().executequery(" select-query "); d.next(); select-query ::= SELECT { d,}* d n FROM d tab { WHERE db-cond }? db-cond ::= ( db-cond 1 {AND OR} db-cond 2 ) (NOT db-cond ) ( d {< = >} db-nt-expr ) db-nt-expr ::= d nteger ( db-nt-expr 1 {+ -} db-nt-expr 2 ) "+( nt-expr )+" nt-expr ::= d tab.getint(" d att ") cond ::= ( d.next( nt-expr )) Wrtng data nto the database. Data can be wrtten nto the database usng smple SQL INSERT, UPDATE or DELETE statements. If the executon of a such a statement provokes a volaton of one of the database schema ntegrty constrants, the database remans unmodfed by the statement and an excepton s thrown wthn the program and the method s executon s stopped. Such exceptons should be caught usng a try/catch structure. stmt ::= con.createstatement().execute(" db-wrte "); try { con.createstatement().execute(" db-wrte "); } catch (SQLExcepton e) { stmt * }; 8

9 db-wrte ::= INSERT INTO d VALUES ( { nt-expr, }* nt-expr n ) UPDATE d tab SET d att = db-nt-expr { WHERE db-cond }? DELETE FROM d { WHERE db-cond }? Transactons management. SQL transactons are managed through the classcal commt and rollback statements. We suppose that a new transacton s automatcally started at the begnnng of the method s executon. The frst call to commt or rollback wll end ths transacton and then starts a new one. Any subsequent call to commt or rollback wll end the current transacton and start a new one. When a commt statement s executed, t makes permanent all the changes made to the database by the program snce the current transacton was started. When a rollback statement s executed, t restores the database to ts state at the start of the current transacton. We suppose that all the changes made to the database snce the last transacton was started are automatcally commtted at the end of the method s executon. stmt ::= con.commt() ; con.rollback() ; 2.2. Relatonal symbolc executon algorthm Inputs and outputs The proposed algorthm, whch s descrbed n ths secton, receves as nputs the SQL DLL code descrbng the schema of the database, the Java code of the method under test and a sngle executon path through ths method. It produces as output a constrant system mxng classcal constrants wth relatonal constrants. Solutons to ths system are such that when the method s executed wth respect to any of these solutons, ts executon wll follow the gven path. Couplng ths algorthm wth any exstng technque (e.g. [23, 24, 25, 26]) able to select a set of paths to test n the program s control flow graph wll allow to generate test data for these paths. The set of paths for whch test data are computed, as well as the process used to select these paths, are thus parameters of the method that we propose. Ths allows the method to be used wthn the context of dfferent code coverage crtera [4]. The subset of Java/SQL supported by our algorthm allows ntegers as only prmary type n Java code and database tables. Ths choce was adopted to make the modellng and use of the constrant generaton rules conceptually smpler. However, ths does not fundamentally lmt the power of the proposed technque snce all other usual prmtve types such as booleans, strngs, and floatng pont numbers, but also data structures such as sets, arrays and matrces, and Java objects can be mapped to ntegers, smulated usng lsts of ntegers, or drectly modelled nto SMTv2. The executon path receved as nput by our algorthm s supposed to be a fnte path n the method s control flow graph [4]. It defnes whch branches were taken at each of the encountered f statements, for each encountered loop how many tmes ts body was executed (ths number must be fnte), and for each encountered try/catch statement whether the catch clause was executed. A path termnates ether when the end of the method s reached or when a return statement s executed. Our algorthm translates the path nto a constrant system, combnng the path constrant wth the database schema ntegrty constrants, expressed n SMTv2 [15], a wdely adopted 9

10 language used as the standard language for many SMT solvers. The constrant system produced for a path can notably be solved usng the Z3 solver [17]. Both Z3 and SMTv2 have been shown to be adequate [16] for relatonal constrant solvng and are already at the core of several symbolc executon tools lke [18, 19]. Solvng the constrant system generated by the algorthm for a gven path allows to fnd values for both the nputs and outputs of the analysed Java method. The nputs nclude the content of each database table at the start of the method s executon, the value of every lst receved as argument by the method, and a value for the part of the nput stream that s consumed durng the method s executon. The outputs nclude the content of each database table at the end of the method s executon, the fnal value of each of the argument lsts of the method, and possbly the value returned by the return statement. If the constrant system produced for a gven path has no soluton, ths means that the path s nfeasble. As the produced constrants are wrtten n a logc that s not decdable n general [16], t can happen that for a gven path the solver may nether be able to fnd a soluton for the generated constrant system, nor be able to establsh that such a soluton does not exst Algorthm prncple The algorthm performs a symbolc executon of the program path receved as nput. Each of the successve values taken by the method s varables and by the database tables durng the executon of the path s represented by correspondng symbols and defned by constrants. Frst, symbolc executon generates constrants over the symbols representng the ntal values of the database tables. These constrants state that, ntally, each table contans data that conform to the database schema ntegrty constrants. Then, symbolc executon analyses one by one the method s statements n the order specfed by the path. Each tme a statement sets or changes the value of a method varable or database table, symbolc executon generates constrants over the symbols representng the new value. These constrants defne how the new value can be computed from the values of the database tables and program varables before the statement s executon. Moreover, every tme the value of a database table s changed, constrants are also added to state that the new value enforce the database schema ntegrty constrants. Fnally, every tme an f, whle or try/catch statement s encountered, symbolc executon generates an addtonal constrant over the symbols such that when the program s executed wth respect to values satsfyng ths constrant, the executon s guaranteed to take the consdered path Constrant generaton rules In the followng paragraphs, we llustrate the executon of the algorthm over the example database program gven n Fgure 1. We detal each step of the symbolc executon process over a path where the whle loop s executed once, the else branch of both the f statements s taken, and the catch clause of the try/catch s executed (lnes 1-6, 8-15, 18-21, 3 and 22). At each step, we present the rules used by our algorthm to generate the correspondng SMTv2 symbols and constrants. All the rules that are part of the complete set of rules defnng our symbolc executon algorthm for the fracton of Java/SQL defned n Secton 2.1 are ether presented durng 10

11 one of these steps, or descrbed formally n a set of tables avalable at the end of ths secton. The frst step executed by our algorthm s to generate SMTv2 symbols and constrants for the SQL DDL code of the database schema. For the database schema descrbed n Fgure 1, the generated SMTv2 code s detaled n the frame below, where new symbols and new symbol types are defned usng the SMTv2 keyword declare, whle new constrants are enforced usng the SMTv2 keyword assert. Frst of all, the algorthm generates new symbol types for the knd of objects stored n each table defned by the schema (the lnes prefxed by (0) n the SMTv2 code below). It wll then generate symbols and constrants descrbng the nput content of each of these tables. The used modellng s nspred by the one proposed n [16] for relatonal types. Frst, a symbol s created (1) to represent the ntal set of objects n each table. Typed as a boolean functon, t returns true for each object present n the nput content of the table. Symbols typed as nteger functons are then generated (2) to assocate to each object n the table one of ts attrbute values. Fnally, constrants are generated to enforce on ths nput content all the check constrants (3), prmary key constrants (4), and foregn key constrants (5) defned n the schema. Note that the orgnal SQL table and attrbute names, as well as the orgnal Java varable names, are used as SMTv2 symbols, suffxed by the natural number 1 (e.g. book1 or d1 n the SMTv2 code below), whch represents the fact that the current symbols represent the ntal values of the represented tables, attrbutes or varables. Subsequent values of a same table, attrbute or varable wll be represented by the same symbol suffxed wth successve numbers. ; New types for tables (0) (declare sort book) (0) (declare sort shelf) ; Input content of table Book (1) (declare fun book1 (book) Bool) (2) (declare fun shelfid1 (book) Int) (2) (declare fun code1 (book) Int) (4) (assert (forall ((a book) (b book)) (=> (and (and (book1 a) (book1 b)) (= (code1 a) (code1 b))) (= a b)))) ; Input content of table Shelf (1) (declare fun shelf1 (shelf) Bool) (2) (declare fun numberofbooks1 (shelf) Int) (2) (declare fun d1 (shelf) Int) (3) (assert (forall ((a shelf )) (> (numberofbooks1 a) 0))) (4) (assert (forall ((a shelf ) (b shelf )) (=> (and (and (shelf1 a) (shelf1 b)) (= (d1 a) (d1 b))) (= a b)))) ; Foregn keys (5) (assert (forall ((a book)) (=> (book1 a) (exsts ((b shelf)) (and (shelf1 b) (= (shelfid1 a) (d1 b ))))))) The second step executed by our algorthm s to defne a new SMTv2 symbol type (called BoundedLst) for lsts of ntegers. All the symbols that wll be subsequently generated to represent the value of a varable typed as a Java lst wll be part of ths new 11

12 SMTv2 type. A BoundedLst symbol represents a record composed of three felds: the snull feld s typed as boolean, the sze feld s typed as nteger and the elements feld s typed as array of ntegers. If the snull feld s true, then the symbol represents the Java null value. Otherwse, the feld sze represents the sze of the lst, and the feld elements represents an array whose ndexes 0 to (sze 1) contan the elements of the lst n the rght order. (declare datatypes () ((BoundedLst (mk bounded lst (snull Bool) (sze Int) (elements (Array Int Int)))))) The thrd step executed by our algorthm s to defne symbols (typed as BoundedLst) for the ntal content of each lst parameter of the method. For the example method consdered n ths secton, the followng code s generated: (declare const newbooks1 BoundedLst) (assert (=> (not (snull newbooks1)) (>= (sze newbooks1) 0))) The algorthm can then proceed wth the symbolc executon of the method. It follows the path receved as nput and consders all statements one by one. In the case of our example, the two frst statements to be executed are assgnments. Symbolc executon for assgnment creates a new symbol of the correct type to represent the new value of the assgned varable (1) and generates constrants to specfy that ths new symbol contans the value computed by evaluatng the expresson on the rght of the = symbol (2). In the partcular case where a lst varable s assgned to a dfferent lst varable, the shared content of the two varables s represented by a sngle symbol. (1) (declare const 1 Int) (2) (assert (= 1 0)) (1) (declare const addedbooks1 BoundedLst) (2) (assert (not (snull addedbooks1))) (2) (assert (= (sze addedbooks1) 0)) The next statement n the path s a whle statement. As the path specfes that the loop body must be executed, a constrant s generated to specfy that the loop condton at ths pont of tme should be true (1). As the sze() method cannot be called on a null object wthout causng a runtme error, a constrant s automatcally added to ensure that the current value of the newbooks varable of the method s not null (2). (1) (assert (and (not (snull newbooks1)) (< 1 (sze newbooks1)))) (2) (assert (not (snull newbooks1))) Then the algorthm proceeds wth symbolc executon of the statements n the loop body, as specfed wthn the nput path. The frst statement s an assgnment statement: (declare const error1 Int) (assert (= error1 0)) Symbolc executon for use of the nput scanner smply creates a new symbol to represent the scanned value: 12

13 (declare const theshelf1 Int) Symbolc executon for select statements creates new symbols to represent the content of the ResultSet varable. A frst symbol (1) descrbes the number of rows returned by the select query. These rows are avalable through a second symbol (2) whch s a functon that returns them n the order n whch they are returned by the ResultSet: (shelves1lst 0) wll be the frst returned row, (shelves1lst 1) the second one and so on. (1) (declare const shelves1sze Int) (2) (declare fun shelves1lst (Int) shelf) The modellng proposed n [16] for constranng the content and cardnalty of relatons s then used to specfy that a row s part of the ResultSet f and only f t s part of the current content of the table on whch the select query s executed and that t enforces the WHERE condton of the select query. In practce, new constrants are added (1) to defne a functon shelves1invertedlst whch s the nverse of shelves1lst. Ths functon s used (1) to ensure that shelves1lst defnes a one to one correspondence between the ntegers 0 shelves1s ze and the elements n the ResultSet. Helper code (2) s also added to ensure a proper nstantaton of the unversal quantfers by the solver. (1) (declare fun shelves1invertedlst (shelf) Int) (2) (declare fun shelves1trgger (Int) Bool) (1) (assert (and (>= shelves1sze 0) (=> (= shelves1sze 0) (forall ((c shelf )) (not (and (shelf1 c) (= (d1 c) theshelf1 ))))))) (1) (assert (forall ((c shelf )) (=> (and (shelf1 c) (= (d1 c) theshelf1 )) (and (>= (shelves1invertedlst c) 0) (< (shelves1invertedlst c) shelves1sze ))))) (1) (assert (forall ((c shelf )) (=> (and (shelf1 c) (= (d1 c) theshelf1 )) (= c (shelves1lst ( shelves1invertedlst c )))))) (1) (assert (forall (( Int)) (=> (and (>= 0) (< shelves1sze)) (= ( shelves1invertedlst ( shelves1lst )))))) (1) (assert (forall (( Int)) (! (=> (and (>= 0) (< shelves1sze)) (and (shelf1 ( shelves1lst )) (= (d1 ( shelves1lst )) theshelf1 ))) (2) :pattern (shelves1trgger )))) (2) (assert (=> (>= 0 shelves1sze) (shelves1trgger 1))) (2) (assert (forall (( Int)) (! (=> (and (>= 0) (< shelves1sze)) (shelves1trgger (+ 1))) :pattern (shelves1trgger )))) As the path specfes that the else branch of the f statement must be executed ths tme, a constrant s generated to specfy that the condton of the f should be false,.e. that shelves.next() should return true. For each ResultSet object, the algorthm records the number of tmes the next() method has been called on ths object. Ths value represents the ndex ncreased by one of the row ponted by the cursor of the ResultSet at the current executon state of the 13

14 path. When the boolean value returned by the next() method s used n an f or whle condton, ths value states f the number of rows n the ResultSet s greater or equal to the number of tmes the next() method has been called so far on ths ResultSet. In ths case, shelves.next() wll return true f the ResultSet shelves contans more than one row (as shelves.next() has been called once on the ResultSet): (assert (not (not (>= shelves1sze 1)))) Symbolc executon for update creates a new symbol (1) typed as an nteger functon, that wll replace the prevous symbol assocatng the attrbute value to each object n the table. As ths new symbol s the second one to represent the value of the attrbute numbero f Books, t s named numbero f Books2. A couple of constrants (2)(3) s then generated to relate the old and new attrbute values n the table: one for the rows that do not match the WHERE condton (2), and one for those that do (3). Fnally, constrants are added to specfy that no ntegrty constrant was volated durng the update. In ths case, a constrant (4) s added to state that the updated attrbute values stll enforce the check constrant defned n the database schema. (1) (declare fun numberofbooks2 (shelf) Int) (2) (assert (forall ((p shelf )) (=> (or (and (shelf1 p) (not (= (d1 p) theshelf1))) (not (shelf1 p))) (= (numberofbooks2 p) (numberofbooks1 p))))) (3) (assert (forall ((p shelf )) (=> (and (shelf1 p) (= (d1 p) theshelf1)) (= (numberofbooks2 p) (+ (numberofbooks1 p) 1))))) (4) (assert (forall ((a shelf )) (> (numberofbooks2 a) 0))) Subsequently, as n our example the path specfes that the catch block of the try/catch statement must be executed, a constrant (1) s added to ensure that the program varables and the database are n a state where the INSERT executon wll volate a schema ntegrty constrant. In ths case, the constrant states that the nserted row has a smlar prmary key as the prmary key of an exstng row n the table or that the nserted row has a foregn key value that does not reference any exstng row n the shelf table. Constrants are also automatcally added to ensure that the sze() (2) and get(nt) (3) methods do not cause any runtme error. (1) (assert (or (exsts ((a book)) (and (book1 a) (= (code1 a) (select (elements newbooks1) 1)))) (forall ((a shelf )) (=> (shelf1 a) (not (= (d1 a) theshelf1 )))))) (2) (assert (not (snull newbooks1))) (3) (assert (>= 1 0)) (3) (assert (< 1 ( sze newbooks1))) The content of the catch block s then symbolcally executed: (declare const error2 Int) (assert (= error2 1)) As the path specfes that the else branch of the f statement must be executed ths tme, a constrant s generated to specfy that the condton of the f should be false: 14

15 (assert (not (= error2 0))) Symbolc executon for Rollback statements tells the algorthm to represent the current content of each database table usng the symbols that were representng the content of the table just before the last start of a new transacton (saved by the algorthm at the begnnng of the method executon and after each call to commt or abort). In ths case, the database state s restored to ts state at the method start,.e. the algorthm rewnds the counters for the database symbols and symbols book1, code1, shel f Id1, shel f 1, d1 and numbero f Books1 represent the content of the database after the con.rollback() statement. The assgnment statement s then symbolcally executed: (declare const 2 Int) (assert (= 2 (+ 1 1))) As the path specfes that the loop body must not be executed any more, a constrant s generated to specfy that, at ths pont n tme, the loop condton should be false: (assert (not (and (not (snull newbooks1)) (< 2 (sze newbooks1))))) (assert (not (snull newbooks1))) As a return statement s met, the algorthm stops and returns the generated SMTv2 constrant model. The Z3 solver [17] can be asked to fnd a valuaton for the defned symbols satsfyng the constrants. As the algorthm records what symbols represent the ntal, respectvely fnal, values of a varable or table, the nput and output values of the method (for the consdered path) can easly be extracted from the soluton to the constrant system. For our example, the data that were obtaned from the soluton to the constrant system are summarsed n the followng tables: Inputs Name Symbol(s) Value TABLE shelf d TABLE book CONTENT: shel f 1, ATTRIBUTES: d1 numbero f Books1 CONTENT: book1 ATTRIBUTES: code1 shel f Id1 newbooks newbooks1 [4] n.nextint() theshel f 1 [6] n.books code s.id 4 12 Outputs Name Symbolc(s) Value TABLE shelf d TABLE book CONTENT: shel f 1, ATTRIBUTES: d1 numbero f Books1 CONTENT: book1 ATTRIBUTES: code1 shel f Id1 newbooks newbooks1 [4] addedbooks addedbooks1 [] n.books code s.id 4 12 For sake of completeness, the followng tables defne the constrant generaton rules used by our algorthm n case of an Insert (table 1), Update (table 2), Delete (table 3) or Add/Remove (table 4) statement. Table 5 explans the abbrevatons used n the prevous tables. 15

16 Table 1: Constrants generaton rules for INSERT statements INSERT INTO d VALUES ( nt-expr 1,..., nt-expr,..., nt-expr n ) f (no excepton thrown n path for ths INSERT ) { ; Inserted prmary key value does not already exst (assert(forall((a d ))( (name( d ) a)(not (= (name(pk) a) smt2o f ( nt-expr pk pos)))))) ; Inserted values constraned by the th foregn key reference exstng rows (assert (exsts ((a fk tab ))( and (= (name(fk pk ) a) smt2o f ( nt-expr fk pos )) (name(fk tab ) a)))) ; Symbol for new table content (declare fun freshsym ( d ) Bool) ; Constrants descrbng new table content (assert (forall (( a d )) ( (name( d ) a) (freshsym a)))) (assert (exsts ((a d )) (and (= (att a) smt2o f ( nt-expr )) (freshsym a)))) (assert (forall (( a d )) ( (and (not (name( d ) a))(not (= (att a) smt2o f ( nt-expr ))))(not (freshsym a))))) ; No duplcate nserted row (assert (forall (( a d ) (b d )) ( (and (and (freshsym a) (freshsym b)) (= (pk a) (pk b))) (= a b)))) } else { // Logcal dsjuncton between every possble constrant // volaton gven the database schema and ths nsert: ; The nserted prmary key value already exsts n the table (exsts (( a d )) (and (name( d ) a) (= (name(pk) a) smt2o f ( nt-expr pk pos)))) ; th nserted foregn key value does not reference a row: (forall (( a fk tab ))( (name(fk tab ) a) (not (= (name(fk pk ) a) smt2o f ( nt-expr fk pos))))) ; An nserted attrbute volates the th check constrant : (not (co rght } smt2o f ( nt-expr co pos ))) 16

17 Table 2: Constrants generaton for UPDATE statements UPDATE d SET d att = db-nt-expr WHERE db-cond f (no excepton thrown n path for ths UPDATE ) { ; Symbol for new attrbute values (declare fun freshsym ( d ) Int) ; Constrants descrbng new attrbute values (assert (forall (( a d )) ( (or (and (name( d ) a) (not smt2o f ( db-cond, d,a))) (not (name( d ) a))) (= (freshsym a) (name( d att ) a))))) (assert (forall (( a d )) ( (and (name( d ) a) smt2o f ( db-cond, d,a)) (= (freshsym a) smt2o f ( db-nt-expr, d,a))))) ; Update on attrbute constraned by th foregn key does not let pendng references (assert (forall (( a d )) ( (name( d ) a) (exsts (( b fk tab )) (and (name(fk tab ) b) (= (freshsym a) (name(fk pk ) b))))))) ; Update on attrbute constraned by prmary key does not let duplcate attrbute values (assert (forall (( a d ) (b d )) ( (and (and (name( d ) a) (name( d ) b)) (= (freshsym a) (freshsym b))) (= a b)))) ; Update on prmary key referenced by th foregn key does not let pendng references (assert (forall (( a fk tab )) ( (name(fk tab ) a) (exsts (( b d ))( and (name( d ) b) (= (name(fk att ) a) (freshsym b))))))) ; Update on attrbute constraned by th check constrant does not volate the constrant (assert (forall (( a d )) (co rght (freshsym a)))) } else { // Logcal dsjuncton between every possble constrant // volaton gven the database schema and ths update: ; Update on prmary key leads to duplcate attrbute values (exsts (( a d ) (b d )) (and (and (name( d ) a) (and (name( d ) b) (not (= a b)))) (or (and smt2o f ( db-cond, d,a) (and smt2o f ( db-cond, d,b) (= smt2o f ( db-nt-expr, d,a) smt2o f ( db-nt-expr, d,b)))) (and (not smt2o f ( db-cond, d,a)) (and smt2o f ( db-cond, d,b) (= (name( d att ) a) smt2o f ( db-nt-expr, d,b)))))) ) ; Update on prmary key referenced by the th foregn key lets pendng references (exsts (( a d ) (b d )) (and (and (name( d ) a) (name(fk tab ) b)) (and (and (not (= (name( d att ) a) smt2o f ( db-nt-expr, d,a))) (= (name( d att ) a) (name(fk pk ) b))) smt2o f ( db-cond, d,a)))) ; Update on attrbute constraned by th foregn key lets pendng references (exsts (( a d )) (and (and (name( d ) a) smt2o f ( db-cond, d,a)) (not (exsts ((b name(fk tab ))) (= (name(fk pk ) b) smt2o f ( db-nt-expr, d,a)))))) ; Update on attrbute constraned by th check constrant volates the constrant (exsts (( a d )) (and (and (name( d ) a) smt2o f ( db-cond, d,a)) (not (co rght smt2o f ( db-nt-expr, d,a))))) } 17

18 Table 3: Constrants generaton for DELETE statements DELETE FROM d WHERE db-cond f (no excepton thrown n path for ths DELETE ) { ; Symbol for new table content (declare fun freshsym ( d ) Bool) ; Constrants descrbng new table content (assert (forall (( a d )) (= (freshsym a) (and (name( d ) a) (not smt2o f ( db-cond, d,a)))))) ; Delete does not let pendng references for th foregn key (assert(forall (( a fk tab ) (b d )) ( (and (name( d ) b) (and (not (freshsym b)) (name(fk tab ) a))) (not (= (name(pk) b) (name(fk att ) a)))))) } else { // Logcal dsjuncton between every possble constrant // volaton gven the database schema and ths update: ; Delete lets pendng references for th foregn key (exsts (( a fk tab ) (b d )) (and (and (and (name( d ) b) (name(fk tab ) a) ) smt2o f ( db-cond, d,b)) (= (name(pk) b) (name(fk att ) a)))) } Table 4: Constrants generaton for add(nt) and remove(nt) statements d.add( nt-expr ); (declare const freshsym BoundedLst) (assert (not (snull name( d )))) (assert (not (snull freshsym))) (assert (= (sze freshsym) (+ (sze name( d )) 1))) (assert (= (elements freshsym) (store (elements name( d )) (sze name( d )) smt2o f ( nt-expr )))) d.remove( nt-expr ); (declare const freshsym BoundedLst) (assert (not (snull name( d )))) (assert (not (snull freshsym))) (assert ( (sze name( d )) 1)) (assert (= (sze freshsym) ( (sze + oldvar+ ) 1))) (assert ( smt2o f ( nt-expr ) 0)) (assert (< smt2o f ( nt-expr ) (sze name( d )))) (assert (forall (( Int)) ( (and ( 0) (< smt2o f ( nt-expr ))) (= (select (elements name( d )) ) (select (elements freshsym) ))))) (assert (forall (( Int)) ( (and ( smt2o f ( nt-expr )) (< (sze freshsym))) (= (select (elements name( d )) (+ 1)) (select (elements freshsym) ))))) 18

19 Abbrevaton freshsym smt2o f (x) smt2o f (x, t, r) name(x) att pk pk pos fk tab fk pk fk pos fk tab Table 5: Abbrevatons lst Meanng A new symbol name that has stll not been used n the SMTv2 code generated so far. Java condton/expresson x translated nto a correspondng SMTv2 condton/expresson. SQL condton/expresson x evaluated for row r n table t translated nto a correspondng SMTv2 condton/expresson. f (x refers to a database table name) then The symbol that represents the current content of table x else f (x refers to a database attrbute name) The symbol that represents the current values of attrbute x else f (x refers to a Java varable name) The symbol that represents the current content of the Java varable x Name of the th attrbute n the lst of attrbutes of table d Name of the prmary key attrbute of table d. Poston of prmary key n the lst of attrbutes of table d Name of the table referenced by the th foregn key n the lst of foregn keys of table d Name of the prmary key attrbute of the table referenced by the th foregn key n the lst of foregn keys of table d Poston of the foregn key attrbute, declared by the th foregn key n the lst of table d, n the lst of attrbutes of table d Name of the table where s declared the th foregn key referencng table d n the whole schema fk att Name of the foregn key attrbute declared by the th foregn key referencng table d n the schema co pos co rght Poston of the attrbute constraned by the th check constrant declared n table d Inverted rght part of the th check constrant declared n table d (.e. nverted rght part of a > 0 s < 0 ) 19

20 3. Relatonal symbolc executon for database programs testng: an expermental valdaton 3.1. Symbolc executon and path exploraton Orgnally ntroduced n [5], symbolc executon has been used as the core prncple of many test data generaton technques. In most of these technques (see e.g. [27, 28] for an overvew), symbolc executon s performed for a fnte set of fnte paths that are statcally computed from the control-flow graph n accordance wth a gven coverage crteron [4]. By solvng the constrants assocated to each of the paths, one obtans a set of nputs and outputs that satsfy the gven coverage crteron. On the other hand, test data generaton technques that combne symbolc executon wth a dynamc path exploraton process have also been proposed (see e.g. [24, 25, 26]). The pont of usng a dynamc path exploraton process s to provde symbolc executon wth addtonal runtme nformaton, to make t more effectve [10, 28]. In practce, the program s frst executed on concrete nputs to produce concrete outputs, but the code s nstrumented so that symbolc executon s performed together wth ths normal run of the program, thereby generatng the constrant system correspondng to the concrete executon. By flppng some of the constrants among those generated by ths symbolc executon, one may produce constrants whose soluton descrbes new concrete nputs trggerng the executon of a dfferent path. The process s then repeated wth these new nputs untl a set of nputs and outputs has been generated for a set of paths coverng a suffcent part of the code, agan accordng to some coverage crteron [4]. The technque we propose n ths paper s orthogonal to the path exploraton process, and can be combned wth both a statc or dynamc approach. However, n order to evaluate the ablty and effcency our technque, we have bult a prototype tool that mplements our relatonal symbolc executon for the defned subset of Java/SQL DML based on a statc path exploraton process Evaluaton process Our tool works as follows. Gven a program to test, the code s analysed statcally to select a set of paths to test n the program s control-flow graph. In a nutshell, the tool smply performs a depth-frst search of the control-flow graph, selectng all possble paths that execute the body of each loop n the program at most K tmes, where K s an parameter of the tool. Consequently, the test data generated by our prototype satsfy a fntely applcable varant of the common path-coverage crteron [4], smlar to the loop count-k crteron orgnally proposed n [29]. For each generated path, the tool apples the relatonal symbolc executon algorthm proposed n secton 2, and solves the produced constrants usng the Z3 [17] solver. If the solver fnds a soluton for the constrants, nput and output data (ncludng database content) s extracted from ths soluton and recorded as consttutng a test-case. Once all the paths have been explored, a so-called test-sute comprsng all the generated test-cases s returned to the user. If, on the other hand, the constrant set s found to be unsatsfable, ths means that the consdered path cannot correspond to a real executon, and the process smply contnues wth the next path. The tool also keeps a separate lst of the paths for whch the solver cannot solve properly the constrants. Ths can happen as the constrants are wrtten n a logc that s not decdable n general [16]. 20

21 We have evaluated our prototype tool by usng t to generate test data for eghteen Java methods, each of them performng SQL operatons over a relatonal database. The methods we have used for the evaluaton can be dvded nto three groups: The frst group of methods were smply crafted to systematcally evaluate the correct symbolc executon of the dfferent constructs of the Java/SQL sub-language proposed n secton 2.1. As such, the methods n ths group exercse the dfferent behavours of the nteger and lst operators, condtonal and loop statements, SQL statements and transacton management prmtves. The second group contans the methods that were used to evaluate a prevous verson of our prototype [21]. The frst method n ths group performs repeated manpulatons of ntegers and lsts usng assgnment, f and whle statements. The second method performs many nterleaved reads and wrtes n a database contanng four tables (representng regular or prospect customers that make purchases of products). The thrd method mxes SQL statements wth tradtonal Java code and uses SQL transactons. The manpulated database contans two tables that represent authors wrtng theater plays. The thrd group contans Java methods extracted from two real-world applcatons, namely UnxUsage 2 and RskIt 3, that have also been used as a bass for evaluaton of other works on test data generaton for database applcatons by symbolc executon [30, 13]. UnxUsage s a montorng applcaton for Unx, manpulatng a database wth eght tables and thrty one attrbutes. RskIt s an nsurance rate estmaton applcaton, manpulatng a database wth thrteen tables and ffty-seven attrbutes. Together, the eghteen methods from our testbed consttute a set of fve hundreds lnes of code, contanng notably eghty SQL statements (ncludng SELECT, INSERT, UPDATE, DELETE statements, as well as transacton management code), over databases contanng up to thrteen tables (subject to prmary key, foregn key and check constrants). The code of these methods, as well as the generated test data, can be found on the web 4. The prototype tool s tself coded n Java 1.6 and run on a dual core Intel Core 5 processor at 1.8GHz (256 KB L2 cache per core and 3 MB L3 cache) wth 8GB of dual channel DDR3 memory at 1600 MHz. The runtme envronment was the Oracle JVM under a 32-bt edton of Wndows 8.1. The verson of the Mcrosoft Z3 solver was used Results descrpton Table 6 synthesses the obtaned results. In total paths were symbolcally executed and all generated constrant sets were properly solved by the solver: paths were dscovered to be nfeasble (.e. not correspondng to a real executon), whle test data were extracted from the computed solutons for the remanng 113 paths. The hgh number of nfeasble paths s prncpally due to the methods n the second group (and, to a lesser extend, the methods for testng condtonal constructs from the frst

22 group), methods that were partcularly constructed n ths way for testng the constrantgeneraton code. The total tme for constrant solvng was nne mnutes and thrty seconds. Although the methods from the frst group are all small and contan only smple operatons, the results bascally serve as a proof of concept showng the ablty of our approach to generate, n a reasonable tme, the correct constrants for a set of executon paths and, by solvng them, to be able to ether derve test data or correctly show the unfeasblty of the path. The results for the methods n the second group reveal mportant mprovements when compared to a prevous verson of our tool [21], n whch the same methods were symbolcally executed, but the generated constrants were expressed usng the Alloy language [22] and solved usng the Alloy analyzer [22]. The Alloy Analyzer solves a set of relatonal constrants by settng an upper bound on the sze of the relatons to be found. Ths makes fnte the set of possble solutons, whch can be exhaustvely explored by transformng the constrants nto an equvalent SAT problem, solved by a dedcated SAT solver. Ths mechansm s very costly n tme and, moreover, dd not allow to establsh that a set of relatonal constrants s unsatsfable [16]. As a consequence, [21] took more than eght mnutes and a half to fnd test data for one path n the second sample and was not able to detect properly any nfeasble path. In contrast, the current verson of our tool, whch makes use of the Z3 solver took thrty-two seconds to fnd test data or to establsh unfeasblty for thrty-sx paths n the second sample, ncludng the path consdered n [21]. Gven the effcency mprovements wth respect to our earler work, we were now able to test our tool on real-world code. Test data generaton for the thrd group of methods requred to extend our symbolc executon algorthm to handle some new parts of Java and SQL, used n UnxUsage and RskIt, or to smulate them n the base language recognsed by the algorthm. In partcular, the management of tables wth no prmary key or wth multple-attrbute prmary key was ntegrated n the algorthm and strng management operatons were smulated usng ether ntegers or lsts of ntegers. As Table 6 show, correct test data was generated n just a few seconds and, moreover, the generated test data allowed to detect a possble falure n the code of RskIt, where a runtme error s thrown when the method createnewuser s called on nputs where the nserted job does not reference any exstng occupaton or ndustry. 22

23 Table 6: Statstcs for the selected samples Name SLOC Number of Number K Number of Paths Solvng SQL statements of tables Feasble Infeasble Tme Language unts testng Integers < 1s Lsts < 1s If and Whle s Condtons s Select s Insert <1s Update s Delete <1s Transactons <1s Samples from [21] Sample m 16s Sample s Sample m 35s Samples from UnxUsage [30] coursenameexsts s getcommandsbycategory <1s getcourseidbyname s getdepartmentidbyname <1s Samples from RskIt [30] createnewuser s deleteusers s ALL METHODS up to 13 up to m30s 23

24 4. Dscusson 4.1. Conclusons Symbolc executon s the core process of many exstng test data generaton technques [9]. These approaches explore a representatve [4] set of fnte paths to test n the program, ether statcally [10] or dynamcally [24, 25, 26], and execute them symbolcally to generate meanngful test nputs and outputs for the program. In ths work, we propose an approach for the symbolc executon of SQL code, whch s a non-trval extenson to tradtonal symbolc executon because of the complex structure of relatonal databases and the complex behavour of SQL statements. Gven a database program mxng tradtonal code wth SQL statements, each database table manpulated by the program s modelled as a varable typed as a relaton and each SQL statement as a relatonal operaton over both these relatonal varables and the tradtonal varables of the program. A classcal symbolc executon process can then be appled to produce sets of mxed relatonal and conventonal constrants over symbols representng the values of both the classcal and relatonal varables of the program. The resultng path constrants can be unfed wth the data ntegrty constrants from the database schema. Any soluton to the resultng constrant system for a path descrbes nput and output data for the program, ncludng a vald database content, wth respect to whch the program can be executed and s guaranteed to follow the same path for whch the constrants were generated. A complete symbolc executon algorthm has been developed for smple Java methods that, usng SQL statements and transactons, read and wrte n a relatonal database; the latter typcally subject to data ntegrty constrants. Gven the schema of the database, the code of the method and an executon path n ths method, our algorthm performs symbolc executon of the path and produces the correspondng constrants. The algorthm has been mplemented n a tool that has been used to generate test data for a number of methods nteractng wth a database, ncludng some real-world applcaton code. For every tested method, we used the symbolc executon algorthm to generate test data for a set of paths that s statcally generated by a bounded depth-frst traversal of the program s control-flow graph [4]. Results revealed that the approach can properly consder thousands of executon paths n a reasonable tme and thus generate meanngful test data. Compared to our prevous work [20, 21] where the generated constrants were expressed usng the Alloy language [22] and solved usng the Alloy analyser [22], the current mplementaton generates SMTv2 [15] constrants whch are solved usng the Z3 solver [17] followng an approach smlar to [16]. SMTv2 s the standard nput language for SMT solvers and Z3 s used as constrant solvng back-end n several symbolc executon tools [18, 19]. Results show that ths change makes the approach able to properly detect nfeasble paths, somethng that the mplementaton based on Alloy was unable to, and allows to reduce the tme needed for constrant solvng by several orders of magntude Related work An early work that has consdered test data generaton for programs nteractng wth a relatonal database s [11]. The paper suggests to transform the program by nsertng new classcal varables representng the database structure, and translatng all SQL statements nto tradtonal program code. Classcal whte-box testng approaches can then be appled to the normalsed program. A conceptually smlar but entrely automated technque s proposed n [13], where an off-the-shelf test generaton technque, based on symbolc 24

25 executon and a dynamc path exploraton process, s appled to the normalsed verson of the program. Ths latter approach s valdated over real applcaton code, consderng methods contanng compound SELECT statements. Normalsng SQL code nto program code and then performng symbolc executon on the result s an alternate approach to ours, where the SQL code s drectly transformed nto relatonal constrants. Replacng a sngle SQL statement by an often complex pece of new program code that smulates the executon of the SQL statement by a DBMS, wll dramatcally ncrease the number of paths to be explored compared to the orgnal code [11]. In partcular, gven a path n the orgnal code, all the combnatons between ths orgnal path and the dfferent subpaths traversng the newly added code should be shown nfeasble, n order to show that the orgnal path was nfeasble n the orgnal code. As the new code wll contan loops to enumerate the unbounded content of the database tables [13], there s an nfnte number of subpaths that traverse ths new code, and an nfnte number of paths should thus be analysed n the combned code to detect properly f a path n the orgnal code was nfeasble. In contrast, n our approach, the executon paths to be consdered are lmted to paths n the orgnal program code and, consequently, our technque s able to quckly detect the defnte unfeasblty of a path. In [12], the authors propose a symbolc executon algorthm for programs performng basc SELECT queres on a relatonal SQL database. Constrants are solved usng an ad-hoc solver for lnear arthmetc and strngs. The proposed algorthm s ntegrated n a test generaton technque based on a dynamc path exploraton process and evaluated over real applcaton code. Compared to ths approach, our technque handles INSERT, UPDATE and DELETE statements, as well as transacton management prmtves, whch are crucal components of database applcatons. Moreover, [12] gnores the data ntegrty constrants defned n the database schema, leadng to the possble generaton of nvald test data. On the other hand, our approach only consders SQL statements whose structure s completely defned statcally, where [12] (and also [13]) can allow to account, at least to some extent, for dynamcally crafted SQL statements. Ths s lnked to the fact that these two technques use a dynamc path exploraton process, where symbolc executon s performed n parallel wth a concrete run of the program, gvng drect access to the code of the dynamc SQL statements, as t s bult by the program durng the concrete run. Followng [12], several works have studed addtonal research questons n the framework of test generaton for database programs usng symbolc executon and a dynamc path exploraton process. Among them, [31] proposes to use a mock database n case the orgnal database s not avalable, [32] consders usng advanced code coverage crtera, [33] consders programs where parts of the executed SQL queres are nputs of the program. Fnally, [30] and [33] study stuatons where the test data should be selected n an exstng database nstead of beng generated from scratch. On a related but complementary level, a substantal amount of work (e.g. [34, 35, 36, 37, 38, 39, 40, 41, 42, 43]) has been done on how to generate test database content exhbtng some desrable propertes, gven only the database schema and a set of queres to be executed over the database. The man dfference between our work and these approaches s that they essentally work wthout consderng the control flow of the programs manpulatng the database. Other work [44, 45] consders mutaton testng of database programs, where our approach performs structural testng. In mutaton testng, the qualty of the test data s no more measured n terms of code coverage, as n structural testng, but n terms of program fault detectng ablty (see [4] for a broader dscusson). 25

26 Some works have also focused over testng of non-functonal aspects of database programs, lke securty testng [46, 47] and performance testng [48] Future work In future work t should be nvestgated to what extent the symbolc executon mechansm can be generalsed and extended towards dealng wth a larger part of the SQL language. In partcular, t should be nvestgated how and to what extent fully dynamc SQL can be ntegrated n our technque, possbly by relyng on statc analyss [49, 50, 51] or by couplng our symbolc executon algorthm wth a dynamc path exploraton process, n a smlar way to [12, 13]. Ideas proposed n [33], where parts of the executed SQL queres are nputs of the program, should also be nvestgated. It also frequently happens that SQL statements have a non-determnstc behavour, because the statement has a non-determnstc semantcs, or because the nteracton wth the DBMS mght not be relable or because the database s modfed concurrently by other methods or programs. How an approach for test data generaton could handle handle such non-determnstc behavour remans a topc for further research. In the perspectve of usng an approach such as ours for testng large real-world programs that manpulate huge databases usng many complex SQL statements, t s to be expected that a tghter ntegraton between constrant generaton (path exploraton) and constrant solvng would be benefcal. The constrant generator should be talored so to generate very effcent sets of constrants, expressed n parts of the logc that are decdable or optmzed for the heurstcs used by the solver. Conversely, the use of solvng algorthms or heurstcs talored to the knd of constrants produced by the symbolc executon of large peces of complex SQL code should also be consdered. In addton to dealng wth more nvolved database manpulatons, the approach would also proft from an ntegraton wth exstng symbolc executon test generaton tools [9], handlng a larger part of Java, lke [52] or [53, 54]. In a smlar van, the use of exstng Z3 plug-ns, lke [55], allowng the natve management of strng constrants wthn the Z3 solver, should permt our algorthm to more easly handle strng values, whch are wdely used n practce wthn database programs. The problem of generatng test data for programs manpulatng an exstng database s a very common problem n practce. Whether and how our technque should select test data from an exstng database, nstead of generatng one from scratch, n a smlar way to [30, 33], s an nterestng matter of further research, as s the problem of consderng or generatng a sngle database content that could be used wth several successve test cases. Fnally, beng somehow parametrsed wth respect to the paths that should be consdered, our approach allows to be used wth respect to any tradtonal code coverage crteron based on the noton of executon path [4]. Nevertheless, several works [56, 57, 58, 59, 60] propose test adequacy crtera partcularly talored towards testng of database programs. Integratng such crtera nto our constrant-based approach s a topc of ongong research. Acknowledgment Ths work has been funded by the Belgan Fund for Scentfc Research (F.R.S.-FNRS). The authors would lke to thank Perre-Yves Schobbens and Glles Perroun for useful dscussons and the anonymous referees for ther constructve remarks on earler versons of ths paper. 26

27 References [1] P. C. Jorgensen, Software Testng: A Craftsman s Approach, Thrd Edton, 3rd Edton, AUER- BACH, [2] C. Kaner, H. Q. Nguyen, J. L. Falk, Testng Computer Software, 2nd Edton, Wley & Sons, New York, NY, USA, [3] R. Ramler, K. Wolfmaer, Economc perspectves n test automaton: balancng automated and manual testng wth opportunty cost, n: Proceedngs of the 2006 nternatonal workshop on Automaton of software test, AST 06, ACM, New York, NY, USA, 2006, pp [4] H. Zhu, P. A. V. Hall, J. H. R. May, Software unt test coverage and adequacy, ACM Comput. Surv. 29 (4) (1997) [5] J. C. Kng, Symbolc executon and program testng, Commun. ACM 19 (7) (1976) [6] P. D. Coward, Symbolc executon systems: a revew, Softw. Eng. J. 3 (6) (1988) [7] C. S. Pasareanu, W. Vsser, A survey of new trends n symbolc executon for software testng and analyss, Int. J. Softw. Tools Technol. Transf. 11 (4) (2009) [8] E. J. Schwartz, T. Avgernos, D. Brumley, All you ever wanted to know about dynamc tant analyss and forward symbolc executon (but mght have been afrad to ask), n: Proceedngs of the 2010 IEEE Symposum on Securty and Prvacy, SP 10, IEEE Computer Socety, Washngton, DC, USA, 2010, pp [9] C. Cadar, P. Godefrod, S. Khurshd, C. S. Păsăreanu, K. Sen, N. Tllmann, W. Vsser, Symbolc executon for software testng n practce: Prelmnary assessment, n: Proceedngs of the 33rd Internatonal Conference on Software Engneerng, ICSE 11, ACM, New York, NY, USA, 2011, pp [10] C. Cadar, K. Sen, Symbolc executon for software testng: three decades later, Communcaton of the ACM 56 (2) (2013) [11] M. Y. Chan, S. C. Cheung, Testng database applcatons wth sql semantcs, n: In Proceedngs of the 2nd Internatonal Symposum on Cooperatve Database Systems for Advanced Applcatons, Sprnger, 1999, pp [12] M. Emm, R. Majumdar, K. Sen, Dynamc test nput generaton for database applcatons, n: Proceedngs of the 2007 nternatonal symposum on Software testng and analyss, ISSTA 07, ACM, New York, NY, USA, 2007, pp [13] K. Pan, X. Wu, T. Xe, Guded test generaton for database applcatons va syntheszed database nteractons, ACM Transactons on Software Engneerng and Methodology. [14] C. Date, An Introducton to Database Systems, 8th Edton, Addson-Wesley Longman Pub. Co., Inc., Boston, MA, USA, [15] C. Barrett, A. Stump, C. Tnell, The SMT-LIB Standard: Verson 2.0, n: A. Gupta, D. Kroenng (Eds.), Proceedngs of the 8th Internatonal Workshop on Satsfablty Modulo Theores (Ednburgh, UK), [16] A. A. El Ghaz, M. Taghdr, Relatonal reasonng va smt solvng, n: Proceedngs of the 17th nternatonal conference on Formal methods, FM 11, Sprnger-Verlag, Berln, Hedelberg, 2011, pp [17] L. De Moura, N. Bjørner, Z3: An effcent smt solver, n: Proceedngs of the Theory and Practce of Software, 14th Internatonal Conference on Tools and Algorthms for the Constructon and Analyss of Systems, TACAS 08/ETAPS 08, Sprnger-Verlag, Berln, Hedelberg, 2008, pp [18] N. Tllmann, J. De Halleux, Pex: whte box test generaton for.net, n: Proceedngs of the 2nd nternatonal conference on Tests and proofs, TAP 08, Sprnger-Verlag, Berln, Hedelberg, 2008, pp [19] P. Godefrod, M. Y. Levn, D. Molnar, Sage: Whtebox fuzzng for securty testng, Queue 10 (1) (2012) 20:20 20:27. [20] M. Marcozz, W. Vanhoof, J.-L. Hanaut, Test nput generaton for database programs usng relatonal constrants, n: Proceedngs of the Ffth Internatonal Workshop on Testng Database Systems, DBTest 12, ACM, New York, NY, USA, 2012, pp. 6:1 6:6. [21] M. Marcozz, W. Vanhoof, J.-L. Hanaut, A relatonal symbolc executon algorthm for constrantbased testng of database programs, n: Source Code Analyss and Manpulaton (SCAM), 2013 IEEE 13th Internatonal Workng Conference on, 2013, pp [22] D. Jackson, Software Abstractons: Logc, Language, and Analyss, The MIT Press, [23] S. Bardn, P. Herrmann, Prunng the search space n path-based test generaton, n: Proceedngs of the 2009 Internatonal Conference on Software Testng Verfcaton and Valdaton, ICST 09, IEEE Computer Socety, Washngton, DC, USA, 2009, pp

28 [24] P. Godefrod, N. Klarlund, K. Sen, Dart: drected automated random testng, SIGPLAN Not. 40 (6) (2005) [25] K. Sen, D. Marnov, G. Agha, Cute: a concolc unt testng engne for c, SIGSOFT Softw. Eng. Notes 30 (5) (2005) [26] C. Cadar, D. Engler, Executon generated test cases: How to make systems code crash tself, n: Proceedngs of the 12th Internatonal Conference on Model Checkng Software, SPIN 05, Sprnger- Verlag, Berln, Hedelberg, 2005, pp [27] J. Edvardsson, A survey on automatc test data generaton, n: Proceedngs of the Second Conference on Computer Scence and Engneerng n Lnköpng, ECSEL, 1999, pp [28] D. D Souza, T. Kavtha, J. Radhakrshnan (Eds.), IARCS Annual Conference on Foundatons of Software Technology and Theoretcal Computer Scence, FSTTCS 2012, December 15-17, 2012, Hyderabad, Inda, Vol. 18 of LIPIcs, Schloss Dagstuhl - Lebnz-Zentrum fuer Informatk, [29] W. E. Howden, Methodology for the generaton of program test data, Computers, IEEE Transactons on 100 (5) (1975) [30] K. Pan, X. Wu, T. Xe, Generatng program nputs for database applcaton testng, n: Proc. 26th IEEE/ACM Internatonal Conference on Automated Software Engneerng (ASE 2011), [31] K. Taneja, Y. Zhang, T. Xe, Moda: Automated test generaton for database applcatons va mock objects, n: In Proc. IEEE/ACM Internatonal Conference on Automated Software Engneerng (ASE 2010), short paper, [32] K. Pan, X. Wu, T. Xe, Database state generaton va dynamc symbolc executon for coverage crtera, n: Proceedngs of the Fourth Internatonal Workshop on Testng Database Systems, ACM, New York, NY, USA, [33] C. L, C. Csallner, Dynamc symbolc database applcaton testng, n: Proceedngs of the Thrd Internatonal Workshop on Testng Database Systems, DBTest 10, ACM, New York, NY, USA, 2010, pp. 7:1 7:6. [34] D. Chays, Y. Deng, P. G. Frankl, S. Dan, F. I. Vokolos, E. J. Weyuker, An agenda for testng relatonal database applcatons: Research artcles, Softw. Test. Verf. Relab. 14 (1) (2004) [35] D. Chays, J. Shahd, P. G. Frankl, Query-based test generaton for database applcatons, n: Proceedngs of the 1st nternatonal workshop on Testng database systems, DBTest 08, ACM, New York, NY, USA, 2008, pp. 6:1 6:6. [36] A. Neufeld, G. Moerkotte, P. C. Lockemann, Generatng consstent test data for a varable set of general consstency constrants, VLDB J. 2 (2) (1993) [37] J. Zhang, C. Xu, S. C. Cheung, Automatc generaton of database nstances for whte-box testng, n: Proceedngs of the 25th Internatonal Computer Software and Applcatons Conference on Invgoratng Software Development, COMPSAC 01, IEEE Computer Socety, Washngton, DC, USA, 2001, pp [38] C. Bnng, D. Kossmann, E. Lo, Reverse query processng, n: n the IDEA System. Int. Symp. on Advanced Database Technologes and Ther Integraton, [39] S. A. Khalek, B. Elkarableh, Y. O. Laleye, S. Khurshd, Query-aware test generaton usng a relatonal constrant solver, n: Proceedngs of the rd IEEE/ACM Internatonal Conference on Automated Software Engneerng, ASE 08, IEEE Computer Socety, Washngton, DC, USA, 2008, pp [40] M. Veanes, P. Grgorenko, P. Halleux, N. Tllmann, Symbolc query exploraton, n: Proceedngs of the 11th Internatonal Conference on Formal Engneerng Methods: Formal Methods and Software Engneerng, ICFEM 09, Sprnger-Verlag, Berln, Hedelberg, 2009, pp [41] C. de la Rva, M. J. Suárez-Cabal, J. Tuya, Constrant-based test database generaton for sql queres, n: Proceedngs of the 5th Workshop on Automaton of Software Test, AST 10, ACM, New York, NY, USA, 2010, pp [42] D. Wllmor, S. M. Embury, An ntensonal approach to the specfcaton of test cases for database applcatons, n: Proceedngs of the 28th nternatonal conference on Software engneerng, ICSE 06, ACM, New York, NY, USA, 2006, pp [43] C. Y. Chan, B. C. Oo, A. Zhou (Eds.), Proceedngs of the ACM SIGMOD Internatonal Conference on Management of Data, Bejng, Chna, June 12-14, 2007, ACM, [44] C. Zhou, P. Frankl, Mutaton testng for java database applcatons, n: Software Testng Verfcaton and Valdaton, ICST 09. Internatonal Conference on, 2009, pp [45] K. Pan, X. Wu, T. Xe, Automatc test generaton for mutaton testng on database applcatons, n: Automaton of Software Test (AST), th Internatonal Workshop on, 2013, pp [46] W. G. J. Halfond, A. Orso, P. Manolos, Usng postve tantng and syntax-aware evaluaton to counter sql njecton attacks, n: Proceedngs of the 14th ACM SIGSOFT nternatonal symposum 28

29 on Foundatons of software engneerng, SIGSOFT 06/FSE-14, ACM, New York, NY, USA, 2006, pp [47] Z. Su, G. Wassermann, The essence of command njecton attacks n web applcatons, SIGPLAN Not. 41 (1) (2006) [48] X. Wu, C. Sanghv, Y. Wang, Y. Zheng, Prvacy aware data generaton for testng database applcatons, n: Proceedngs of the 9th Internatonal Database Engneerng & Applcaton Symposum, IDEAS 05, IEEE Computer Socety, Washngton, DC, USA, 2005, pp [49] G. Wassermann, C. Gould, Z. Su, P. Devanbu, Statc checkng of dynamcally generated queres n database applcatons, ACM Trans. Softw. Eng. Methodol. 16 (4). [50] S. Thomas, L. Wllams, T. Xe, On automated prepared statement generaton to remove sql njecton vulnerabltes, Inf. Softw. Technol. 51 (3) (2009) [51] A. Chrstensen, A. M,àö,àèller, M. Schwartzbach, Precse analyss of strng expressons, n: R. Cousot (Ed.), Statc Analyss, Vol of Lecture Notes n Computer Scence, Sprnger Berln Hedelberg, 2003, pp [52] K. Sen, G. Agha, Cute and jcute: concolc unt testng and explct path model-checkng tools, n: Proceedngs of the 18th nternatonal conference on Computer Aded Verfcaton, CAV 06, Sprnger-Verlag, Berln, Hedelberg, 2006, pp [53] S. Anand, C. S. Păsăreanu, W. Vsser, Jpf-se: A symbolc executon extenson to java pathfnder, n: Proceedngs of the 13th Internatonal Conference on Tools and Algorthms for the Constructon and Analyss of Systems, TACAS 07, Sprnger-Verlag, Berln, Hedelberg, 2007, pp [54] C. S. Pǎsǎreanu, P. C. Mehltz, D. H. Bushnell, K. Gundy-Burlet, M. Lowry, S. Person, M. Pape, Combnng unt-level symbolc executon and system-level concrete executon for testng nasa software, n: Proceedngs of the 2008 Internatonal Symposum on Software Testng and Analyss, ISSTA 08, ACM, New York, NY, USA, 2008, pp [55] Y. Zheng, X. Zhang, V. Ganesh, Z3-str: A z3-based strng solver for web applcaton analyss, n: Proceedngs of the th Jont Meetng on Foundatons of Software Engneerng, ESEC/FSE 2013, ACM, New York, NY, USA, 2013, pp [56] W. G. J. Halfond, A. Orso, Command-form coverage for testng database applcatons, n: Proceedngs of the 21st IEEE/ACM Internatonal Conference on Automated Software Engneerng, ASE 06, IEEE Computer Socety, Washngton, DC, USA, 2006, pp [57] G. M. Kapfhammer, M. L. Soffa, A famly of test adequacy crtera for database-drven applcatons, n: Proceedngs of the 9th European software engneerng conference held jontly wth 11th ACM SIGSOFT nternatonal symposum on Foundatons of software engneerng, ESEC/FSE-11, ACM, New York, NY, USA, 2003, pp [58] M. J. Suárez-Cabal, J. Tuya, Usng an sql coverage measurement for testng database applcatons, n: Proceedngs of the 12th ACM SIGSOFT twelfth nternatonal symposum on Foundatons of software engneerng, SIGSOFT 04/FSE-12, ACM, New York, NY, USA, 2004, pp [59] M. J. Suarez-Cabal, J. Tuya, Structural coverage crtera for testng SQL queres, Journal of Unversal Computer Scence 15 (3) (2009) [60] C. Zhou, P. Frankl, Mutaton testng for java database applcatons, n: Proceedngs of the 2009 Internatonal Conference on Software Testng Verfcaton and Valdaton, ICST 09, IEEE Comp. Soc., Washngton, DC, USA, 2009, pp