Migration of MS Access Databases to Mendix Platform Schema Migration and Foreign Key Recovery

Transcription

1 Migration of MS Access Databases to Mendix Platform Schema Migration and Foreign Key Recovery Theodora Boudale September 28, 2014, 52 pages Supervisor: Host organisation: Internship Supervisor: Tijs van der Storm Mendix Technology BV Jouke Waleson Universiteit van Amsterdam Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering

2 Contents Abstract 2 1 Introduction Background and Context Problem Description Research Questions Outline Problem Analysis Schema Migration Data Migration Queries Migration Overview of our approach Access2Mendix: Access Schema Migration Solution: Schema Mappings Implementation Evaluation Foreign Key Discovery Approach: Inclusion Dependencies Implementation Evaluation Discussion Schema Migration Foreign Key Recovery Queries Migration Related Work 35 7 Conclusion and Future Work Summary and Conclusion Future Work Bibliography 39 Appendices 41 A Access Field & Table Properties 42 B Access Expression Language - Mendix Expressions 47 C Access SQL - Mendix XPath Operators 49 D Access SQL - Mendix OQL Predicates & Operators 51 1

3 Abstract This thesis is concerned with the migration of Microsoft Access databases to Mendix Platform. We investigate similarities and differences between the data models of the two systems, discuss issues in regards to data migration and also examine possible options for migrating database queries. A tool was implemented for the automated migration of an MS Access schema to Mendix Domain Model and we present its possibilities and limitations. Additionally, since many real-world databases lack an explicit definition of foreign key relationships, we implemented a tool for discovering those constraints. Our approach was to examine the data for inclusion dependencies, and our results show that these constraints are successfully retrieved. However, a big number of false positives is also discovered in databases that make extended use of Autonumber fields. 2

4 Chapter 1 Introduction 1.1 Background and Context Legacy information systems have an important role within many organizations. They usually operate daily supporting important business processes. They are also the source of valuable business information, since many of them represent a long-term investment of a company. However, these systems often cause problems to their host organizations. Some of those systems may be poorly designed or become complex with time. As a result, maintenance becomes a very time-consuming and expensive task. Adding new functionality or fixing a bug may require a significant amount of time and result in making the program more complex. Lack of documentation can make this task even more difficult. Additionally, skill shortage for obsolete technologies, for example COBOL, makes it difficult for companies to find experts in supporting some legacy systems [HCHH08]. For these reasons many organizations choose to migrate from one platform to another or one language to another. There are several examples of software migration projects in literature, for example converting from one language to another, converting to a newer version of a language, or migrating to a different type of database [AGA + 06]. A software migration project requires the investigation of a major issue: what can be migrated from the source system to the target one. A mapping between the components of the two systems needs to be determined. These components may have different semantics and a major challenge here is the resolution of this semantic gap. As Andrade et al. note [AGA + 06], the width of this gap decides the feasibility and complexity of such a conversion. The wider the gap, the less feasible, more complex and less automated the migration is [AGA + 06]. In migration projects one may also have to deal with underspecified data sources. Information about the database schema might be missing, and there might be no documentation to aid in its recovery. For instance, many relational databases lack an explicit definition of Foreign Key Relationships. The recovery of such constraints is an interesting challenge and a necessary step for a successful migration. 1.2 Problem Description Mendix is a company based in Rotterdam that offers a Model-Driven Development (MDD) Tool for developing web and mobile applications. MDD tools aim to separate the business logic of a program from its implementation technology [MRA05]. They allow developers to work on a higher level of 3

5 abstraction by creating models to capture a program s structure and behavior. These models are consequently used in order to automatically generate third generation language code [MRA05]. Many customers of Mendix have applications built with Microsoft Access and want to migrate them to Mendix Platform. Mendix wants to offer their customers a tool that can perform an automated migration of MS Access applications to their platform. Additionally, many of those customers applications do not have Foreign Key constraints explicitly defined. The automated discovery of such constraints would be a big aid in correctly migrating the Relationships of MS Access schema to Mendix Overview of Platforms Microsoft Access is a 4GL application design and development tool. It combines the Microsoft Jet database engine with software development tools, and can be used for the development of desktop and web applications. A MS Access application consists of the Database, the User Interface, and the Application Code. In Table 1.1 we present the main elements of a MS Access application from a developer s viewpoint, categorized along these three dimensions. Database Table: A set of data elements that is organized in rows and columns. Query: Queries are used in order to retrieve or operate on data. Form: An object that is used to create the user interface for an application. User Interface Application Code Report: An object that is used to display or summarize data, usually formatted for printing. Macro: A script for performing a task. It allows automation of tasks and adds functionality to forms, reports and controls. Module: A collection of Visual Basic declarations, statements and procedures, stored as one named unit. Table 1.1: Elements of a MS Access Application Mendix Business Modeler is the tool that allows for the development of Mendix applications with the creation of models. There are three main model types that can be used to build a Mendix application. In Table 1.2 we give a brief overview of them. Domain Model: A data model that describes information in the application domain. This model is used for the generation of the database schema. Pages: Pages are used in order to create the interface for the end-user. A number of widgets are available. Microflows: Process models that express the logic of the application. Table 1.2: Elements of a Mendix application 4

6 1.2.2 Scope In this project we are going to focus on the first dimension of the migration, the Database. Our approach is based on the Database First migration method [WLB + 97]. A MS Access database consists of the Tables (schema and data) and the Queries. In this research we will explore mappings between MS Access schema elements and Mendix Domain Model elements. We will discuss the data migration process, and finally investigate options for migrating MS Access Queries. Afterwards we will create a tool for the automatic migration of a MS Access database schema to Mendix Domain Model. Additionally, since many real-world applications do not have Relationships explicitly defined, we will create a tool for recovering Foreign Key constraints. We will use version 2010 of Microsoft Access and version of Mendix Business Modeler. MS Access can also be part of a more complex system, comprising of other databases and programs. In this project we will focus on applications built with MS Access only. We do not consider databases that make use of passwords or applications with multiple user accounts, since user-level security is not available in Access Microsoft Access might be referred to as Access only in the rest of this thesis. 1.3 Research Questions The main question we will try to answer is: Can we automatically migrate a Microsoft Access database to Mendix Business Modeler? We have the following subquestions: Which features of an Access database schema can be mapped to Mendix Domain Model? Which features cannot be mapped and what are the consequences? Can we automatically migrate MS Access Queries? When or where is human intervention needed? Can we recover missing Foreign Key Relationships from an Access database? 1.4 Outline The remainder of this thesis is structured as follows: In Chapter 2 we give a detailed problem analysis along the three dimensions of the database migration process, namely the schema, data, and queries. In Chapter 3 we present our solution for schema migration. We discuss about the tool we implemented, the features it migrates and its limitations. Chapter 4 is concerned with foreign key recovery. In Chapter 5 we discuss our results in relation with our research questions. In Chapter 6 we give an overview of related work. Finally, in Chapter 7 the conclusion and directions for future work are presented. 5

7 Chapter 2 Problem Analysis 2.1 Schema Migration Schema conversion is the translation of the legacy database structure into an equivalent database structure expressed in the new technology [HCHH08]. This process requires identifying similar or semantically related elements between the source and target systems [DSDR07]. In this section we are going to present MS Access schema elements and explore possible mappings to Mendix Domain Model elements Main Schema Elements Microsoft Access is a relational database management system; data is represented in tuples, grouped into Tables. Each Field of a Table retrieves its values from a certain domain (it has a certain Data Type). Every Table can have exactly one Primary Key that uniquely identifies each row in the table. Additionally, Tables may relate to other Tables with a Foreign Key Relationship. Indexes may also be defined on Fields to speed up search and sorting. The Domain Model in Mendix Business Modeler is based on concepts similar to the relational model. It consists of Entities and their relations, Associations. An Entity can have a number of attributes, each one with its own type and properties. Indexes may also be defined on attributes. We present the mapping between the main constructs of the two systems in Table 2.1. MS Access Schema Elements Tables Fields Relationships Mendix Domain Model Elements Entities Attributes Associations Table 2.1: MS Access Schema - Mendix Domain Model Elements Field Data Types Every Field in an Access Table has a certain Data Type that determines what kind of data can be stored in that Field. Similarly, Attributes in Mendix Domain Model have their own types. We examined the available Data Types in both systems and present them in Table

8 MS Access Field Data Types Text Memo Number Byte Integer Long Integer Single Double Replication ID Decimal Date/Time Currency Autonumber Mendix Domain Model Attribute Types String String Integer Integer Long Integer Float Float Not available as an attribute type Float Date and Time Currency Long Integer Increment Random Replication ID Yes/No Hyperlink OLE Object Attachment Calculated Lookup Field Autnonumber Not available as an attribute type Not available as an attribute type Boolean Not available as an attribute type Not available as an attribute type Not available as an attribute type Not available as an attribute type Not available as an attribute type Table 2.2: Access Field - Mendix Attribute Data Types Most Field Data Types have an equivalent Attribute Type in Mendix. As we can see in Table 2.2 we have mismatches in certain cases, and discuss those below. Replication ID is a Data Type used in database replication, a process where multiple copies of an Access application are created and used in locations that are not always connected to each other. A Field of that type holds a Global Unique Identifier for every row that distinguishes it from any other row in the replica set. This functionality is not available in Mendix Platform. Autonumbers with random values are not available in Mendix Domain Model, and neither are Hyperlinks. Fields of the latter Data Type store text that can be either a UNC path or a URL, and can be followed directly. A possible option would be mapping them to a String Field while the possibility for it to act as a Hyperlink could be determined in Mendix GUI via a special widget. OLE Object and Attachment Fields store files. OLE Object Fields can hold exactly one file for each row, while Attachment fields can hold multiple files. Such an Attribute Type is not available in Mendix, although a possible workaround is available. An Entity in Mendix can inherit from System.FileDocument, a predefined Entity that can represent a file. OLE Object and Attachment Fields could be mapped to such an Entity. An Association should also be created between this Entity and the Entity mapped to the original Access Table. The Association would have to be 1-1 for OLE Object Fields and 1-* for Attachment Fields. Calculated fields are calculated based on other Fields of the same Table. Although there is no such 7

9 Attribute Type in Mendix, it is possible to specify that a certain Attribute is calculated and retrieves its values from a Microflow. The Expressions used to calculate those Fields in Access use operators and functions of Access Expression Language. Thus, MS Access Expression Language needs to be mapped to Mendix Microflow Expressions. The difficulty that arises here is that certain functions and operators in Access do not have an equivalent in Mendix. In Appendix B we present a detailed mapping of the two expression languages operators. The mapping of functions is not provided due to their very large number (over 100) and limitations of time. Lookup Fields are not a separate Data Type although it appears as such in Access. A Field defined as Lookup has an actual Data Type of Text or Number and retrieves its values either from a Table Column, a Query Column or a user-defined List. In the first two cases we have in essence a Foreign Key Field to another Table Column. However, in the case of a user-defined List, we have an extra construct with data that also needs to be migrated. We distinguish two cases: One column value list: A similar construct exists in Mendix Domain Model, the Enumeration. The latter holds a List of user defined values and can be assigned to an Attribute. Multi column value list: A Field of that type retrieves its values from a user-defined multicolumn List. Multi-column Enumerations are not available in the Domain Model. Multi-value Lookup Fields: Fields defined as Lookup may be set to allow Multiple Values. This means that for a single row, the Field can hold multiple values from the Table/Query/List that it retrieves values from. This is equivalent to a Many-To-Many Relationship, where the implementation of the middle Table is hidden from the user. In Mendix Domain Model, Attributes cannot store multiple values for a single object Field & Table Properties A Table Field in Access has a number of Properties that define its characteristics or aspects of its behavior. Certain properties are relevant to the Data Model, for example the Field Size property that determines the length of characters of Text Fields. However, there are also properties that specify GUI related settings and are not related to the database schema. For the schema migration process we are interested only in those properties that are related to Access data model. In Appendix A we present a detailed list of all Properties available for Access Fields and Tables and possible mappings to Mendix Platform Primary Keys In relational databases a Primary Key consists of one or more Fields that uniquely identify each row of a Table. The concept of Primary Keys is not present in Mendix Domain Model. The issue of unique identifiers for each row is dealt internally in the physical database. A unique identifier column is created for every table when the actual database is created and its values are filled automatically. It is possible though to preserve the uniqueness of values for Primary Key columns when they are migrated to Mendix. The Domain Model offers the possibility to set Validation Rules for the Attributes of each Entity. Via such a Validation Rule, Attributes that correspond to Primary Key columns can be set as Unique. We note that these rules cannot be applied to a combination of Attributes. Thus, the case of multi-column Primary Keys cannot be properly mapped. 8

10 2.1.5 Relationships & Foreign Keys In relational databases there is often the need to refer to an object of a Table from a different Table. This is achieved with the use of Foreign Key Relationships. A Foreign Key is a Field that retrieves its values from a Primary Key field. Foreign Keys In Mendix Domain Model, Associations relate Entities in a similar way that Foreign Key Relationships relate Tables in Access. However, the concepts of Primary and Foreign Keys are not present. In Access Relationships the Foreign Key Field of a Relationship points to the Primary Key Field. In the Domain Model Associations are not defined over Attributes, but Entities themselves. We illustrate the difference in Figures 2.1 and 2.2 below. Consider an Access Database holding information about Customers and Orders, with the following schema: Figure 2.1: Foreign Key Relationship in MS Access The equivalent Domain Model for that schema in Mendix Platform would be the following: Figure 2.2: Association in Mendix We can see that Foreign Key Fields are not necessary in Mendix Domain Model for the implementation of Associations. Types of Relationships The following types of Relationships are available in MS Access: One-To-Many Relationship: a row in Table A can have many matching rows in Table B, but a row in Table B has only one matching row in Table A. One-To-One Relationship: each row in Table A can have only one matching row in Table B. Similarly, each row in Table B can have only one matching row in Table A. 9

11 Many-To-Many Relationship: A row in Table A can have multiple matching rows in Table B, and a row in Table B can have multiple matching rows in Table A. This type of relationship requires the definition of a third table. The same types of Associations with the same semantics are available in Mendix Domain Model as well. The difference is that the Many-To-Many Association is not implemented with the use of an extra (middle) Entity. Referential Integrity Referential Integrity is one of the most important constraints in relational databases. It requires every value of a Foreign Key Field to exist as a value of the Field that it references. This constraint is important for the consistency of data, however it is one that is not always enforced in Access Relationships. Three options are possible regarding the enforcement of Referential Integrity in Access: Referential Integrity is enforced without Cascade Deletes/Updates: In this case users cannot enter an invalid Foreign Key value. Deleting referenced records is prohibited, as is updating a referenced Primary Key value. Referential Integrity is enforced with Cascade Deletes/Updates: Invalid Foreign Key values are prohibited. The deletion of referenced records is possible and it is cascaded to the referencing records as well. The same applies for the update of referenced Primary Key values. Referential Integrity is not enforced: Users are not prohibited from entering invalid Foreign Key values, deleting referenced records, or updating referenced Primary Key values. We can see that there are three actions that affect the consistency of data and are affected by the enforcement of Referential Integrity. Those are: a) entering Foreign Key values b) deleting referenced records c) updating referenced Primary Key values. The latter is not applicable in the Domain Model since an Association does not make use of Primary Keys. Entering (valid or invalid) foreign key values is an action equivalent with entering an associated/unassociated object in Mendix, while the deletion of referenced rows in Access is equivalent with the deletion of Mendix objects. In Mendix Platform it is not possible to enter an object with a non-existing reference. It is possible, however, to enter an object with no reference at all. Finally, the deletion of referenced objects can be determined via Associations settings, and the options available correspond to the behaviors that exist in Access, as described before Indexes Indexes are used in Access in order to speed up sorting or searching by a particular Field. The definition of single or multi-column Index is possible. In Mendix Domain Model Indexes can be defined on Attributes, and they can also be single or multi column.this results in the creation of an Index in the underlying database when the model is executed Underspecification Underspecification is a common problem in many software systems. An important constraint that is often missing is the explicit definition of Foreign Key Relationships. The database consists of a set of Tables with columns and there is often no knowledge about how these refer to each other. 10

12 Microsoft Access is often used for development from people that do not have extensive knowledge on software or database design. This idea is clearly expressed by the motto used in Microsoft s website for Access: Create custom apps fast without being a developer. Applications are often built from people that lack the necessary knowledge, and as a result many applications lack an explicit definition of Relationships. Additionally, Microsoft s Access option for Lookup Columns allows for the creation of relationships that are not explicitly defined. Consider the Access schema that we presented in Subsection with Customers and Orders Tables. The CustomerID column of the Table Orders is a Foreign Key Field to the CustomerID column of the Customers Table. Instead of defining it this way, one can do the following: Define the Query Customers Extended as presented in Listing SELECT CustomerID, FirstName, LastName, FirstName + LastName 2 FROM Customers Listing 2.1: SQL Query: Customers Extended Afterwards, define the CustomersID field of table Orders as a Lookup Field, retrieving values from the CustomerID column of Customers Extended Query. 1 SELECT CustomerID 2 FROM Customers Extended Listing 2.2: SQL Query for Lookup Field This Field will retrieve values from the CustomerID field of Customers Table. The Field is explicit in the Query, and it would be possible to retrieve it with parsing of both SQL Queries. However, we consider it as a form of underspecification because no Relationship is explicitly created between the two fields: it is not visible in the Relationships window of Access, it cannot be retrieved as such with available tools and no Referential Integrity can be enforced on it. 2.2 Data Migration Once the Access database schema is converted to Mendix Domain Model, the next step is the migration of data. Data migration is handled by a process referred to as Extract-Transform-Load and requires three steps [HCHH08]: First, the data are extracted from the legacy database. Second, they are transformed in such a way that their structure matches the target schema. Finally, they are loaded to the new database. This process relies on the mapping between the source and target database schema. Different databases may be used as back-end in Mendix. The most common case is that an application is deployed on Mendix Cloud, where PostgreSQL is used. It is also possible for a user to configure their own server, like Oracle or MS SQL Server Finally, a Mendix application may be deployed locally for the users and developers to test and the built-in HSQL database is used then Different source-target schema The databases used as back-end are all relational, and the mapping between the Domain Model elements and the physical database elements is the following: For each Entity of the Domain Model, a Table is created in the database, while Entity Attributes correspond to Table Columns. Additionally, 11

13 an extra Column is created with a unique identifier for each row. For each Association, a separate Table is created with Columns the unique identifiers of the associated Tables. Consider again the following Access database schema: Figure 2.3: MS Access Schema - Customers & Orders And the corresponding Domain Model for that schema: Figure 2.4: Domain Model - Customers & Orders In the images below we present the tables created for the two Entities, Customer and Order, and the Association between them, in the built-in HSQL database: Figure 2.5: Customer Table in Mendix built-in HSQL Database Figure 2.6: Order Table in Mendix built-in HSQL Database Figure 2.7: Table for the CustomerOrder Association in Mendix built-in HSQL database The target database that is going to be generated in Mendix will have a different schema than the source database. First, additional Columns are created in every Table that stores an Entity, holding 12

14 unique identifier values for each row. Separate extra Tables are created for every Association of the Domain Model. Additionally, since Foreign Key Fields are not necessary for Associations in Mendix Domain Model, they could be omitted. Finally, Attachment and OLE Object Fields do not have equivalent Attributes in Mendix, but they can be mapped to a separate Entity. For Access applications that make use of such Fields, the target database will have extra Tables as a result of mapping those Fields Data Quality Data is a significant resource in all organizations. Activities and important decisions in companies are based on data and information obtained from data analysis [SSPA + 12]. Thus, its quality is critical. Sidi et al. [SSPA + 12] present an overview of different dimensions that affect data s quality. One such dimension is the consistency of data, which refers to the violation of semantic rules defined over a set of data [BCFM09]. In section we presented the constraint of Referential Integrity and the impact that its absence can have on the consistency of data. We consider again the example with Customer and Order Tables: If referential integrity is not enforced in the Relationship between them, the users may enter Orders with a Customer reference that does not exist. The CustomerID field of the Customer table may be updated for a certain referenced row, with no warning from Access, and no consequent update in the Orders that reference it. Additionally, Customers with existing Orders may be deleted, which will result in rows in Orders Tables with an invalid Customer reference. An additional feature of Microsoft Access also allows for inconsistent data. In section we presented Lookup Fields in Access, that retrieve values either from a Table, a Query or a user-defined List. Via a property available for those fields, it is possible to specify that the values entered are not limited to the List. In this case, users may enter values that do not exist in those Fields data source, and thus refer to a non-existent object. Finally, data quality problems may arise at instance level as well [SSPA + 12]. For instance, redundant duplicates entered by the users of the software system. For those reasons, data cleaning may be a necessary step before the migration of data. 2.3 Queries Migration Queries are Microsoft Access Objects that are used in order to retrieve or operate on data, or define data structures. Access uses Structured Query Language (SQL) as a query language. Queries can be used in various locations in an Access application: they can be a data source for a Form, a Report, a GUI widget, or a Lookup Field. They may also be used in Macros or called from the application code. A single query may be used in more than one location. In Mendix Business Modeler there are two constructs that perform retrieval and operations on data: Microflows and Object Query Language (OQL). OQL is a query language similar to SQL that retrieves data from Entity objects and tt can be used as a data source in Mendix Reports only. Microflows can perform a series of actions and data retrieval is one of them. They can be used as a data source in Mendix Pages, although this use is not very common. Thus, the optimal mapping for Access SQL is not immediately apparent. The set of statements available in SQL can be divided into the Data Definition Language (DDL) and the Data Manipulation Language (DML). The Data Definition Language consists of the set of statements for defining the database schema. With Access SQL it is possible to define, alter or delete the following constructs: Tables, Users, Indexes, Views, and Procedures. In Mendix Business Modeler, Entities, Indexes and Users are created in the graphical editor of Mendix Business Modeler. There is 13

15 no language or other construct for the automatic creation of these objects. We will therefore explore a mapping for Queries that constitute the Data Manipulation Language Mapping Queries to Microflows In Mendix Business Modeler, Microflows are constructs that can perform a series of Actions. Those Actions include retrieval of Entity objects, creation, deletion or change of Entity objects. Microflows accept parameters, they can perform Actions in Loops and may return a result after their execution. Their execution is possible with a button call from Mendix Pages, within another Microflow, and they can additionally be used as a data source in widgets of Mendix Pages SQL Action Queries Action Queries perform changes on data. There are three types of Action Queries, for deleting, updating and inserting data respectively. In Table 2.3 we present the three SQL statements that belong in Action Queries and a possible mapping to Mendix Microflows. Microsoft Access Query DELETE Statement DELETE [table.*] FROM table WHERE criteria INSERT INTO Statement Single-record append query INSERT INTO target [(field1 [, field2 [, ]])] VALUES (value1 [, value2 [, ]) Multiple-record append query INSERT INTO target [(field1 [, field2 [, ]])] [IN externaldatabase] SELECT [source.]field1 [, field2 [, ] FROM tableexpression UPDATE Statement UPDATE table SET newvalue WHERE criteria; Mendix Microflow Microflow Action: Retrieve a list of Entity objects Map criteria to XPath constraint Action: Delete list of Entity objects For single-record append query Microflow Action: Create an object of a specific Entity Set values for its Attributes Commit object to Database For multiple-record append query The mapping depends on the complexity of the Query. See information later on this chapter. Microflow Action: Retrieve a list of Entity objects Map criteria to XPath constraint Action: In a Loop, set new values for the Attributes of each Entity object Table 2.3: SQL Action Queries - Mendix Microflows The mapping of an SQL Action Query to a Microflow may or may not be possible depending on the features that it includes. Specifically, we note two issues: a) Mapping the criteria in WHERE clause to Mendix XPath b) Mapping Queries with nested SELECT statements, like the multi-record append query in Table 2.3. We elaborate on those in the next two subsections. 14

16 WHERE clause to Mendix XPath The WHERE clause specifies the criteria that records must satisfy to be included in the Query results. The criteria have the form of an expression, which can be a comparison statement, an expression including Access special operators, a function, or an exists statement. In Microflows such criteria can be determined with an XPath constraint. Mendix XPath is a query language used in Mendix Modeler in order retrieve data from Entity objects, their Attributes or Associations. Only the constraint part of the XPath query is used in a Microflow, and we therefore need to investigate if SQL expressions can be mapped to XPath expressions. SQL expressions in WHERE clause may perform comparisons, for example checking if a Table Column has a certain value. These expressions are linked by logical operators and may also include arithmetic operations as well. In Appendix C we present a list of Comparison, Special, Logical and Arithmetic Operators that are used in SQL and their equivalent in Mendix XPath. For most of them have there is an equivalence in Mendix XPath. However, we note mismatches in some cases, like the Like and In operators. Criteria that use such operators cannot be translated to XPath. An additional difficulty is migrating Queries that use functions in the WHERE clause. Over 100 VBA functions are available in Access SQL. Due to time constraints and the big number of functions, we do not present a detailed list of mapping for those functions. However, in XPath constraints the functions available are only 18. Thus, for many of those a mapping is not possible. Finally, a SELECT query may be used in a WHERE clause, either as part of a comparison or in a Subquery. Investigating the mapping of SELECT queries to XPath Queries would be the next step in this process. However, an XPath Query cannot be included in the XPath constraint part of Microflows. Since this mapping is not possible, we did not investigate this any further. Nested SELECT statements Multi-record append queries use a SELECT statement in order to retrieve the data that are going to be inserted in a certain Table. The possibility to map such a Query depends on the features that it uses. Certain features can be mapped to Microflows, but there are also cases where the mapping is not possible. We discuss the mapping of SELECT statements to Microflows in the next subsection. This analysis applies here as well SQL Select Queries The syntax of a SELECT query is the following: SELECT [predicate] { table. [table.]field1 [AS alias1] [, [table.] field2 [AS alias2] [,...]]} FROM tableexpression [,...] [IN externaldatabase] [WHERE...] [GROUP BY...] [HAVING...] [ORDER BY...] [WITH OWNERACCESS OPTION] Listing 2.3: SQL Select Statement The arguments in square brackets are optional. A SELECT statement may return columns of one or multiple tables, by performing a JOIN operation in the FROM clause. In Microflows an XPath constraint can be used to retrieve objects for which an Association exists. This is equivalent with an INNER JOIN in SQL. The operations of LEFT and RIGHT JOIN cannot be performed in a Microflow. Additionally, a Microflow can return objects of only one Entity. Thus, for Queries that 15

17 return objects from multiple tables a mapping to a Microflow is not possible even if an INNER JOIN is used. We can see that a big number of features is available in a SELECT query. The mapping of the WHERE clause to Mendix XPath constraint was examined in the previous section for Action queries, and it applies here as well. The possibility to migrate this clause depends on the operators and functions used in the Query. The ORDER BY clause is a feature that is available in Microflows. However, the GROUP BY clause provides functionality that cannot be performed in Microflows. Although it is possible to aggregate objects in Mendix Microflows, grouping results by a certain column the way it is performed in SQL is not possible. Finally, a SELECT query may perform operations among Table columns and return those in the results as well. However, a Microflow cannot return additional columns along with an Entity s Attributes. The features [IN externaldatabase] and [WITH OWNERACCESSOPTION] are not relevant with our research, as they concern retrieval of data from external databases and multi-user environments and are outside of our scope Mapping Queries to OQL Object Query Language (OQL) is a relational query language used in Mendix Platform in order to retrieve data from the database. OQL is very similar to SQL, and it uses Entity and Association names instead of the actual database table names. It supports SELECT queries only. In Listing 2.3 we presented the syntax for an SQL Select statement. Below we give the syntax of an OQL Select statement, where the arguments in square brackets are optional: SELECT [ DISTINCT ] { { entity name from alias }. { expression [ [ AS ] column alias ] } [,... n ] } FROM { entity name ( sub oql query ) } [ [ AS ] from alias ] { { INNER { { LEFT RIGHT FULL } [ OUTER ] } } JOIN entity name [ [ AS ] from alias ] ON <constraint> } [,... n] [WHERE <constraint>] [GROUP BY expression [,...n ]] [HAVING <constraint>] [ORDER BY { order by expression [ ASC DESC ] } ] [LIMIT number ] [ OFFSET number ] Listing 2.4: OQL Select Statement The SELECT clause specifies which Entity Attributes are returned by the query. Attributes from multiple Entities may be return as well as expressions like in SQL. Aliases and predicates are used in both languages. We give a detailed list of predicates supported in both languages in Appendix D. We note a mismatch with the [TOP n [PERCENT]] and DISTINCT predicates, which are not supported in OQL. The FROM clause specifies the Table(s) or Query(-ies) that the data is retrieved from. Similarly with SQL, OQL can retrieve data from Entities or other OQL subqueries. In the latter case, the subquery 16

18 should be included within the main query, as it is not possible to reference an OQL Subquery by name as in SQL. The types of JOINS available are the same in both languages. However, in OQL a JOIN may be performed on Entities that are associated. In SQL, it is possible to perform a JOIN on Fields that do not have a Primary-Foreign Key Relationship. The WHERE clause of SQL specifies the criteria that records must satisfy in order to be included in the query result. As we already saw, the expression in the WHERE clause may perform a comparison using Table columns, and it may include Subqueries and VBA functions. In Appendix D we present a mapping from SQL to OQL operators. Most of them are the same in both languages. The operators ANY, ALL and SOME that are used with Subqueries in SQL are not available in OQL. The ORDER BY and GROUP BY clauses are used in both languages with the same functionality. The only difference is the Aggregate functions supported, which are presented in Appendix D as well. 2.4 Overview of our approach In this project, we will focus on Schema Migration of Access applications to Mendix Domain Model. We will create a description of mappings between Access and Mendix schema elements, and based on that we will implement a tool that performs an automated schema migration. Additionally, since many real-world databases lack an explicit definition of Foreign Keys, we will try to recover those constraints by examining the data for inclusion dependencies. The following two chapters describe details of our approach, the solution given, and its limitations. 17

19 Chapter 3 Access2Mendix: Access Schema Migration After examining the schema elements of the two systems we implemented a tool, Access2Mendix, for migrating the database schema of MS Access applications to Mendix Domain Model. In this chapter we present a description of the schema mappings and details of our implementation. In the end, we perform an evaluation of our tool and discuss Access2Mendix s possibilities and limitations. 3.1 Solution: Schema Mappings In section 2.1 we investigated the similarities and differences between the schema elements of the two systems, Access and Mendix Modeler. As Hainaut et al. note in [HCHH08], schema conversion produces a formal description of the mapping between the objects of the source and target system. In this section we present those mappings. In Table 3.1 we can see the mappings of the main Access schema elements. Tables are mapped to Domain Model Entities, while Table Fields are mapped to either Attributes or an Entity and Association. The latter case applies for Fields whose Data Type is OLE Object or Attachment. Relationships are mapped to Associations and Indexes to Mendix Indexes. MS Access Schema Elements Mendix Domain Model Elements Tables Entities Fields Attributes OR Entity + Association Relationships Associations Indexes Indexes Table 3.1: MS Access - Mendix : Mapping of main schema elements Fields are mapped according to their Data Type, as presented in Table 3.2: 18

20 MS Access Field Data Types Text Memo Number Byte Integer Long Integer Single Double Replication ID Decimal Date/Time Currency Autonumber Mendix Domain Model Attribute Types String String Integer Integer Long Integer Float Float Not mapped Float Date and Time Currency Long Integer Increment Random Replication ID Yes/No Hyperlink OLE Object Attachment Calculated Lookup Field - One Column Value List Lookup Field - Multi Column Value List Multi-value Lookup Field One Column Value List Multi-value Lookup Field Multi Column Value List Autonumber Autonumber Not mapped Boolean String Entity that inherits from System.FileDocument 1-1 Association with the Entity mapped to the Table this field belongs to Entity that inherits from System.FileDocument 1-* Association with the Entity mapped to the Table this field belongs to Set Attribute as Calculated and create a Microflow Attribute of Type String or Number, according to the actual Data Type Enumeration assigned to Attribute Entity with Attributes the Columns of the Value List 1-1 Association with the Entity mapped to the Table this field belongs to Entity with an Attribute holding the List values *-* Association with the Entity mapped to the Table this field belongs to Entity with Attributes the Columns of the Value List *-* Association with the Entity mapped to the Table this field belongs to Table 3.2: MS Access - Mendix Data Type Mappings 19

21 Fields of type Replication IDs are not migrated. In case that a Microsoft Access application is part of a replica set, only one copy of the replica set will be used for the migration. Since replication is not supported in Mendix, Fields of that type would not be useful and thus we chose to omit them. We decided to map Random Autonumbers from MS Access to incremental Autonumbers in Mendix. The random Autonumber values could be replaced with incremental ones in data migration. Extra care should be taken if those Fields are part of a Foreign Key Relationship. In that case, one should also correctly replace and migrate the referenced values as well. Regarding the mapping of OLE Objects and Attachment Fields to a separate Entity and Association, we also have to correctly map the Delete Behavior of that Association. Consider a Table Customer that includes an Attachment Field named Picture. When a row is deleted from Customers Table in Access, the contents of Picture are deleted as well. In the migrated Domain Model, Customer and Picture are represented by separate Entities. In order to simulate the delete behavior of a row in Access, we specified the Delete Behavior of such an Association as On Delete of a Customer Object, Delete Picture as well. and On Delete of Picture object, keep Customer object(s).. Fields that are Calculated are mapped to an Attribute in Mendix with Type the actual Data Type of the Field. Additionally, they are set as Calculated and a Microflow is created and assigned to the Attribute. This Microflow performs the Action of creating a new Variable. The initial value of this Variable is set as the expression from Access, and then returned to the Attribute. In order for those Fields to be migrated correctly, the Expression Language of Access needs to be mapped to Microflow Expressions. The two languages have some differences in the operators and the functions they use. Due to time limitations, we did not perform that mapping. We performed a mapping from Access Field Names to Attribute Names and transfered the rest of the expression as is. As a result, some Microflows may contain errors that will have to be manually corrected, where it is possible. Finally, we were not able to map Multi-value Lookup Fields because of limitations in the API we used. In order to correctly migrate those fields to Mendix we need to know what their data source is, the number of columns of the data source and the values themselves. Our API did not provide methods for retrieving this information for Multi-Value Fields. Thus, their migration is a task that must be done manually by the user, according to Table 3.2. Field and Table Properties in Access determine their characteristics or behavior. As we discussed in certain properties are related to the Data Model while others determine GUI related settings. We migrated only the properties that are relevant to the Data Model, and we present those in the following Table: MS Access Field Properties Mendix Properties Default Value Default Value Required Validation Rule of Required for the relevant Attribute Field Size Max length Expression (for Calculated Fields) Initial Value in the variable of the Microflow Table 3.3: MS Access - Mendix : Mapping of Field Properties In order to preserve the uniqueness of values for Attributes that correspond to Primary Key columns, we implemented in Mendix a Validation Rule of Unique for them. This was not possible for multicolumn Primary Keys, as Mendix Validation Rules can be applied to a single Attribute only. 20

22 MS Access Primary Keys Mendix Unary Primary Keys Validation Rule of Unique for the relevant Attribute Multi-column Primary Keys Not mapped Table 3.4: MS Access - Mendix : Mapping of Primary Keys Relationships are mapped to Associations according to their type, as presented in Table 3.5. MS Access Relationships 1-1 Relationship 1-* Relationship Mendix Associations 1-1 Association Foreign Key Fields omitted 1-* Association Foreign Key Fields omitted Table 3.5: MS Access - Mendix : Mapping of Relationships A many-to-many Relationship is implemented in Access with the use of a third (middle) Table. Each of the two related Tables has a one-to-many relationship with the middle Table. The Primary Key of the middle Table consists of two fields: each of them is a Foreign Key to one related Table. A many-to-many Association in Mendix does not require the use of a middle Entity and one could think omitting this Table. However, apart from the Foreign Keys to the related Tables, this middle Table may contain extra Fields. Thus, we decided to migrate it. Finally, we specified the Delete Behavior of Mendix Associations based on Referential Integrity settings of Access Relationships. The mappings are presented in Table 3.6. MS Access Referential Integrity Referential Integrity not enforced Referential Integrity enforced, with Cascade Deletes Referential Integrity enforced, without Cascade Deletes Mendix Associations - Delete Behavior On Delete of Object A Keep Object B On Delete of Object A Delete Object B Delete Object A only if it is not associated with B Objects Table 3.6: MS Access - Mendix : Mapping of Delete Behavior 3.2 Implementation Access2Mendix was implemented using Java and C#. The implementation was part of Mendix Modeler, which allowed us to reuse existing C# code for the generation of the Domain Model. Java was used for the retrieval of MS Access schema, since the library Jackcess provided an easy to understand and use interface. We describe in this section the algorithms we used for migrating the Access schema elements and also details of our implementation Algorithms The main constructs of an Access schema are Tables, Fields, Relationships and Indexes. Our tool iterates through all Tables and their Fields in order to generate the migrated Domain Model in Mendix. 21

23 Afterwards, it iterates through the Relationships in order to create the necessary Associations. The generation of the Domain Model is described in Algorithms 1 and 2. Data: Tables, Fields and Indexes of Access Schema Result: Entities, Attributes and Indexes in Mendix Domain Model foreach Table t do create Entity e; foreach Field f in Table t do if f is OLE Oject then create Entity entfile that inherits from System.FileDocument; create 1-1 Association between entfile and e; end else if f is Attachment then create Entity entfile that inherits from System.FileDocument; create 1-* Association between entfile and e; end else create Attribute attr; add attr to e; if f is Calculated then create Microflow m; assign m to attr; end end end foreach Index in Table t do create Index i; end end Algorithm 1: Algorithm to create Domain Model Entities, Attributes and Indexes Data: Relationships of Access Schema Result: Associations in Mendix Domain Model foreach Relationship r do if Type of r is 1-1 then create 1-1 Association; end else create 1-* Association; end if Referential Integrity is enforced with Cascade Deletes then On Delete of Object A Delete Object B; end else if Referential Integrity is enforced without Cascade Deletes then Delete Object A only if it is not associated with B Objects; end else On Delete of Object A Keep Object B; end end Algorithm 2: Algorithm to create Domain Model Associations 22