A Database Re-engineering Workbench A project proposal by Anmol Sharma Abstract Data is not always available in the best form for processing, it is often provided in poor format or in a poor quality data model. This can impact performance, usability and raise numerous other negative issues. Companies require many databases to aid in the information systems sector; they have old databases referred to as legacy systems which need upgrading periodically. The conversion process of these systems/databases can be costly for a large company. The aim of this project is to produce a workbench capable of re-engineering databases. The workbench should allow for a visual mapping of the database relational model using a drag-and-drop graphical user interface (GUI). The user should be able to produce a new database according to what the user specifies as the conversion preferences such as new data relationships. The project contains several key areas, such as research, development, testing, as well as a written report. 1. Introduction Data is not always available in the best form for processing, it is often provided in poor format or in a poor quality data model. The database schema, the blueprint of how the database is constructed, can have multiple bugs, be over complicated, complex and even be of awkward design. This can impact performance, usability and raise numerous other negative issues. Databases are often used in scenarios where thousands of users will require access concurrently, therefore the database needs to able to perform at its maximum capabilities, and this requires a good quality data model. Companies require many databases to aid in the information systems sector; they have old databases often referred to as legacy systems. These systems need to be upgraded to cope with the new standard of databases along with the new improved versions of the database management systems (DBMS). The conversion process of the database for use with the new DBMS can be costly for a large company due to the size of the database. With evolving technology, software and changing business needs, a company needs to periodically engage in database re-engineering. This is where a database re-engineering workbench can be used to minimise costs and promote ease in the transfer of the database and the conversion process in general. It can help in the maintenance, evolution and quality control of databases. The aim of this project is to produce a workbench capable of re-engineering databases, involving both reverse and forward engineering. Reverse engineering is required to break down the old database into manageable parts, this is for easy manipulation. Forward engineering is then required to create the new database. The workbench will allow for a visual mapping of the database relational model utilising the database s schema. This will be achieved using a graphical user interface (GUI) which will showcase the information found in the old database s schema. The GUI will provide a drag-and-drop interface from the old schema into the new schema, which on completion would generate a new database. The user will be able to produce a new database according to what the user specifies as the conversion preferences such as new data definition or data relationships. - 1 -
The report contains a background into the other database re-engineering workbench solutions currently available, comparing the solutions and establishing some general base requirements of the workbench. The proposed project section details how the workbench can be developed, its features to be implemented, how the proposed solution can be tested and finally the design method to be used. The programme of work showcases the overall development schedule including a Gantt chart showing the breakdown of tasks. Finally, the required resources section outlines the required hardware or software and the references section has all references used throughout the report. 2. Background Relational Databases have multiple tables which feature attributes along with their individual domain properties and overall the blueprint of how the tables are related. All of these aspects of a database are stored into a database schema. The schema of a database is in a formal language which is understood by a DBMS. The schema holds information crucial for a good quality data model such as the Primary Key and Foreign Key. The two keys are used to introduce relationships between the tables in a database. The schema is what is used to reverse engineer a database and further to import properties to help forward engineer a resulting database. Traditional applications, such as MS Access, used to manipulate databases do not offer functionality such as that described above, therefore other independent solutions are required to provide the functionality. Through research it is obvious that there are not many solutions available which are the same as the proposed solution. An application called ER/Studio produced by the company embarcadero [1] offers many of the described functionality. It can be used to operate on databases across multiple platforms and is used by clients which wish to manipulate data such as data architects. The software allows the user to create and manage database designs, Figure 1 - Logical Modelling of a database in ER/Studio. document and reuse data assets. The application does offer full reverse and forward engineering of databases by the tarditional methos of viewing the databases schema but the application has limitations and numerous drawbacks. The application has a complicated user interface (UI) which can help promote human error. The UI is over encumbered with features and panels with additional functionality all around the database layout in the centre of the application; this can be noticed in Figure 1. While the application does boast about its multiple services, it lacks simple automation of features such as specifying a new primary key of a database which would then spawn a new relationship table. This can greatly improve the quality of a database, therefore very crucial. The quality is improved as the data is divided further into smaller manageable tables which will promote faster searching. There are multiple plugins, applications and other miscellaneous methods which enable the reverse engineering of databases, however there are no applications, other than the one previously described, available which offer forward engineering leading on from the derived result of the reverse engineering. A plugin produced for an application called Astah [2] provides concise functionality of reverse engineering a database and providing a logical mapping of the discovered - 2 -
schema details. The application is however only available for MAC and the plugin only provides visual reverse engineering in the form of an Entity- Relationship (ER) diagram, the user cannot in any way manipulate the schema in the database. Figure 2 showcases an example of the plugin in operation; a database has been converted into an ER diagram. DBMS vendors do offer instructions on how to reverse and forward engineer databases in the form Figure 2 - The plugin shown in the Astah application. of guides[3] and tutorials[4]. These however oppose the purpose of the proposed solution as they do not provide an easy-to-use interface or any type of automation, which is essential in being cost-effective, promoting reliability and performance. These factors are essential for any company, in this case, very essential when reducing the time a system is offline while upgrades occurs such as the legacy systems previously described. The aim of the project is to provide an automated, simplistic and easy-to-use workbench which can provide the functionality of creating a new database from an old database taking into account the users defined preferences. 3. The Proposed Project The overall outcome of this project is to produce a workbench capable of re-engineering databases, involving both reverse and forward engineering. The workbench should allow for a visual mapping of the database relational model utilising the database s schema; using a graphical user interface (GUI) which will showcase the information found in old databases schema. The GUI should provide a dragand-drop interface from the old schema into the new schema, which on completion would generate a new database. The user should be able to produce a new database according to what the user specifies as the conversion preferences such as new data relationships. 3.1 Aims and Objectives The main solution will need to deliver the following features described below: Reverse Engineering the capabilities to examine a databases schema and gather all important information such as tables, each table s attributes and their relationships. Forward Engineering the ability to derive a new database using the previous databases data and schema. Drag-and-drop GUI An easy-to-use interface which consists of a window partitioned into two parts, the left part showcasing the old databases schema information and the right part available for the user to drag-and-drop in the schema information from the left. The application in terms of aesthetics will be simplistic and only feature a small amount of buttons. The user interface most not over encumber the screen with options as seen in previously researched similar programs. Referential Integrity Checking The program must be able to check the databases, whether it be the old database or the newly produced database for any breaches of referential integrity. This will help improve the quality of the data model. - 3 -
3.2 Initial Steps The key aspects described below will need to be initially carried out to aid in the future development of the workbench: Application Programming Interface (API) first of all, a formal language will be required to build the foundation for the overall workbench. This will most likely be an extension of the SQL language and will involve introducing automation of certain steps, such as creating new tables and copying a previous tables content to the new table. o This will not be a new programming language such as Java or C, but simply a highlevel language designed more than likely in Java which when executed will call upon multiple low-level SQL code to complete the task. o The language will mimic the style and format of SQL statements. A simple GUI this is solely to compliment the API described above. This would consist of a simple window which requests text input from the user such as the developed API methods and language. The steps above need to be initially carried out to first develop core functionality of the workbench. The main solution can then extend the developed API. 3.3 Programming Language The language to use to develop the workbench also needs to be considered carefully as particular languages have their advantages to the project and also their own disadvantages. The main language under consideration is Java. Java features core functionality such as the JDBC API[5] which will aid in providing communication between the database and the language/application. JDBC would further utilise the MySQL[6] or SQLite[7] component; more than likely SQLite as it provides easier database handling in comparison to MySQL. 3.4 Testing The process of testing will occur in multiple stages of the development of the workbench, it will occur throughout the project. Testing of the solution can be carried out in multiple ways, the solution itself clearly will have to be tested but so will the result of using the solution. The interface will have to be tested to ensure it is simple to use and easy to understand by the user. The features and functionality will have to be tested at every progressive step of the development of the workbench to ensure correct function and that it solves the problem of the user suitably. The different tests described above could be carried out using surveys or by running a small trial; which would involve asking the occupants to carry out a small number of tasks which then would be analysed to reach a conclusion. The resulting database from using the workbench will also need to be tested for validity and accuracy. This can be done quantitatively by measuring the success rate and calculating a percentage of how often there are inaccuracies or inconsistencies. E.g. there often was inaccurate data and inconsistencies in the data less than 10% of the time. On the other hand, the test may have to be carried out manually, in terms of manually assessing the resulting database to check if the output was correct. This will yield accurate results however will be time-consuming and very unlikely chosen as the testing style due to the short time period of the project. - 4 -
3.5 Expansion Possibilities The main solution, if there is time available and the core requirements described above are completed, may also deliver the following features: Entity-Relationship Diagram Modelling to provide a visually appealing breakdown of the database schemas and the database s table relationships. Domain Integrity Checking ensure the data values inside the database, inside each table follow the defined rules for values, range, and format. 4. Programme of Work The programme of work is as follows: Workbench Testing Testing, such as trials or surveys, will be carried out throughout the project to test the multiple stages such as the implemented API, the GUI and finally the integrated final workbench. This will help discover bugs and other faults in the implementations. During the initial research period, testing will occur with the research found to assess the importance in relation with the project. Assess prior research Before the design and implementation process can begin, any previous implementations will need to be thoroughly assessed and all possible routes of implementation explored in order to find the best suited to the workbench. Design API The API will be designed and outlined taking into consideration possible future expansion. An abstract architecture will be considered. Implement API The API, once designed, will then be created incorporating SQL. Design GUI The GUI will be outlined taking into consideration the simple and easy-to-use design objectives. Implement GUI The GUI will then be implemented standalone taking into consideration of the future integration. Research API integration Multiple way of integrating the two parts together will be assessed by comparing the negatives and positives of each way taking into account possible future expansion. Integrate API into GUI Finally the two major components of the workbench will be integrated. Finalise workbench and finish report The figure below (Figure 3) shows the programme of work in a Gantt chart. Figure 3 - The schedule of the project. - 5 -
5. Resources Required Programming IDE A programming IDE such as NetBeans[8] to aid in the implementation of the workbench. A database test bed A set database which will be used throughout the project to test the workbench. JAVA Runtime and Development Kit 6. References [1] - Embarcadero Technologies, Inc. (N/A). ER/Studio. Available: http://www.embarcadero.com/products/er-studio. Last accessed 21st May 2014. [2] - Change Vision. (2014). DB Reverse Engineering. Available: http://astah.net/features/dbreverse-plugin. Last accessed 19th May 2014. [3] - Microsoft. (N/A). Reverse engineer an existing database into a database model. Available: http://office.microsoft.com/en-gb/visio-help/reverse-engineer-an-existingdatabase-into-a-database-model-ha010115485.aspx. Last accessed 19th May 2014. [4] - Oracle. (N/A). Forward and Reverse Engineering Models and Working with Logical Model Diagrams, Displays and Subviews. Available: http://www.oracle.com/webfolder/technetwork/tutorials/obe/db/sqldevdm/r30/datamode l2moddm/datamodel2moddm.htm. Last accessed 19th May 2014. [5] Oracle. (N/A). The Java Database Connectivity (JDBC). Available: http://www.oracle.com/technetwork/java/javase/jdbc/index.html. Last accessed 19th May 2014. [6] Oracle. (N/A). MySQL. Available: http://www.mysql.com/. Last accessed 19th May 2014. [7] N/A. (N/A). SQLite. Available: http://www.sqlite.org/. Last accessed 19th May 2014. [8] Oracle Corporation. (N/A). Netbeans. Available: https://netbeans.org/. Last accessed 19th May 2014. - 6 -