Migrating MS Access Database to MySQL database: A Process Documentation

Transcription

1 Migrating MS Access Database to MySQL database: A Process Documentation Juma Lungo November 10, 2004 Introduction A health database from India has over 3,500,000 records in MS Access 2000 database. A Java based GIS application was developed and is required to connect to the MS Access database as its external database. It is further aimed that this GIS application should be open source software. This has resulted into the need for migrating the MS Access database to an open source database, MySQL. This article describes the documentation process of migrating a large MS Access database to MySQL. Supporting Tools In order to migrate the MS Access 2000 database to MySQL, we need to (1) create a MySQL database then (2) import MS Access tables (with data) into the MySQL database. While it is possible to export the MS Access data tables to a text file and import it to the MySQL, this will require creating table features (attributes) from scratch, an extra task. A tool that could support a direct link from MS Access to MySQL could facilitate export of table feature with its data. The supporting tools (software) acquired are MyODBC and Navicat where MyODBC is a MySQL ODBC driver and Navicat is a Visual editor of MySQL client interface. Creating Data Source Install MyODBC driver in the windows machine where the MS Access database is located. Create an ODBC DSN name connecting to the MySQL database as follows (in WinXP): 1. Start Control Panel Performance and Maintenance Administrative Tools Data Source (ODBC) 2. Click Add button Select MySQL ODBC Driver 3. Complete the MySQL DSN configuration form (see Figure 1) appropriately and then Test Your Connection This will create an ODBC data source name on your windows machine which connects to the MySQL database located either in the same machine or in a remote database server. In this case the MySQL database is also located in the same machine where the MS Access 2000 database is located

2 Figure 1: MySQL ODBC DSN Configuration Exporting Data To export the MS Access tables with data to MySQL, open the Access database, use the File Export, to export data as follows: 1. Open the MS Access database 2. Select the table you want to export 3. Select File Export menu 4. Select ODBC Database () option of the Save as Type in the Export Table dialogue 5. Click OK in the Export To dialogue 6. Select your MySQL datasource in the Select Data Source dialogue (Figure 2) 7. Click OK to initiate data transfer This looks very simple but in practice, it is not. When a table is transferred to MySQL it will be recreated with its attributes. Attributes are defined with data types. The problem is that MySQL does not know all the data type implementation of MS Access database. For example, a Number datatype in MS Access is converted into int(11) data type in MySQL. The yes/no Boolean data type of MS Access is converted to tinyint data type in MySQL, etc. MySQL has date and datetime data types, however, all date/time data type in MS Access are converted into datetime data type in MySQL

3 Figure 2: Importing MS Access to MySQL using ODBC data source Pre-processing MS Access database tables With MS Access 2000, a date datatype can be represented with a date/time or a Number data type. If using the later type (Number) you must specify the Field size as Long Integer and Format as Short Date in the field properties of the corresponding table field. In MS Access, working with date/time data type is very risk because it relies on the locale setting of the date format of the user s computer. Thus, while designing Access applications, it is recommended to specify all date fields with Number data type as this will ensure that all users, regardless of the date setting on their local machines, access date values in the same format. The DHIS application has adopted this, and all date fields in the DHIS Data Mart are of Number data type. However, when exported to MySQL, all MS Access number data type will simply be converted to int(11) data type. Thus, the first task is to search all date fields in MS Access database tables defined as Number data type and change them to date/time datatype. After converting the data types of the date fields to date/time, you can simply export the tables to MySQL, everything will be fine. In our case, Table 1 presents all MS Access database tables with date fields

4 Table 1: Date fields in DHIS Data Mart MS Access Table (relation) Date Field DataType ChangeTo DataMonth DataMonth Number Date/time MonthlyData DataPeriod Number Date/time MonthlyDataIndicator ValidFrom Number Date/time ValidTo Number Date/time MonthlyDataOutStandingInputForm DataPeriod Number Date/time MonthlyIndicatorOrgUnit4 DataPeriod Number Date/time MonthlyIndicatorOrgUnit5 DataPeriod Number Date/time OrgUnit DateOpen Number Date/time DateClose Number Date/time OrgUnitData ValidFrom Number Date/time ValidTo Number Date/time OrgUnitStructure DateOpen Number Date/time DateClose Number Date/time MS Access 2000 is a file based table locking database system. Thus, when you are changing field data types in a table which contain data, MS Access is locking all the records in that table. Since MS Access is a file based database system, your Windows system will process the locking process as if it is locking files; in this case each record in that MS Access database table is treated as a file. Since there is a limited number of files that can be locked concurrently, if a database table has many records (in our case, the MonthlyData table has 3,598,902 records), out of memory registry system error will popup. If the maximum number of locks per file was exceeded, you can increase the number by editing a registry entry for the local computer, find the MaxLocksPerFile registry value using the Windows registry editor (regedit.exe) and increase the value. However, this is not a recommended option. If you edit a value in the Windows registry, there is no guarantee that the change will not corrupt the registry and render Windows unusable. Up to this point, we cannot manage to export the MonthlyData table to the MySQL because all the date values of DataPeriod field will be converted into integer when reaching to MySQL database, and we cannot convert that field into date data type because there are too many records to lock. Do you think my computer is not powerful? Stop, I am working on Pentium 4, 1.8GHZ 256MB of RAM with a registry MaxLocksPerFile variable value of 9,500, should I risk increasing it to 3,598,902? No way to convince me! An alternative is to export the data into MS Excel spreadsheet. The DataPeriod field will reach there as a number field and therefore all date will be recorded as number. However, MS Excel knows the secret of the land. To convert the DataPeriod column values to date format do the following: - 4 -

5 1. Select the whole column of DataPeriod 2. Format Cell Date (use English South Africa for the DHIS database) All the values in the DataPeriod column will go back to date again! Then you need to create the MonthlyData table manually in the MySQL database and import the data. The risk here is that, field names of the MonlthlyData table should match exactly as those column names in MS Excel file, otherwise you will have field mismatch import problem. It is easier to save the MS Excel file as Tab delimited Text file and import the text file. However, with visual tools like Navicat, that table could be created on fly while importing just like importing using ODBC. MS Excel Worksheet Row Limit The final challenge is that MS Excel worksheet has a limit. It only accepts 65,535 data rows and 1 row for column heading. When I export the MS Access MonthlyData table with 3,598,902 records to MS Excel it takes the first 65,535 records then it fairly tell me that it cannot take more records in the MS Excel worksheet, see Figure 3. Figure 3: Export Error due to Excel Worksheet Row Limit The solution to this problem is to export the first 65,535 records, then delete them in the source table, then export the next 65,535 records, then delete them in the source, then repeat until all 3,598,902 have been transferred. The question here is how can I count the first 65,535 records? This is terrible! MS Access 2000 support Transact SQL so I can tell the database to just give me the first 65,000 records (my computer clipboard can hold 65,000 records maximum) and export them to MS Excel to format the date field then save it as Tab Delimited Text File to be imported in MySQ database, then delete them from the database. So I just needed to play with the Mouse like a little kid (I had to create 55 text files of 65,000 each). Fine, here are very simple Transact SQL statements I called them top65000 and deletetop65000 respectively. Select first 65,000 records: SELECT TOP [65,000] * FROM MonthlyData; Delete first 65,000 records: DELETE top65000.* FROM top65000; - 5 -

6 top65000 and deletetop65000 queries in the MS Access Database Figure 4: Two queries used to extract data from large tables Importing 55 files to the MySQL database is another tedious task I am not ready to go for. To work around, I merged the files and import them at once. On importing my first file which comprises 8 merged files of 65,000 records each went successful as presented in Figure 5. Time is the limit; consider automating part of the data extraction process. Figure 5: Navicat Screenshot for MySQL Import Report of 520,000 records - 6 -