NEW YORK FILEMAKER DEVELOPERS' GROUP Site Home > New York FileMaker Developers' Group > FileMaker Damaged Files, Corruption, & Recover FileMaker Damaged Files, Corruption, & Recover Home Damaged Files and Hidden Corruption in FileMaker Pro (A Guide to Recover) Posted 10/13/2003 08:59 AM... Updated 1/5/2007 Gregory Durniak Printer Friendly Version: [link] The following information is based on my experience as a FileMaker developer, FileMaker's TechInfo Knowledge Base, and on numerous discussions with FileMaker Tech Support The "Hidden" Corruption Problem: FileMaker Pro, and FileMaker Server will open, run, and backup files that are corrupt. In the last five years, I have seen this happen four times: 1. a file that crashes only when two guests are on the same list layout, and a script is triggered to set a number field. This file ran for a full year before the corruption was discovered ( All the backups were corrupted ). FileMaker Tech Support was unable to help. 2. a file that has an annoying pause after any user action, e.g. clicking a button, finding records, etc. This file is much too large to rebuild. 3. a file that runs fine, until you save as clone, and you find that three layouts are missing in the clone copy. This file did not crash. It was damaged when it was copied to an IMac with a faulty hard drive. This file ran for a week before the corruption was discovered. 4. a file with over 190,000 records, that suddenly displays "Records: 0", and appears blank, even though these records are still visible in a related portal. I understand that files can be corrupted by power surges, and by bad hard drives, and can NOT be prevented. What I don't understand is why FileMaker's utilities do not detect it. When damaged files run, regular backups and "master" clones are useless. There is no backup strategy that works, because there is no way to be sure that the files you are backing up are OK. This problem is actually documented in FileMaker Answer ID 2943 "Strategies for Protecting Files Whose Structures are as Important as Their Data" [link] "... Any time you make changes to your file s structure... you will need to save a new master clone. Place your new master clone in a safe place... At this point, you have a clone with no records, but you may also have a file structure that is possibly damaged due to corruption carried over from the original file..." This problem is also noted in "FileMaker Server Best Practices" (p.13) http://www.filemaker.com/downloads/pdf/fms_best_practices.pdf "Any system failure causing FileMaker Server to shutdown inappropriately could result in corrupted files... Even if the files seem to re-open but have to go through a consistency check or recovery, some corruption could be buried in the file" What are your chances of getting "hidden" corruption? The four failures above are the result of five different organizations, each running an average of 24 files, for 5 years, using FileMaker's best practices. Versions include FileMaker 4 thru 6, on OS 9.x and Windows 2000. Removing Hidden Corruption: FileMaker's "File Recovery Service" will attempt to restore both data and structure (Answer ID 1633) [link], but there are no guarantees, and a 3-6 week wait Upgrading to a new FileMaker file format may help. I was able to "fix" one of my hidden corruption files, by upgrading an entire solution from FileMaker 4 to 5. Therefore, upgrading to FileMaker 7 or 8 is worth a try ( but not as easy as upgrading to FM5 ). In the FileMaker 7 Tech Brief "Migrating Existing Solutions" Page 1 sur 5
http://www.filemaker.com/downloads/pdf/techbrief_fm7_migration.pdf FileMaker claims that "the conversion process does not propagate corruption that may have existed in a prior version", but goes on to say that "On very unusual occasions, a damaged field or script or layout element may be lost". Now all we need is a conversion process that does not propagate corruption that may exist in the SAME version. FileMaker folklore is filled with "sure-fire" methods for dealing with corruption, e.g. "export data to a text file, import back into a recovered clone, then save a compressed copy". These methods apply (often repeatedly) recover, save as clone, and save as compressed in various combinations, and may, or may not, work. FileMaker's "official" method to recover and rebuild files is described in Answer ID 1580 "How To Organize The Recovery Process When Recovering Related Databases" [link]. If your files are not stable, this is always worth a try. It will clean up your data and structure, just keep in mind that it may not remove the corruption, and may instead remove unexpected portions of your files. Also, for the export data step, you can use a "merge" file, as explained below. It's easier. Removing Data Corruption: The files above were running with hidden structure damage. The data was fine. Data corruption is actually more common, and much easier to remove. Export your text, number, date, and time fields to a "merge" (text) file, check the text export for "illegal" ASCII characters per Answer ID 949 [link], then import back into a clone, using "matching field names". (NOTE: This Answer ID applies to FM6, and may not be a concern in FM7,8) Be sure to check that any repeating fields import properly. No need to export calculation, or summary fields. Container fields must be restored separately. Removing Corrupt Records: If FileMaker crashes during data export, or while browsing, there may be a corrupt record. See FileMaker Answer ID 4666 "Removing Corrupt Records From a File" [link] Removing a Damaged Layout: Terms: Recover: If a layout object is damaged, it may be difficult to open that layout in order to delete it. See FileMaker Answer ID 4704 "Deleting a Damaged Layout" [link] FileMaker uses the terms damage and corruption interchangeably, as I do here. Some Developers prefer to use "damaged" for a file that will not open, and "corrupt" for a file that runs, but is not stable. Unfortunately, FileMaker uses the terms data and structure interchangeably, e.g. "The underlying action of Consistency check and Recover is to preserve as much of the data as possible. In this context, data generically refers to tables, records, layouts, scripts, and field definitions" or "Saving a compressed copy... not only reclaims unused space in the file, it also rebuilds the file s structure". For this article, I use structure to mean everything left in a clone, and data to mean everything removed in a clone. The "Recover" command will not remove hidden corruption if it does not recognize the hidden damage as "bad". The true nature of "Recover" is described in Answer ID 2944 "Tips on Backup and File Recovery" [link] "The primary goal of the Recover command is to retrieve the good parts of the file, resulting in a good working file. Corrupted portions of the file are often eliminated from the recovered file, not fixed. If the recovered file is missing significant portions of your database, one option is to send the database to FileMaker, Inc.... to attempt a more complete recovery of the file"... and in Answer ID 4426 "Corrupt / Damaged Files" [link] "Because the recovery process removes structures that may harbor corruption, you may not want to use recovery for routine maintenance. If you do, examine the recovered file carefully to be sure that all objects are intact; objects that are possibly suspicious will be removed" The myth that Recover "repairs" damage goes back to FileMaker 2.0. FileMaker 2.0 "Help" did state that recover "creates a repaired copy of a damaged file" ( A quick lunchtime survey at a recent FileMaker Developer Conference showed more than half still thought recover to be a "repair" utility ). Page 2 sur 5
To help end the myth, FileMaker should rename the "Recover" command, perhaps to "Please open this damaged file, even though I probably shouldn't use it anymore" Unfortunately, FileMaker has reverted to saying Recover will "repair" again in Answer ID 312 "A File Crashes During The Recovery Process" [link] "In most cases the recovery process will repair your file to the point where you will be able to use it or access it again". A long lost description of the Recovery process is reproduced here [link], from Article 102516 "What Happens During Recovery?" (No longer in the TechInfo Knowledge Base) File Recovery (FM4, 5, 6) on Windows will remove Font Formatting of Fields on Layouts (Answer ID 2523) [link]... "When performing a recovery in FileMaker 4.0 and above under the Windows Operating System, the font formatting information for the file is stripped out, thus leaving all fonts defaulted to Arial. This is a normal part of the file recovery and cannot be 'turned off'. The recovery feature in FileMaker Pro is intended for damaged files only and is not to be used as a 'tune up' for your file(s)." File Recovery (FM8v1, v2) may remove the contents of global container fields, and may damage variable names in calculations (Answer ID 5907) [link] Recover's closing dialog is not ( and never was ) reliable. It may, or may not report the fact that bytes have been removed ( even in FM8 ) Index Problems: Tech Specs: Unfortunately, FileMaker still recommends using "Recover" to re-index multiple fields, in Answer ID 5685 "Auto-Enter Calculation Field May lose its Index" [link]. We need a separate re-index / rebuild utility. Make sure you haven't exceeded FileMaker's Technical Specifications, e.g. by defining 5,900 fields in FM5. The file may work, but appear to be corrupted Technical Specifications Of FileMaker Pro 5.0 (Answer ID 1657) [link] Technical Specifications Of FileMaker Pro 7 (Answer ID 569) [link] Additional Notes: For info on routine file maintenance, see Answer ID 4426 "How to Avoid the Need for Recovery" [link] The FileMaker Pro Consistency Check is briefly described in Article 103745 (Removed from the Knowledge Base), which is reproduced here [link] Saving a compressed copy will not remove hidden corruption, but it is recommended for routine maintenance in FileMaker Answer ID 4426 "Corrupt / Damaged Files: How to Avoid the Need for Recovery" [link] "Saving a compressed copy rewrites the entire database, fitting as much data into each block as is possible. This procedure not only reclaims unused space in the file, it also rebuilds the file s structure". This option has been renamed "save as compacted copy (smaller)" in FileMaker 7, and is better defined in Answer ID 656 "Save A Copy As Options" [link] "... only the deleted items from a file are removed when a "smaller" version of the file is saved" Page 3 sur 5
FileMaker 7 According to FileMaker's Customer Support Services, two common causes of hidden corruption are damage that occurs to an image pasted on a layout, and damage that occurs to printer instructions stored in a "print setup" script step. These are not verified by FileMaker's utilities. If the problem can be isolated to a single object ( script, layout, etc. ) try re-building that item in the corrupted file Please do not confuse the risk of file corruption with overall stability. FileMaker Pro is very stable software. Database Files do not need to crash to become corrupted. Corruption is fickle. Files crashed & trashed for years may run fine, yet a brand new file might be unstable (I have seen this happen) ANY file type can get corrupted, FileMaker 7, Excel, or even Outlook Express: http://www.microsoft.com/windows/ie/community/columns/filecorruption.mspx Hidden corruption may slow down FileMaker overall, as the engine tries to interpret instructions that no longer make sense. Rebuilding a single damaged file can dramatically improve the performance of your entire solution (I have seen this happen). However, it may be difficult to figure out which file to rebuild. With no reliable utility to detect damage, finding corruption is mostly a trial and error process. FileMaker's backup strategy falls apart in Answer ID 2943 "Strategies for Protecting Files Whose Structures are as Important as Their Data" [link] by saying that a master clone can be "possibly damaged due to corruption carried over from the original file. This clone needs to be recovered". This is wrong, and misleading. Recover is not intended for routine file maintenance. Whenever possible, revert to a backup copy. Avoid using a file that has crashed, or been recovered. FileMaker has posted Tech Info Answer ID 5317 "How To Handle A Corrupt FileMaker File", specifically for FileMaker 7 [link] and Answer ID 5421 "What to do when your file is corrupt" [link], which pretty much repeats what's already in Answer ID 5317. There is no indication from FileMaker that the utilities in FileMaker 7 have in any way been improved. FileMaker 7 "Help" states that "The underlying action of Consistency check and Recover is to preserve as much of the data as possible... These utilities do not guarantee that the file has been completely repaired" and goes on to say, as before, that "The Recover command aggressively attempts to correct a file so you can open it and recover your data. To do this, the Recover process may delete corrupt fields, layouts, layout objects, scripts, and data. For this reason, you should only use the Recover command when you cannot open a file. Do not use this command for regular file maintenance" Although heavily marketed as "uncorruptible", FileMaker 7 was ( and is ) probably the worst of all versions FileMaker 8 / 8.5 For file repair options, see Answer ID 5371 "Consistency Check When Opening Files In FileMaker Server 8", if problems are found [link] and Answer ID 5421 "What to do when your file is corrupt" [link] It appears that "Recover" in FM8 may have moved up from "last" resort, to "2nd" resort. However, it is still not clear what this "kinder, gentler" recover actually does, or does not do. Be careful. At the 2006 FileMaker Developers Conference, Recover was (finally) moved back to "last" resort. FileMaker (again) advised that we should "avoid using a file that required recovery" FM8 Recover has been found to (silently) remove the contents of global container fields (I have seen this happen), and may damage variable names in calculations (Answer ID 5907) [link] Page 4 sur 5
FM8 "Help" states that "In general, recovering a file should be reserved for files that will not open, or are displaying index problems. Databases that are returning records incorrectly from a find should be fixed by recover... and... "The Recover command makes an aggressive attempt to reopen a damaged file. It is intended for data recovery, not file repair. Do not use the Recover command as part of routine file maintenance. These statements are contradictory. see also "Differences Between File Maintenance Options and Save a Copy As Options" Answer ID 5594 [link] What I would like to see: 1. A "verify" utility, to help identitfy files that are corrupt ( not to certify that the file is "OK", just a warning if parts of the file no longer make sense ) 2. An effort by FileMaker to collect damaged files that run, to see how they get past the consistency check and recover utilities 3. An improved "Save as Clone" feature, with an option to verify the structure, instead of just truncating the tables 4. A whitepaper on FileMaker's Utilities, written for developers, explaining the functions and limitations of the consistency check, the recover command, save as clone, and save as compressed 5. Separate Menu Options for Recover's 3 basic functions: "Check File (Read Only)", "Re-Index", and "Rebuild (Remove Damaged)" Tips on Re-Building a Damaged File: Every item in FileMaker is given a "hidden" serial ID number, based on creation order. In order to maintain the links between related files, the ID numbers used in the re-built file must be the same as the original. Use the Design Functions FieldIDs, FieldNames, ScriptIDs, ScriptNames, and ValueListIDs, ValueListNames (in FileMaker 6 and higher) to show the ID's used. The ID's of Layouts, and Relationships are not as important, since they have no external references. To preserve the hidden ID's, create all Fields, Scripts, and Value Lists in their original creation order, including previously deleted items. Names of deleted items are not important. Resist the urge to remove "deleted" items in the re-built file, since they may turn out to be in use, if the original ID reference was corrupt (I have seen this happen). Avoid importing scripts, or copying layout objects. They may be corrupt. There is no documentation to say if this will propagate corruption. Avoid importing data directly from the original file. Export to a text file first, as explained above. To save typing, print all of your fields and scripts to a PDF file, then copy paste the text definitions into the re-built file. This is particularly useful for long IF statements, and calculations. For this to work, all Field and Relationship names must be the same as the original. Copyright Gregory Durniak info@fileshoppe.com http://www.fileshoppe.com/ Printer Friendly Version: [link] Posted at Jun 12/2004 11:58 PM: Ilyse Kazar btw, exporting data to text & reimporting does not always remove data corruption. you need to open the text export in a good text editor & examine for the presence of low-ascii "control" characters & upper-ascii char's as well. Keep low-ascii's that are tabs, end-of-record returns, vertical tabs, and group separators (repeating field delimiters). REMOVE all other low-ascii's, they cause crashes. Ilyse Kazar kazar@datatude.net Thanks!... this note has been added above... Gregory Durniak Back to Page Page 5 sur 5