SyncTool for InterSystems Caché and Ensemble.

SyncTool for InterSystems Caché and Ensemble.

Table of contents Introduction...4 Definitions...4 System requirements...4 Installation...5 How to use SyncTool...5 Configuration...5 Example for Group objects by...9 Studio integration...10 The daily work...11 Synchronization history...12 How does it work in detail...14 The SyncTool API...15 How to extend SyncTool...17 Define format...17 Select source objects...18 Settings of File type manager...19 Further information...19 Known Bugs...19 FAQ...20 Contributors...20 Developers...20 Testers...20

Introduction This document and the software described here is for software developers who realize their projects with InterSystems Caché and products built hereon (Ensemble, etc...). Basic knowledge of a source control software is needed for a better understanding of this document. If in the following just the word Caché is alluded of course all products built on it are meant by it. Caché saves it's sources of routines and classes in it's database. That makes it difficult to use a mainstream source control software that works on a file basis. If one wants to use a source control software with Caché there are two possibilities: You buy a source control software that runs natively on Caché. The range and functionality is small here and you have to pay high license fees. One uses a mainstream product. These products have nothing to do with Caché and that leads to exporting the sources out of the Caché database. SyncTool supports software developers in using an external source control software. In most cases it does export and import sources out of and in to a Caché database automatically. This feature makes it possible to use an external source control software of all kind. You can use a commercial or an open source product. Each Caché developer or a development team can use a source control software of their choice regardless of fore-mentioned problems. I do have many years of experience in using CVS with Caché. Based on this experience I designed SyncTool to be extendable. Users can create extensions to implement own file formats (File Type Managers) and use SyncTool for other synchronization tasks. Read on to find out more. SyncTool is OpenSource software and is published under the Apache License. Further details in: http://www.apache.org/licenses/license-2.0 If you extend SyncTool or correct errors I'd be pleased to get your changes so I can put them into main SyncTool development. This procedure ensures that you profit on your own and my developments and on the developments of other contributors. Read on to see how to install and use SyncTool and to understand how it works. Definitions Caché Object database from InterSystems Source object Abstract term for something that is saved in the Caché database. Primarily this term means Classes, Routines and persistent Objects. System requirements SyncTool runs on Caché 2008.2 or higher and it was successfully tested on the Windows and Linux operating system. It should run on other Unixes also (like MacOS X). OpenVMS was not tested so far because of the lack of a test system.

Installation If you used an older version of SyncTool it is recommended to remove it prior to an upgrade. Please run the following in the Caché Console: Do $system.obj.uncompile("synctool.*") Do $system.obj.delete("synctool.*") The SyncTool sources are deployed in a XML file. Please import this file via Studio and compile the imported classes or use the following to import and compile from the command line: Do $system.obj.load("path_to_xml_file/synctool-x.y.xml","ck") How to use SyncTool Configuration Please create a new empty directory somewhere. This will be the working directory. Make sure that Caché has write permissions on this directory. Export files from the source control software to this directory if needed. Point your browser to this URL: http://server-ip/csp/namespace/synctool.gui.main.cls You will get the following dialog:

Click on the Search button and choose the working directory from above. If you want to compile the source objects on an import please choose Default in the Compile manager field. The setting Keep backups for... manages if the source objects will be exported to a backup area before the appropriate files are imported. Please enter a number of days for which the backups will be kept. That means SyncTool will delete the backup data automatically after the entered number of days. Please leave the field blank if SyncTool should never delete the backup files. If you enter 0 no backups will be made. You can overwrite this setting in each module. It is highly recommend to activate this setting. With the parameter Keep history for... you can setup how long SyncTool should keep the synchronization event history. The parameters works the same as Keep backups for.... The event history can be called via the Working directory/history... menu. Activate Synchronize meta data to export Source Selections of each module into a XML file of a separate (hidden) module. You can manage those files with your source control software so all colleagues work with the same Source Selection and File Type Manager settings. Please click Working directory/save in the menu to save the working directory settings. You will see a new module entry called SyncTool. This is a module to manage the SyncTool's sources itself. If you wish you can check in the files of this module into your source control software and ensure the correct deployment of SyncTool to all your colleagues. SyncTool automatically imports the files in this module (if they are changed) before any operation, to make sure to work with the best and newest SyncTool version. If you don't like that just delete this module. Now you are ready to create subdirectories and the files contained there. The so called modules. Please click the New module button. Type the name of the subdirectory in the appropriate field. You could use foo or foo/bar. If you leave this field blank the files will be exported directly into the working directory. Please choose the File type manager. The File type manager manages which file formats to use for which source objects. Please see the following table for details: Type Source objects File extension Description XML files for classes, routines and globals (Default) Flat files for routines XML files for persistent objects Classes, all routines, globals All routines except OBJ Objects of XML enabled persistent.xml Depending on routine type:.int,.mac,.inc,.bas,.mvb,.mvi.xml The well known XML format of Caché. You should use this Type by default. Routines are exported to flat files. The file name is the routine name. Good choice for exporting

For SyncTool source only SyncTool meta data classes. settings/parameters to XML files. The SyncTool itself.xml Synchronization of SyncTool sources. Selections of source objects.xml SyncTool uses this File Type Manager to deploy the source selections to your colleagues via the source control software. Use Backup source objects to define per module if source objects should be backuped before a file import. The Backup subdirectory is the subdirectory for backups activated with the setting Backup source objects. The default subdirectory is.backup.this is a hidden directory on Windows OS. A leading dot in the directory name hides the directory on a UNIX OS. The file names in that directory have following definition:.#prefixoforiginalfilename.date_time.suffixoforiginalfilename The date is in the format YYYYMMDD and time in hhmmss. Here is an example: Original filename: Backup filename: My.Class.xml.#My.Class.20090615_074530.xml Use the Namespace field to synchronize source objects of a foreign namespace with the current working directory. You will need this feature if you can't install SyncTool in a specific namespace (Ex: %SYS). Click Save module to save the module settings. You select source objects in the Source selection field. Dependent on the source objects type you selected by the File Type Manager you will get different dialogs. Click the Edit button to call this dialog. Here is what such a dialog looks like for the File Type Manager XML files for classes, routines and globals (Default) :

Please type in the column Selection one or more source selections. You can use an asterisk (*) as a joker character. Type the correct suffix for the source objects to define their type. Select add or remove in the Selection type column to do the same. Check Preview enabled to see a preview of the selected source objects. You can easily see if you got the right sources. As a default setting package names are contained in the filename on export. Sample: The class User.MyClass will be exported to the file User.MyClass.xml. Activate the checkbox Map packages to subdirectories to make subdirectories out of packages. Sample: The class User.MyClass will be exported to User/MyClass.xml. Please choose the setting as you wish. Click OK to save the settings. The File Type Manager Flat files for routines uses the same source selection dialog as above. That's why I won't describe it here. File Type manager XML files for persistent objects has the following source selection dialog:

Please type a SQL statement in the SQL column and only select the ID (%ID) of the object(s) to export. The column Type shows the class type of the persistent objects you selected with the SQL statement. You can change the type if you like. The Group objects by column specifies which objects belong to which file. Please read the following section for details. Click OK to save the settings. Example for Group objects by Assume the following persistent and XML enabled class: Class User.Colors Extends (%Persistent, %XML.Adaptor) { Property ID As %Integer; Property Color As %String; Property Object As %String; Index IdKey On ID [ IdKey, Unique ]; } There are following objects of this class: ID Color Object 1 Green Grinch 2 Green Joker 3 Blue Ocean 4 Blue Sky Assume further you typed the following source selection for the above class: SQL: Type: Group objects by: select %ID from SQLUser.Colors User.Colors (empty)

The objects are automatically grouped by the %ID property if nothing is explicitly given. This has an effect on the file names and the objects contained within the files. In that case you will get these four files: User.Colors.1.xml User.Colors.2.xml User.Colors.3.xml User.Colors.4.xml You can consider the naming convention of the files here. The file name always contains the class name concatenated with a dot and the value of the Group objects by property. It's the ID in that case. This is the content of such a file (User.Colors.1.xml): <?xml version="1.0" encoding="utf-8"?> <Objects> <User.Colors ID="1"> <Color>Green</Color> <Object>Grinch</Object> </User.Colors> </Objects> Assume you used the Color property in Group objects by. Then you will get these two files: User.Colors.Blue.xml User.Colors.Green.xml The content of User.Colors.Blue.xml is like this: <?xml version="1.0" encoding="utf-8"?> <Objects> <User.Colors ID="3"> <Color>Blue</Color> <Object>Ocean</Object> </User.Colors > <User.Colors ID="4"> <Color>Blue</Color> <Object>Sky</Object> </User.Colors > </Objects> Studio integration SyncTool comes with a Studio integration. You can activate it in the System Management Portal (SMP). In Caché 2008.2.x start the SMP and go to Configuration/Studio Source Control Settings. Select the correct namespace and check SyncTool.Studio. Click OK. This dialog may be found at another location on higher Caché versions. It could look like this:

Restart Studio to force reread this setting. Now each time you create, save, delete or compile a source object within Studio it will be exported to the file system as long it is managed by SyncTool. If external files are edited (ex. manually or by an external Source Control Software) they are automatically imported into the Caché database if you open the appropriate source object in Studio. Another useful feature is a new menu within Studio. You can easily open the SyncTool GUI from Studio. Please notice: The Studio integration does not work for source objects resided in an other namespace. The daily work If you like to use a source control software or use SyncTool for other synchronization tasks the daily work is easily described like this: Open the SyncTool GUI (directly via the browser or from the Studio menu), open the working directory and click the Synchronize menu entry. When SyncTool is finished make your work in the source control software (commit, update, etc...). Go back to SyncTool and click Synchronize again. Finished! It is highly recommended to use the Studio integration. If you only synchronize source objects edited in Studio, the Studio integration ensures the export synchronization and the above first Synchronize task can be omitted. That means after editing classes and routines in Studio you can directly start with the source control tasks. But please run the Synchronize task afterwards to import the files changed by the source control software and bring the Caché database into a consistent state. The imported source objects are compiled automatically if you chose to use a Compile Manager. There can be collisions in some cases. A collision in the sense of SyncTool is a file that needs to be imported but one of the corresponded source objects needs an export. In that case SyncTool does nothing but needs a decision by the user. If such a case occurs you will be prompted the following dialog at the end of the synchronization:

Make your decision in the Keep column. Click the left radio button to keep (import) the file or click the right radio button to keep the source objects in the database (export them). Highlight the middle radio button to do nothing with the file. The file will appear on the collision list at the next synchronization. The decision what file to export or import can be eased by displaying differences between the external file and the internal source objects visually. Just click on the diff button in the last column. The visuall diff requires GNU diffutils to be installed. Diff has be known by the system by the PATH environment variable. Synchronization history Each event on a file is logged. The log it is called history here can be opened by the Working directory/history... menu. You will get the following dialog:

The column Task started displays the start time stamp of each synchronization task. That means, if there were 30 files involved in one synchronization task they will all get the same Task started time stamp. The column Action date/time shows the time stamp of the export or import of each file. In the column Action you see what happened to the file and the column Known displays if the file is known by SyncTool. Unknown files are just imported and can be manually exported. You see some controls above the columns. These are filters. With that filters you can for example examine a specific synchronization task or a specific file. How does it work in detail This section describes in detail how SyncTool works. On each synchronization task SyncTool tries to determine for each managed file what to do. There are only two possibilities: To import the file or to export the appropriate source objects to that file. SyncTool implements that behavior by creating managing data for each file and source object. (These are objects of type SyncTool.File and SyncTool.SrcObject.) To determine if a file changed, SyncTool saves the update time stamp of the file and compares it with the update time stamp of the file in the next synchronization. If they are different, the file was changed and has to be imported. Each file has at least one source object (in the case of classes and routines each file has exactly one class or one routine). Some File Type Managers (ex: for persistent classes) allow to save more than one source object within a file. An export is done if at least one source object of the file was changed. SyncTool determines the change of a source object in multiple stages. In the first stage it tries to get the update time stamp of each source object. If this information is not available it tries to get a version number of the object. If one of this information exists and they differ from the last

synchronization all source objects belonging to this file need an export. If none of this information exists SyncTool works with hashes. The hash will be created on the last export of a file. On the next synchronization the source objects are exported to a temporary file. A hash of this file is calculated and compared with the saved hash. If the hashes differ then obviously one of the belonging source objects has been changed and they will be exported. If such a file is imported, then the source objects are exported to a temporary file again and this file is hashed and will be saved for the next synchronization. If SyncTool determines that a file needs an export as well as an import then this is a collision as described above. The SyncTool API There are often situations where the synchronization as described above needs to be called programatically. This could be the integration with a build system or the implementation of an automatic commit of files in a source control system or other scenarios. How ever. SyncTool can achieve that. The following statements always reference an instance of a working directory. To open an instance you will need its ID. If you want to find out the id you can run this SQL query: select %ID, WorkingDirectory from SyncTool.WorkingDirectory Use the determined ID to open an instance of the working directory: Set WD=##class(SyncTool.WorkingDirectory).%OpenId(1) To synchronize the sources within the defined working directory with the file system run: Set Status=WD.SyncParallelized() You're highly advised to check the returned status of each called method. If an error is found you could display the error message or do something else. But do something! If 'Status Do $system.status.displayerror(status) Let's assume you want to synchronize the working directory with the ID 1. The complete sample would look like this: Set WD=##class(SyncTool.WorkingDirectory).%OpenId(1) Set Status=WD.SyncParallelized() If 'Status Do $system.status.displayerror(status) The sample above in that or similar form is adequate for most tasks. However there could be some special tasks. They are implemented like this: If you'd just like to import changed files, run: Set Status=WD.RefreshSelectedObjects() If Status Set Status=WD.ImportDir(,.ImportedObjects) If 'Status Do $system.status.displayerror(status) In the parameter ImportedObjects you will get an one dimensional array with the names of the imported source objects. If you'd like to import all files regardless if they are changed or not just call: Set Status=WD.RefreshSelectedObjects()

If Status Set Status=WD.ImportDir(0,.ImportedObjects) If 'Status Do $system.status.displayerror(status) To export changed source objects call: Set Status=WD.RefreshSelectedObjects() If Status Set Status=WD.Export() If 'Status Do $system.status.displayerror(status) There are similar API calls for modules. That means you can synchronize, export or import individual modules via API calls. Please look at the class documentation of SyncTool.Module for further details.

How to extend SyncTool One important goal in the development of SyncTool was the ability to extend it. SyncTool can synchronize any source object with the file system. This is realized by File type managers. One can define and implement own File type managers. SyncTool comes already with important file type managers. Define format If there are source objects that are not managed by SyncTool itself you can extend the class SyncTool.ExportImportAbstract (or one of the subclasses) and implement a new File type manager. There are some methods that have to be implemented. Here they are: Method Export Import ObjectExists Filename2Objectname ImportDeleted or DeleteSources Description Exports given source objects to a file. Imports given file and returns the names of imported source objects. Returns whether a given source object exists in the database. Converts a filename to an array of source object names. If your File type manager manages exactly one source object per file and this source object is deleted on import then please implement the ImportDeleted method. Otherwise implement DeleteSources. The most important methods are Export() and Import(). You define the file format with these two methods. Please overwrite the parameter DISPLAYNAME. Give your File type manager a human readable name here. It will be used in the GUI. There are some other methods you can implement, depending on what you want to achieve: Overwrite GetObjectDateModified(), if your source objects contain the date modified information. It's the same for GetObjectVersion(). If your objects contain a version number please overwrite this method. It's sufficient to implement one of these methods. If your File type manager manages many source objects in a single file, please implement the method Objectname2Filename(). Source objects that belong to one file must return the same file name. If your File type manager manages exactly one source object per file please overwrite this method if you have a special naming convention for your files. It's recommended to have a naming convention for the files so that one could suggest the name(s) of the source object(s) on the file name. Please implement ExportNeeded() if your algorithm should be different from the described above. GetImportableFileMask() should be overwritten, if you would like to import a subset of files in a direcotry. It is recommended to give each File type manager a unique file extension.

Other files than the managed one could stay in the module directory and would not bother the File type manager on importing files. If GetImportableFileMask() is not specific enough to select importable files you can implement GetFilesResultSet(). In this method you can prepare an adequate Caché class query and return the resultset. This query can follow complex logic to select files to import by your File type manager. The returned Query must contain the same fields as the query FileSet of class File. Overwrite ImportNeeded() if you wish an other recognition algorithm for importing file as described above. In the method GetFileExtension() one can implement a differing behaviour for determining the file extension of a given filename. If your File type manager recognizes the necessity to export a file by hashes (like described above). Then you can overwrite the Hash() method and implement a different hash algorithm as used by SyncTool. Select source objects All methods are described now that are needed to implement the file format and ensures proper export and import of data. But, how does SyncTool know which source objects to manage? The simplest is just to implement the method GetManageableSources() in your File type manager. This method simply returns an one dimensional array with the names of source objects in the first dimension. The better possibility is more complex but allows to select source objects via a GUI. Several steps need to be taken: 1. Subclass SyncTool.SourceSelection (or one of the subclasses). 1.1. Define in this new class some properties you need to save the selection of the source objects. 1.2. Overwrite ExpandSelection(). This Method must expand the actual source selection to an one dimensional array of source object names. That means this method primarily interprets some joker characters. 2. Create a new ZEN page subclassed from SyncTool.GUI.SourceSelection() (or one of the subclasses). 2.1. Implement in this class the GUI (Contents block) as you like and create methods to save, read and delete source selections. Save the source selections in the class you created in step 1. 2.2. Overwrite the method Summary(). This method should return a short summary of the source selection in HTML format. 2.3. Overwrite in the class created in step 1 the parameter EDITCLASS and set it to

the name of this class (from step 2). 3. Overwrite in the File type manager class the parameter SOURCESELECTIONCLASS and set it to the class from step 1. Now you are able to easily make the source selections of your special source objects. Settings of File type manager If you need to save some settings belonging to your fresh File type manager, please use a special possibility for that. Subclass FileTypeManagerSettings in a new class. Define some properties in this new class. The class FileTypeManagerSettings connects your settings to the module and the File type manager and makes the settings synchronizable as meta data of SyncTool. In your code please open a new or an existing instance of the settings class with the method GetFileTypeManagerSettings() of the class SyncTool.Module. There are samples of how to do this in the class SyncTool.GUI.StudioObjectExportImport, method SaveSelection(). A sample of subclassing FileTypeManagerSettings is the class SyncTool.ExportImportSettings. Further information A documentation like this is in many cases not enough to write extensions for a software. Most developers wish to have sample code. Luckily SyncTool is OpenSource software what means you have the source code. If you write your own File type manager please have a look at the class documentation of SyncTool.ExportImportAbstract for further information and the description of methods and parameters. As samples for File type managers you can look at the contained classes SyncTool.ExportImport, SyncTool.RoutineExportImport and SyncTool.ObjectExportImport. You can see some techniques in these classes you could use for your own implementations. As samples for source selections are used the classes SyncTool.StudioObjectSelection and the corresponding GUI class SyncTool.GUI.StudioObjectSelection. Other sample classes are SyncTool.ObjectSelection and SyncTool.GUI.ObjectSelection. Known Bugs 1. If you use the Studio integration of SyncTool, you will encounter error messages on using the importing functionality of Studio. I've checked this problem intensively and came to the conclusion, it is a bug in Caché. I've reported that to InterSystems under WRC #724146. 2. If you work with Caché 2008.2.5 or higher within 2008.2 SyncTool won't work. Several error messages are thrown by SyncTool. The reason is a bug in those Caché versions. InterSystems has fixed this bug under the AdHoc #8886. Recompile SyncTool classes after applying this adhoc.

FAQ 1. Is SyncTool able to export classes in a format as seen in Studio? No. Caché does not natively support the export in this format. A new File type manager has to be written to achieve that. Contributors Special thanks for contributing visible improvements on SyncTool go to these people: Developers Alexander Riemer Testers Albert Ojrich Henning Jahn Mark Molloy