Introducing Microsoft Sync Framework: Sync Services for File Systems

Introducing Microsoft Sync Framework: Sync Services for File Systems Microsoft Corporation November 2007 Introduction The Microsoft Sync Framework is a comprehensive synchronization platform that enables collaboration and offline scenarios for applications, services and devices. Developers can build sync ecosystems that integrate any application, any type of data, using any protocol over any network. As part of the new framework, we're also working on simplifying the development of end-to-end sync solutions by providing domain-specific components for common scenarios such as synchronizing relational databases, file systems, devices, etc. Towards this goal, we have developed a reusable provider for synchronizing the contents of file system directories on PCs and removable media such as USB thumb drives. In this article, I ll cover the details of this new reusable Sync Services for Files Systems component we call simply, the File System Provider, along with enabled scenarios and sample code for getting started. Overview The File system provider is designed to be a generic, reusable component that can be used for synchronizing files and folders between any NTFS or FAT formatted file volumes. It uses the Sync Framework s powerful metadata model to enable peer-topeer sync of file data with support for arbitrary topologies (client/server, full-mesh, P2P) including support for removable media such as flash drives, USB thumb drives, etc. It enables 3 rd parties to build file synchronization and roaming scenarios into their end-to-end sync solutions without having to worry about directly interacting with the file system. Key features of the File system provider include: Incremental synchronization of changes between two file system locations specified via a local or UNC path. Synchronization of file contents, file and folder names, file timestamps, and attributes. Support for optional filtering of files based on filename/extensions, subdirectories, or file attributes Optional use of file hashes to detect changes to file contents if file timestamps are not reliable Reliable detection of conflicting changes to the same file and automatic resolution of conflicts with a no-data-loss policy Allow for limited user undo operation by optionally allowing file deletes and overwrites to be moved to the Recycle Bin Support for Preview mode which provides a preview of the incremental sync operation without committing changes to the file system

First-class support for the scenario where the user may start synchronization with equal or partially equal file hierarchies on more than one replica. Support for graceful cancellation of an ongoing sync operation such that the remaining changes can be synchronized later without having to re-sync changes that were already synchronized. Note - In this CTP1 release, the File system provider does not sync NTFS security ACLs for files/folders or alternate streams. Also, sync of encrypted files or folders is only supported if the encrypted file or folder is on a local volume. Enabled Scenarios One of the goals of the File system provider design is to enable a broad set of scenarios for synchronizing files and folders. Since the File system provider component is available for use and redistribution through the Sync Framework, ISVs are now able to build file sync functionality into their existing applications or create new applications to enable interesting file and folder sync scenarios. Some of the possible end-user scenarios enabled by the Sync Framework and this component include: Multi-master file sync between multiple PCs: An application can be written to synchronize a user s files between their various PCs. The user can update and manage their files in the most convenient location at any given time and do a bi-directional synchronization of data between various PCs on demand or on an automated schedule to ensure that the latest files are always available to them at all places. E.g., they can work on their laptop PC when they are away from the office but when they connect back up to the office network they can synchronize all changes from the laptop to the office PC and vice versa. The File system provider will ensure that new and updated files are propagated in each direction and renames and deletes are repeated on the other side as well (without causing file data transfer). If there are conflicting changes made to the same file on the two sides, the conflict will be resolved in favor of the copy that was last updated, preventing data loss. Synchronization between PCs using a USB Drive: Sometimes it is not possible to have a direct network connection between different PCs, e.g., a home PC and a work PC. In this scenario, an application can be written to enable synchronization of files between the two PCs via a USB flash drive. The application is installed on both PCs and it allows the user to synchronize their file data between the PC and the USB drive. The USB drive in this case is used as the physical file transport between the work and home location. Taking a network share offline: In an information worker environment many times team members have to work with workgroup documents on a network share. It would be great if files from the network share could be incrementally synchronized to a user s mobile PC or USB flash drive so they can review or modify these offline when they are not connected to the corporate network. The File system provider can be used to develop an application to provide offline access to network shares by synchronizing data from the share to the user s mobile PC, even if the network share is configured to be read-only to a particular user.

Maintaining a backup copy of files: Many times users would like to backup files from one PC to another or from one PC hard disk to another. The File system provider enables rich archiving scenarios by allowing for a one-way synchronization of files from the primary PC or hard-disk to the backup PC or hard-disk. File System Provider Usage To put some of the following material in context, let s start with a short code snippet for how the File system provider can be invoked by an application.. The following is part of a full code sample provided later in this article. The code below is written in C# and uses the.net wrapper classes for the File system provider but the capabilities and patterns for C++ code to invoke the native COM component are very similar. // Copyright (c) Microsoft Corporation. All rights reserved. public static void SyncFileSystemReplicasOneWay( SyncId sourcereplicaid, SyncId destinationreplicaid, string sourcereplicarootpath, string destinationreplicarootpath, FileSyncScopeFilter filter, FileSyncOptions options) FileSyncProvider sourceprovider = null; FileSyncProvider destinationprovider = null; try sourceprovider = new FileSyncProvider( sourcereplicaid, sourcereplicarootpath, filter, options); destinationprovider = new FileSyncProvider( destinationreplicaid, destinationreplicarootpath, filter, options); destinationprovider.appliedchange += new EventHandler<AppliedChangeEventArgs>(OnAppliedChange); destinationprovider.skippedchange += new EventHandler<SkippedChangeEventArgs>(OnSkippedChange); SyncAgent agent = new SyncAgent(); agent.localprovider = sourceprovider; agent.remoteprovider = destinationprovider; agent.direction = SyncDirection.Upload; // Sync source to destination Console.WriteLine("Synchronizing changes to replica: " + destinationprovider.rootdirectorypath); agent.synchronize(); finally // Release resources if (sourceprovider!= null) sourceprovider.dispose(); if (destinationprovider!= null) destinationprovider.dispose();

Detecting Changes on the File System The File system provider synchronizes incremental changes between two file system locations (also defined as a Replica by the Sync Framework). In order to do this, at the start of every sync session, it needs to evaluate incremental changes on each replica since the last synchronization. This process is called change detection. The provider stores a small amount of information (called metadata) which describes where and when the item was changed giving a snapshot of every file and folder in the replica. Changes are detected by comparing the current file metadata with the version last saved in the snapshot. For files, the comparison is done on the file size, file times, file attributes, file name (case-sensitive), and optionally a hash of the file contents. For folders, the comparison is done on folder attributes and folder name (case-sensitive). A heuristic algorithm is used to determine file renames. If a file was only renamed or moved, just that operation will be performed on the other replica, avoiding a full stream transfer, which can be expensive. For folders, a rename or move is currently processed as a delete and create on other replicas. Files inside the renamed folder however will be processed as rename and no unnecessary stream transfers will be done. By default, the File system provider detects changes at the start of every sync session. This behavior can be modified by the application by using initialization flags and the explicit DetectChanges method on the provider. Given the above method of detecting changes (to support the FAT file systems), if there is a large number of files in the replica, change detection may be an expensive operation and should be done only as often as is necessitated by the user experience of the application. Progress Reporting and Preview Mode For interactive applications, the File system provider supports extensive progress reporting during the sync operation. This enables applications to present a progress UI to the user and display the current state of the sync operation. Progress information is reported to the application via a callback interface which the application can optionally supply to the File system provider at initialization time. The progress reporting callback invoked before every change to the file system (OnApplyingChange) also allows the application to skip a given change. The File system provider supports Preview mode synchronization which enables applications to inspect and display what changes would happen upon synchronization. In preview mode, no changes are committed to the file system or the metadata, however most of the progress reporting notifications are still sent out for changes that would be applied, if run without preview. This can be used by the

application to present verification UI to the user with all changes that will be made if the synchronization is executed. Note that the File system provider does not provide an upfront estimate of the total number of files to be synchronized before the sync operation starts because this can be expensive to figure out depending on the size of the file system replica and all applications may not want to pay the price. Applications can, however, collect the same statistics by doing a two-pass approach: first run a Preview mode sync session between the two replicas to collect statistics about the total number of changes, including the number of creates, deletes, updates, etc., and then run the real sync session with the proper progress bar display. Filtering Out Files from Synchronization An application using the File system provider can configure the provider to filter out certain files or folders from the sync operation. For example, the user may want to synchronize everything but media files from a given folder. The File system provider can be initialized with a scope filter which determines the set of files to synchronize. Sync applications can choose to filter files based on file names, sub-directories or file attributes. Filter Type Default Example Filename-based exclusion No files excluded Do NOT synchronize files that match *.mov,*.wmv or foo.txt Filename-based inclusion Subdirectory exclude File attributebased exclusion All files included No sub-folders excluded Nothing excluded based on attributes ONLY synchronize files that match *.doc, *.txt, special.jpg Do NOT synchronize this list of sub-directories that match c:\users\videos. Do NOT synchronize system or hidden files Note that only files/folders that pass ALL the scope filters are included in the sync. Also, applications should ensure that they use the same scope filter for all replicas in the sync community for a consistent user experience and to ensure convergence of synchronized data between all replicas. As mentioned above in the Progress Reporting section, applications can ask the provider to skip certain files via the OnApplyingChange callback method in the File system provider callback interface. Lastly, certain system files used by Windows Explorer are always excluded from the sync, e.g., desktop.ini and thumbs.db (they do need to be marked with both the SYSTEM and HIDDEN attributes for this to happen).

Conflict Handling There are times when users will modify the same file / folder independently on different file system replicas. The next time these replicas are synchronized with each other, the File system provider will attempt to reconcile these conflicting changes in the best possible way. In general, the File system provider conflict resolution follows certain key principles which are determined by desired user experience (these are also principles followed by and enabled by the underlying Sync Framework): - Conflict resolution must be done in a way that guarantees convergence of data across all replicas - Built-in conflict resolution must be deterministic, i.e. the same outcome should apply regardless of which two replicas synchronize the conflicting changes. - Data loss must be avoided in all possible scenarios. In this CTP1 release, the File system provider only supports a fixed policy for automatic resolution of conflicts. The following is the general policy used by the File system provider when automatically reconciling conflicting changes: - Concurrent update conflicts: If an existing file is modified independently on two replicas, the file with the last write time (i.e. updated later) will be preserved on the next synchronization. If the last write times happen to be the same, a deterministic resolution policy will be used, such that the user ends up with the same file contents on all replicas. If Recycle Bin support is requested from the File systems provider, the losing file will be placed in the Recycle Bin on the losing replica so it can be recovered if the user wishes to do so. - Concurrent update/delete conflicts: If an existing file is modified on one replica but the same file or its container is deleted on another replica, the File system provider will always favor the update over the delete. If the path needs to be recreated on the side where the file was deleted, the File system provider will do that as well. - Concurrent child create / parent delete conflicts: Same policy as above is also used for the case where a new file may be created under a given folder on one replica, but the folder may have been deleted on another replica the create will override the folder delete and the file creation will be synchronized to all replicas with the deleted folder getting resurrected.

- Name collision conflicts for folders: Sometimes users may end up creating a folder with the same name on two different replicas. This can also happen when users start synchronizing replicas that already have similar file system hierarchies and content. This can also happen if an existing folder is renamed to a new name which happens to be the same as the name of a new folder on another replica. In all such cases the user will end up with a single folder and the contents of the folder will be merged on the next synchronization. - Name collision conflicts for files: The scenarios mentioned above for folders can also happen with files. In this case however, a different resolution happens depending on the scenario: o Create create collision: In the case where two files were independently created on two replicas with the same name, the resolution on the next sync operation is to preserve a single file with the same name, but with contents from the side which had the later update timestamp. In case the file timestamps are the same, the content is assumed to be the same and one of the file streams is picked as the winner deterministically. If Recycle Bin support is requested from the File systems provider, the losing file will be placed in the Recycle Bin on the losing replica so it can be recovered if the user wishes to do so. o Create rename collision: In the case where a rename of an existing file collides with a create of a file with the same name on a different replica, the resolution is always to keep both files by renaming one of them. This is done because in this case it s always safer to assume that the user intended these two files to have different identities. The file to be renamed is picked deterministically, such that the same resolution will apply regardless of which two replicas sync the conflicting changes. As can be seen there s a lot of intricate logic that needs to be implemented to handle these situations correctly, especially when synchronizing across more than two replicas with an arbitrary sync topology. The good news is that the File system provider handles most of this without the application writer having to worry about it. Error Handling and Recovery The File system provider is designed to gracefully handle single-file errors while synchronizing a set of files. A host of potentially transient errors can happen when trying to read or write files (locked files, file changed after change detection due to concurrent file activity, access denied, not enough space on disk for a given file, etc.). The File system provider handles these by skipping these files during the sync operation such that the rest of the data can be synchronized successfully.

For every file change that s not applied successfully to the destination replica, an application callback method is invoked with the file details and error information. In some cases, the synchronizing application may wish to handle these programmatically and try to re-synchronize again after fixing up the problem, in other cases it can be presented to the user at the end of the synchronization to ensure that they are aware of files that didn t synchronize successfully. It is also possible that the entire sync operation may fail sometimes due to a transient error, in which case, the proper error code will be returned from the Synchronize method call and the application can react accordingly. For example, concurrent sync operations on the same replica are not currently supported trying this will result in a replica in use error code which can be handled by retrying the synchronization after some time. Concurrency and Consistency Guarantee The File system provider is designed to correctly handle local concurrent operations by other applications on files that may be part of an ongoing sync operation. If a local concurrent change has happened on a file after the last change detection pass on the replica, either on the source or the destination, to prevent loss of the concurrent change, any changes to that file will not be synchronized till the next synchronization session (or the next change detection pass, if the application is using explicit change detection). Also, when synchronizing changes to a file s contents, the new file stream will be applied atomically to the destination location to avoid the situation where the user ends up with a partially correct copy of the file. Using the FileSyncProvider Sample Code The following sample code is written in C# for a CLR-based managed code application, however it should be very similar for a native COM/C++ based application. // Copyright (c) Microsoft Corporation. All rights reserved. using System; using System.IO; using Microsoft.Synchronization; using Microsoft.Synchronization.Files; public class FileSyncProviderSample public static void Main(string[] args) if (args.length < 2 string.isnullorempty(args[0]) string.isnullorempty(args[1])!directory.exists(args[0])!directory.exists(args[1]))

Console.WriteLine( "Usage: FileSyncSample [valid directory path 1] [valid directory path 2]"); return; string replica1rootpath = args[0]; string replica2rootpath = args[1]; string idfilename = "filesync.id"; SyncId replica1id = GetReplicaId(Path.Combine(args[0], idfilename)); SyncId replica2id = GetReplicaId(Path.Combine(args[1], idfilename)); try // Set options for the sync operation FileSyncOptions options = FileSyncOptions.ExplicitDetectChanges FileSyncOptions.RecycleDeletes FileSyncOptions.RecycleOverwrites; FileSyncScopeFilter filter = new FileSyncScopeFilter(); filter.filenameexcludes.add(idfilename); // Exclude the id file // Explicitly detect changes on both replicas upfront, to avoid two change // detection passes for the two-way sync DetectChangesOnFileSystemReplica( replica1id, replica1rootpath, filter, options); DetectChangesOnFileSystemReplica( replica2id, replica2rootpath, filter, options); // Sync in both directions SyncFileSystemReplicasOneWay(replica1Id, replica2id, replica1rootpath, replica2rootpath, filter, options); SyncFileSystemReplicasOneWay(replica2Id, replica1id, replica2rootpath, replica1rootpath, filter, options); catch (Exception e) Console.WriteLine("\nException from File System Provider:\n" + e.tostring()); public static void DetectChangesOnFileSystemReplica( SyncId replicaid, string replicarootpath, FileSyncScopeFilter filter, FileSyncOptions options) FileSyncProvider provider = null; try provider = new FileSyncProvider(replicaId, replicarootpath, filter, options); provider.detectchanges(); finally // Release resources if (provider!= null) provider.dispose();

public static void SyncFileSystemReplicasOneWay( SyncId sourcereplicaid, SyncId destinationreplicaid, string sourcereplicarootpath, string destinationreplicarootpath, FileSyncScopeFilter filter, FileSyncOptions options) FileSyncProvider sourceprovider = null; FileSyncProvider destinationprovider = null; try sourceprovider = new FileSyncProvider( sourcereplicaid, sourcereplicarootpath, filter, options); destinationprovider = new FileSyncProvider( destinationreplicaid, destinationreplicarootpath, filter, options); destinationprovider.appliedchange += new EventHandler<AppliedChangeEventArgs>(OnAppliedChange); destinationprovider.skippedchange += new EventHandler<SkippedChangeEventArgs>(OnSkippedChange); SyncAgent agent = new SyncAgent(); agent.localprovider = sourceprovider; agent.remoteprovider = destinationprovider; agent.direction = SyncDirection.Upload; // Sync source to destination Console.WriteLine("Synchronizing changes to replica: " + destinationprovider.rootdirectorypath); agent.synchronize(); finally // Release resources if (sourceprovider!= null) sourceprovider.dispose(); if (destinationprovider!= null) destinationprovider.dispose(); public static void OnAppliedChange(object sender, AppliedChangeEventArgs args) switch (args.changetype) case ChangeType.Create: Console.WriteLine("-- Applied CREATE for file " + args.newfilepath); break; case ChangeType.Delete: Console.WriteLine("-- Applied DELETE for file " + args.oldfilepath); break; case ChangeType.Overwrite: Console.WriteLine("-- Applied OVERWRITE for file " + args.oldfilepath); break; case ChangeType.Rename: Console.WriteLine("-- Applied RENAME for file " + args.oldfilepath + " as " + args.newfilepath); break; public static void OnSkippedChange(object sender, SkippedChangeEventArgs args)

Console.WriteLine("-- Skipped applying " + args.changetype.tostring().toupper() + " for " + (!string.isnullorempty(args.currentfilepath)? args.currentfilepath : args.newfilepath) + " due to error"); if (args.exception!= null) Console.WriteLine(" [" + args.exception.message + "]"); public static SyncId GetReplicaId(string idfilepath) SyncId replicaid = null; if (File.Exists(idFilePath)) using (StreamReader sr = File.OpenText(idFilePath)) string strguid = sr.readline(); if (!string.isnullorempty(strguid)) replicaid = new SyncId(new Guid(strGuid)); if (replicaid == null) using (FileStream idfile = File.Open( idfilepath, FileMode.OpenOrCreate, FileAccess.ReadWrite)) using (StreamWriter sw = new StreamWriter(idFile)) replicaid = new SyncId(Guid.NewGuid()); sw.writeline(replicaid.getguidid().tostring("d")); return replicaid; Summary In this article we have seen how Sync Services for File Systems can be used to synchronize the contents of file system directories on PCs and removable media such as USB thumb drives, and discussed the details of this new sync provider which is part of the Microsoft Sync Framework. The Microsoft Sync Framework includes all of the things required to integrate applications into an offline or collaboration based network by using the pre-created providers or writing new custom providers. Providers enable any data source to participate in data synchronization regardless of network or device type. For more information or to get a copy of the Microsoft Sync Framework CTP1, please visit http://msdn.microsoft.com/sync.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This white paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in, or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2005 Microsoft Corporation. All rights reserved. Microsoft, Windows, and the Windows logo are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.