InstantAtlas TM Server Data Transfer Tools User Guide

Transcription

1 InstantAtlas TM Server Data Transfer Tools User Guide Author: GeoWise User Support Released: 06/11/2012 Version: 6.5.1

2 InstantAtlas Server Data Transfer Tools Table of Contents 1. Introduction Installation Prerequisites Installation options Overview IAS system requirements Metadata Error logs IAS Data Transfer Wizard Choose Source Server Choose Destination Server Define Geo-Type Relationships Automatically creating a Geo-Relationship Manually creating a Geo-Relationship Creating a new Geo-Type in your destination system Clearing Geo-Relationships Choose Indicators Choose Indicator Instances Data Synchronisation Settings Standard Advanced Choose Task Schedule Completion screen Known issues Data Synchronisation Service Manager IAS Data Synchronisation Service Manager Scheduled Tasks Scheduling tasks to run on a client PC Support Appendix - Importing data via IASBulkDataLoad.exe... 24

3 InstantAtlas Server Data Transfer Tools Page 1 1. Introduction IAS Data Transfer Tools consists of two components. These components can be installed on the same PC or Server or installed in separate locations. IAS Data Transfer Wizard is a wizard based utility that allows the user to copy Indicator data, along with related Metadata and geo-data from one installation of InstantAtlas Server to another. Through the Wizard it is possible to import data immediately. IAS Data Synchronisation Service is a Windows Service which runs saved individual data transfer tasks which can be scheduled to run either once at a specified date/time, or at user configurable intervals in the future (e.g. once a month at 2am).

4 InstantAtlas Server Data Transfer Tools Page 2 2. Installation These tools can be installed and run on a client PC or on your IAS server. To be able to use the program you must be able to access the IAS Web Services of both source and destination IAS systems from the PC/server where the tools are installed. You can check this by going to the IAS Web Services of the two systems from the PC/Server that you intend to run the program. The default address of the Web Services is as follows: It is recommended that the Data Synchronisation Service is only installed on a server as PC s are routinely switched off and any scheduled tasks will not run when the PC is turned off. See section 5.3 for information on what happens if a task is scheduled to run at a time that the PC is turned off Prerequisites The IAS Data Transfer Tool requires version 3.5 or higher of the Microsoft.NET Framework to be installed before you can install the program. Your IAS web server will already have this installed. If you do not have a suitable version of the.net Framework already installed you will be taken to the Microsoft Download Center website to download it. Once this is installed you will be able to install the IAS Data Transfer Tool. To install IAS Data Transfer Tool you must be a member of the Administrators group 2.2. Installation options When installing you will be asked if you want to install using the following options: Typical this option will install the Data Transfer Wizard and the Data Synchronisation Windows Service. Custom this option allows you to choose which of the two components should be installed. Complete this option will install the Data Transfer Wizard and the Data Synchronisation Windows Service.

5 InstantAtlas Server Data Transfer Tools Page 3 3. Overview 3.1. IAS system requirements For the Data Transfer Tools to work your two IAS systems must meet the following requirements: They must be at the same overall major version, e.g. 6.4.x where a system will be compatible with a system, but a system will not be compatible with a 6.5.x system. Indicator Unique Codes must be the same in both source and destination IAS systems to be able to synchronise Indicators which already exist in the destination system otherwise they will be created as new (potentially duplicate) Indicators. Date/Time periods for which you are transferring data must have the same Unique Codes in the source and destination IAS systems. Geo-Types (or Geo-Type subsets) must be present in both the source and destination IAS systems which have the same number of Geo-Features, and the Unique Codes of the individual Geo-Features must be the same. It is not imperative that Geo-Types be present the first time you transfer data as Geo-Types missing from the destination system will be created by the wizard. Themes will be created if they are missing from the destination IAS system, however they will be created with their name set to the Unique Code of the Theme on the source IAS system, see Known issues (section 4.9) for further information Metadata When importing a new Indicator in to the destination IAS system both Indicator and Instance level metadata is copied across. When importing a new Geo-Type in to the destination IAS system Geo-Type and Geo-Feature metadata is not copied across. The Advanced Settings on the Data Synchronisation Settings page, see section 4.6.2, allow the user to control which metadata elements are updated when transferring new or existing Instances of existing Indicators Error logs Errors are logged to the locations shown in Table 1: Table 1 Data Transfer Tool log file locations OS Windows Server 2003 Windows Server 2008 Windows XP Windows Vista Windows 7 Location C:\Documents and Settings\All Users\Application Data\GeoWise\InstantAtlas\Server\Logs\ C:\ProgramData\GeoWise\InstantAtlas Server\Logs\ C:\Documents and Settings\All Users\Application Data\GeoWise\InstantAtlas\Server\Logs\ C:\ProgramData\GeoWise\InstantAtlas\Server\Logs\ C:\ProgramData\GeoWise\InstantAtlas Server\Logs\ Log files are named using the following formats: ias2ias-wizard-<yyyy-mm-dd>.log, e.g. ias2ias-wizard log iasdatasyncservice-<yyyy-mm-dd>.log, e.g. iasdatasyncservice log

6 InstantAtlas Server Data Transfer Tools Page 4 The error log level in the iasdatasyncservice log file is controlled by the LogLevel setting in the following file: {IAS install directory}\datasync\bin\geowise.ias.datasynchronization.service.exe.config

7 InstantAtlas Server Data Transfer Tools Page 5 4. IAS Data Transfer Wizard 4.1. Choose Source Server The Choose Source Server page (Figure 1) allows you to specify the details of the source IAS server that data will be transferred from. Figure 1 Choose Source Server The URL of your source IAS server should be entered in the following format: You must also enter a username and password for a user who is a member of the IAS_Administrators, IAS_DataManagers, or IAS_DataUsers groups on the source server. See Table 2 for a summary of user permissions required in the source and destination IAS systems.

8 InstantAtlas Server Data Transfer Tools Page 6 Table 2 Permissions required to run the wizard IAS User Group Source System Permissions Destination System Permissions IAS_Administrators Download Import IAS_DataManagers Download Import IAS_DataSuppliers None None IAS_DataUsers Download None All other IAS_XXXX groups None None Note members of the above groups must also be members of the IAS_Users group. If you do not enter a username and password the wizard will attempt to connect using your current Windows credentials. It is not recommended you leave the username and password fields blank if you intend to save the task to run at a later date using the IAS Data Transfer Service Manager as the automatic user who runs the Data Transfer Service Manager will not have permission to connect to the source and destination IAS systems. Web Proxy: Auto will use the proxy details stored in your internet connection, No Proxy will attempt to make a direct connection to the source server bypassing any proxy settings. Locale: it is possible to request item names from the source IAS system from a specific locale using the locale choice box. This setting controls the Geo-Types listed on the Define Geo-Type Relationships page (see section 4.3), and the Indicators listed on the Choose Indicators page (see section 4.4). Note it is important that you import the data in to an existing Geo-Type that has feature names in the default (en-gb) locale as well as the locale for which you are requesting the item names. Data will be imported in to all available locales in the destination system. See Known issues (section 4.9) for more information on synchronising data from multi-locale systems Choose Destination Server The Choose Destination Server page (Figure 2) allows you to specify the details of the destination IAS server that data will be transferred to.

9 InstantAtlas Server Data Transfer Tools Page 7 Figure 2 Choose Destination Server Enter the URL of your destination IAS server in the following format: You must also enter a username and password for a user who is a member of the IAS_Administrators or IAS_DataManagers groups on the destination server. See Table 2 for a summary of user permissions required in the source and destination IAS systems. See note in section 4.1 regarding leaving the username and password fields blank. Web Proxy settings are the same as described in section 4.1.

10 InstantAtlas Server Data Transfer Tools Page Define Geo-Type Relationships Figure 3 Define Geo-Type Relationships Geo-Types are matched through the Unique Codes of the features they contain. Table 3 lists permissible matches. Table 3 Geo-Type relationships Source IAS System Geo-Type Geo-Type subset Geo-Type Geo-Type subset Destination IAS System Geo-Type Geo-Type Geo-Type subset Geo-Type subset A relationship can only be created if the number of Geo-Features and the Unique Codes of these Geo-Features are identical between the two IAS systems. In this release the wizard does not allow arbitrary matches of features between Geo-Types; you have to match whole Geo-Types or Geo-Type subsets. For example Geo-Type A contains 10 features and you want to map it Geo- Type B which is a superset with 200 features but no subsets the tool will insist on subsets. You should define relationships for all Geo-Types that you wish to import data for. For example if you wish to import data for LSOA and Ward Geo-Types you must define appropriate relationships between these two Geo-Types in the source and destination systems. If you only wish to import data at the LSOA level you only need to define one Geo-Type relationship.

11 InstantAtlas Server Data Transfer Tools Page 9 Note if you are importing/synchronising data for complete Geo-Types that contain subsets and the Geo-Types contain exactly the same features you must not define relationships between the subsets Automatically creating a Geo-Relationship Relationships can be created automatically using the Auto button, see Figure 3. When this button is pressed the tool compares the Unique Codes of the Geo-Types in the source and destination systems and for all those where it finds a match it will then compare the number and Unique Codes of the Geo-Features in each Geo-Type. A relationship will only be created automatically if the Geo-Type Unique Codes match as well as the number and Unique Codes of the Geo-Features Manually creating a Geo-Relationship Relationships can be defined manually by dragging a Geo-Type (or Geo-Type subset) from the Source list on the left hand side and dropping on to a Geo-Type (or Geo-Type subset) in the Destination list on the right hand side. Once a relationship has been defined the icons change and when you hover over the source or destination Geo-Type label an arrow will be drawn indicating the established relationship, see Figure 4 below. Figure 4 Viewing Established Geo-Type Relationships Creating a new Geo-Type in your destination system If you do not already have the Geo-Type in your destination system you can create a new Geo- Type by dragging a Geo-Type (or Geo-Type subset) from your Source system on the left hand

12 InstantAtlas Server Data Transfer Tools Page 10 side and drop it on to the server node on the Destination system on the right hand side. The new Geo-Type created in your destination system will have the same name and Unique Code as the Geo-Type in the source system Clearing Geo-Relationships Any relationship that has been established can be cleared using the Clear and Clear All buttons, see Figure 3. Clear clears the selected relationship. Clear All clears all relationships Choose Indicators The Choose Indicators page (Figure 5) lists all Indicators available in the source IAS system for at the Geo-Type levels for which relationships have been established (see section 4.3). Note the Choose Indicators page lists all Indicators in the source IAS system, including those that are in the Staging Area of the source system. You can browse and select the Indicators you wish to import using three tabs: By Geo-Type By List By Search Figure 5 Choose Indicators

13 InstantAtlas Server Data Transfer Tools Page 11 You can select an individual Indicator on the Choose Indicators page then right click on it to bring up the Available Dates context menu (Figure 6). Selecting this option opens up a window showing all available dates for the selected Indicator (Figure 7). Figure 6 Selected Indicator context menu Figure 7 Selected Indicator available dates window On the Choose Indicators By Geo-Type tab you can select all Indicators available at a particular Geo-Type by selecting the check box next to the actual Geo-Type (Figure 8). Figure 8 Select all Indicators available at a particular Geo-Type Indicators listed on the Choose Indicators By List tab (Figure 9) can be displayed in a number of different ways: List Details Small Icon Large Icon

14 InstantAtlas Server Data Transfer Tools Page 12 Figure 9 Choose Indicators By List tab To select multiple/all Indicators: go to the As List tab (Figure 9) click on the name of the first Indicator then scroll down to the bottom of the list, hold the Shift key and click on the name of the last Indicator - all of the Indicators are now selected, click on one of the check boxes next to any of the selected Indicators and they will all be checked. The search function On the Choose Indicators By Search tab (Figure 10) finds Indicators by searching metadata in the source IAS system. It does not search Indicator names. Figure 10 Choose Indicators by Search tab 4.5. Choose Indicator Instances The top half of the Choose Indicator Instances page (Figure 11) allows you to specify which Instances of the Indicators chosen on the Choose Indicators page (section 4.4) are to be imported in to your IAS system. Here you can specify: Latest Date Only Latest X Dates All Available Dates

15 InstantAtlas Server Data Transfer Tools Page 13 Figure 11 Choose Indicator Instances The bottom half of the Choose Indicator Instances page (Figure 11) allows you to specify for which Geo-Types these Instances will be imported against. This section only shows the Geo-Types in the source IAS system for which you have defined Geo-Type Relationships (see section 4.3). Here you can specify: All Available Geo-Types Selected Geo-Types if this option is selected you must select at least one Geo-Type using the check boxes at the bottom of the window. Note the settings selected on this page have effect on all the Indicators selected on the Choose Indicators page (section 4.4), it is not currently possible to specify that only the latest Instance of one Indicator should be imported and all Instances of another Indicator. To do this you would have to define two separate data transfer tasks Data Synchronisation Settings The Standard and Advanced data import settings replicate the settings available in the IAS giving users of the Data Transfer wizard the same flexibility when importing data from another IAS system as you have when importing data through the IAS Admin Website. Further information on these settings can be found in the IAS Administration Guide and the Advanced Bulk Data Load Guide. Please contact the GeoWise User Support Team (see section 6) if you wish to receive a copy of these documents.

16 InstantAtlas Server Data Transfer Tools Page Standard Figure 12 Standard Data Synchronisation Settings The Standard settings (Figure 12) are described below: Preserve System Metadata same as setting in Admin Website, please see IAS Administration Guide for full details. [only used when importing new Instances or overwriting existing Instances] Ignore Imported Metadata this setting is only relevant when importing data against existing Indicators. It refers to descriptive metadata and determines whether a subsequent load should update any descriptive metadata already present. The default is unchecked, which means that descriptive metadata is always overwritten when loaded. Bypass Staging Area this setting can be used to control if the data uploaded is imported in to the Staging Area or not. Replace Existing Data this setting is only relevant when importing data against existing Indicators. If selected any existing data at the same Instance as being imported will be overwritten, if unselected any existing data is not overwritten. Use Single Instance Transfers when selected the wizard will download the selected Indicator data in separate XML files for each Indicator Instance, if unselected the wizard will download the data in to one file per Geo-Type/Indicator combination. Smaller files are less likely to trigger timeout errors when downloading/importing the data across a network. The default setting is to have this checkbox selected.

17 InstantAtlas Server Data Transfer Tools Page Advanced Figure 13 Advanced Data Synchronisation Settings The Advanced settings (Figure 13) are described below: Themes Action (existing Indicators only) The setting controls how Theme data is treated for existing Indicators and has the following settings: o Add : all new Themes from the source system will be added to the Indicators being transferred. o Update and override (default): all existing Theme information will be deleted and the Themes from the source system will be added to the Indicators being transferred. o Ignore Themes in XML file : Theme information on the Destination IAS system will not be modified for the Indicators being transferred. Metadata Action (existing Indicators only) this setting is only relevant if Ignore Imported Metadata (see section 4.6.1) is set to false and you are importing data against existing Indicators. The setting controls how metadata is treated for existing Indicators and has the following settings: o Add with no replacement : all new metadata items from the source system will be added to the existing Indicator/Instance level metadata in your destination system o Full (default): all existing metadata items will be deleted and the supplied metadata will be inserted. o Selective : only those metadata items the names of which match those in the supplied list will be deleted and the supplied metadata will be inserted. Metadata Level (existing Indicators only) this setting is only relevant if Ignore Imported Metadata (see section 4.6.1) is set to false and you are importing data against

18 InstantAtlas Server Data Transfer Tools Page 16 existing Indicators. The setting allows you to control the level at which metadata values are imported and has the following settings: o Both (default): metadata values for indicators and indicator instances will be processed. o Indicator : only metadata values for indicators will be processed. o Instance : only metadata values for instances will be processed. Import Connection Timeout (minutes) this controls the maximum length of time that the application will wait before generating an error. Increase this limit when importing large XML files. Transfer Folder select the folder that the data is to be temporarily downloaded in to on your source IAS system. The user running the wizard must have modify permissions on this folder if importing straight away through the wizard; if the task is scheduled to run in the future then the NETWORK SERVICE user and USERS group must have modify permissions on this folder. XML Download Only if selected the program downloads the data from the source system only and does not attempt to import the data in to the destination system. The data is saved in the folder specified in the Transfer Folder setting in XML files. See Appendix - Importing data via IASBulkDataLoad.exe (section 7) for an example of how to use the IASBulkDataImport command line tool to import the data created via this download only mode Choose Task Schedule The Choose Task Schedule page (Figure 14) allows you to control when the task is run, you have three options: 1. Import the data immediately. 2. Schedule tasks to import at a specific date/time in the future, e.g. out of normal office hours. 3. Schedule tasks to import on a user defined period, e.g. to set up a routine task to download the latest Instance of the Indicator(s) selected on the Choose Indicators page (section 4.4). Each of these options is described in greater detail below.

19 InstantAtlas Server Data Transfer Tools Page 17 Figure 14 Choose Task Schedule Transfer Now with this selected the transfer task begins immediately when the Next > button is pressed. Transfer Later this option is only available if the IAS Data Synchronisation Service is installed and running on the PC/Server you are running the Data Transfer Wizard from (see section 1). o Transfer at a specific date/time this allows the user to schedule a task to run once at a specific date and time. o Transfer every... these settings allow the user specify a particular frequency for the task to run. For example you could set up a task to import the latest Instance of the Indicators selected on the Choose Indicators page (see section 4.4).that runs at 2am on the first of every month, as shown in Figure 14.

20 InstantAtlas Server Data Transfer Tools Page Completion screen Figure 15 Completion screen Completion screen shows all success/failure messages. For tasks set to run immediately this screen includes the details of the XML files imported, as shown in Figure 15 above. You can copy the details from the summary window by highlighting the text and using the context menu (available from the right mouse button) Known issues Aggregated Instances are imported in to IAS as new Instances in their own right. They do not retain their aggregated status as the lookups used to aggregate the data on the source system may not be present in the destination system. If you create a new Theme using this wizard it will name the new Theme in your destination system with the Unique Code of the Theme from the source system. Workaround: create the Theme(s) in your destination system before using the wizard ensuring that your new Theme has the same Unique Code or the wizard will import the source Theme again. Note: to be able to synchronise data from a multi-locale system you must ensure that all your Indicators have names in the locale that you wish to synchronise data from. For example if you wish to synchronise data from the en-gb locale all Indicators in your IAS system must have names in this locale.

21 InstantAtlas Server Data Transfer Tools Page Data Synchronisation Service Manager The IAS Data Synchronisation Service is a Windows service that runs saved individual data transfer tasks at specified date/times, or at user configurable intervals in the future (e.g. once a month at 2am). The program consists of a Windows service called InstantAtlas Server (IAS) Data Synchronization Service and a user friendly graphical user interface (GUI) which allows you to stop/start this Windows service (see section 5.1) and also view all scheduled tasks (see section 5.2). During installation the Windows service is set to automatically start each time the PC/Server is rebooted. As a result any automatic tasks will run according to their defined schedule. If you wish to disable the Data Synchronisation Service without deleting any previously defined regular tasks you should stop the Windows service and set its start-up mode to Disabled. Note you must have administrative rights to run the Data Synchronisation Service Manager IAS Data Synchronisation Service Manager The Service tab of the Data Synchronisation Manager allows you to control the IAS Data Synchronisation (Windows) Service (Figure 16). Figure 16 Data Synchronisation Service Manager The following buttons are available on this tab: Start Start the Windows service

22 InstantAtlas Server Data Transfer Tools Page 20 Stop Stop the Windows service Restart Restart the Windows service Refresh Refresh the current connection status of the Windows service Edit Open up the standard Windows Services Manager window Scheduled Tasks The Scheduled Tasks tab of the Data Synchronisation Manager allows you to view all scheduled tasks (Figure 17). Figure 17 Scheduled Tasks This tab shows a summary of all the tasks that are currently set up. The function of the Refresh and New Task buttons are described below: Refresh this button refreshes the scheduled task list to update with any new tasks that have been scheduled since it was last refreshed. New Task this button launches either NeSS Data Transfer Tool or Data Transfer Tools wizards. You can select an individual task and right click to bring up a context menu (Figure 18) with a number of additional options:

23 InstantAtlas Server Data Transfer Tools Page 21 Figure 18 Scheduled task context menu Details opens up a window (Figure 19) where you can view the details of the selected transfer task. Reschedule/Rename opens up a window (Figure 20) where you can reschedule or rename the selected transfer task. Delete removes the selected task. Disable disables the selected task. Enable re-enables the selected task. Note if you stop and restart the Windows Service, or reboot the PC/Server running the Data Transfer Service, all previously disabled tasks will be set back to being enabled/active. Figure 19 Scheduled task details

24 InstantAtlas Server Data Transfer Tools Page 22 Figure 20 Reschedule/Rename task window 5.3. Scheduling tasks to run on a client PC As mentioned in section 2, if the Data Synchronisation Service is installed on a client PC any scheduled tasks will not run when the client PC is turned off. In the case that a task was scheduled to run when the PC was turned off the following will happen next time the PC is turned on: The Data Synchronisation Service automatically restarts (unless you have changed the start-up mode of the Windows service (see section 5)). The next run date/time of each active task is checked. The interval of each active task is checked. If the start date/time of any task is in the past, the program adds the task interval until it reaches a point in the future. For example if you had a task that was scheduled to run on the 1 st day of each month, and your PC was turned off whilst you were away on the 1 st of May, when you switched your PC back on after the 1 st of May, the Data Synchronisation Service would automatically start but the task would not run again until the 1 st of the following month e.g. the 1 st of June. Note: tasks do not run immediately to catch up if they did not run when a PC was turned off. However when they do run again, they will take account that a previous synchronisation was missed. For the example above, this means that any data that would have been downloaded on the 1 st of May and the 1 st of June would download on the 1 st of June.

25 InstantAtlas Server Data Transfer Tools Page Support You can access additional documentation and troubleshooting resources from My InstantAtlas If you require further assistance please contact your support provider.

26 InstantAtlas Server Data Transfer Tools Page Appendix - Importing data via IASBulkDataLoad.exe Included below is a brief overview of the settings you should use when importing data exported via the download only mode of the Data Synchronisation Wizard (see section 4.6.2). Further information on using the command line Bulk Data Load program (IASBulkDataLoad.exe) is available in Advanced Bulk Data Load Guide. Please contact the GeoWise User Support Team (see section 6) if you wish to receive a copy of this document. When importing data exported from IAS using the download only mode you must tick the following Advanced data import options in the Admin Website (Figure 21): Use unique codes to find Themes Use unique codes to find Geo-Types Use unique codes to find Time Periods Figure 21 Advanced data import settings However as you can only import individual files through the Admin Website it is likely that will use the command line IASBulkDataImport.exe program to bulk load multiple XML files. The equivalent setting and arguments when using the command line version of this tool are shown below: /c Themes,GeoTypes,TimeConstants Below is a sample command line to import the data exported via the Data Transfer Tools (see Advanced Bulk Data Load Guide for further details): C:\Program Files\GeoWise\InstantAtlas Server\Bin\IASBulkDataLoad /s /u iasadmin /p IA5A9m!n /a /v /k /c Themes,GeoTypes,TimeConstants /f *.xml /l C:\dataimport\log.txt C:\dataimport

27 InstantAtlas Server Data Transfer Tools Page 25 This command will import all XML files from a folder at C:\dataimport and write a log file to C:\dataimport\log.txt.