IBM i2 Intelligence Analysis Platform Developer Essentials Data Acquisition Developer Guide. Version 1 Release 1

Transcription

1 IBM i2 Intelligence Analysis Platform Deeloper Essentials Data Acquisition Deeloper Guide Version 1 Release 1

2 Note Before using this information and the product it supports, read the information in Notices on page 29. This edition applies to ersion 1, release 1, modification 1 of IBM i2 Intelligence Analysis Platform Deeloper Essentials and to all subsequent releases and modifications until otherwise indicated in new editions. Copyright IBM Corporation 2012, US Goernment Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents Chapter 1. Data acquisition in the Intelligence Analysis Platform Chapter 2. Approaches to data acquisition Data load direct Data load direct diagram Data load ELP stage Data load ELP stage diagram Data access on-demand Solution types External data subset creation External data subset exploration External data retrieal Chapter 3. Data conersion Conersion of external data to XML XML schema definitions XSD reference Conersion of XML to Intelligence Analysis Platform items Modification of Intelligence Analysis Platform items 22 Chapter 4. Deelopment process for data access on-demand Supported operations Data source queries Data transformation Creation and deployment Notices Programming interface information Trademarks Copyright IBM Corp. 2012, 2014 iii

4 i Data Acquisition Deeloper Guide

5 Chapter 1. Data acquisition in the Intelligence Analysis Platform Data acquisition extends the functionality of the Intelligence Analysis Platform by enabling users to isualize and inestigate data from third-party proiders. Users hae the choice of leaing the data in place or adding it to the Analysis Repository, in full or in part. Data acquisition solutions enable users to manipulate the data in external sources with the tools in the Intelligence Portal and Analyst's Notebook Premium. Howeer, different sources and different data imply different approaches to writing data acquisition projects: If your organization has an existing store of analysis-ready data, you can use data acquisition to reproduce that data in the Analysis Repository. For most purposes, loading the data into the repository is a one-off process, although later manipulation is possible in some circumstances. For a data store that changes relatiely slowly, and that you want to retain as a separate store, there is a different approach. You can take a snapshot of the store, and use data acquisition to load its data into a parallel of the Analysis Repository that is read-only for users. Another possibility is a data store that contains lie data that you want to query on demand. (A data store that generates short-lied result subsets in response to external queries has a similar profile.) In this case, a data acquisition solution can obtain results, present items to the user, and enable analysis as though those items were repository data. In all of the preceding approaches, the data that the user sees might be out-of-date, compared with the original data store. The final approach to data acquisition takes an item that was retrieed through another approach, and retriees the latest ersion from its source. (With some data stores, you are also able to retriee items that are related to the specified item.) For some data proiders, it is appropriate to use more than one approach to data acquisition. In particular, it can be useful to combine the ability to analyze data with the ability to retriee the most up-to-date information. Copyright IBM Corp. 2012,

6 2 Data Acquisition Deeloper Guide

7 Chapter 2. Approaches to data acquisition In the IBM i2 Intelligence Analysis Platform, data acquisition is about connecting to, importing, and analyzing information from data proiders other than the Analysis Repository. Data acquisition projects can inole different technologies, depending on the nature of the data that is being acquired. Data acquisition is a term that the Intelligence Analysis Platform uses to describe a number of different approaches to accessing external data. Data load direct Data acquisition projects that use data load direct enable users to import external data into the Analysis Repository. Clients can interact with the imported data in the same way as they interact with any other data in the repository. As its name suggests, this approach is the only one that causes the addition of third-party data to the Analysis Repository without further user interention. Data load ELP (entity, link, property) stage In data acquisition, an ELP stage behaes like an Analysis Repository that is read-only for Intelligence Analysis Platform users. An administrator can use data acquisition to load data into an ELP stage, allowing clients to use that data as they would use the repository. The data in an ELP stage is not mixed with repository data. Howeer, clients can choose to add items from an ELP stage to the Analysis Repository. Data access on-demand Data acquisition projects that use data access on-demand enable users of the Intelligence Portal to search and retriee data from third-party proiders. These facilities closely match the standard Analysis Repository operations, and the retrieed data is displayed in the Intelligence Portal. A data access on-demand solution can present functionality to users through the standard Intelligence Portal user interface, or through a custom user interface that is integrated into the Intelligence Portal. Data that clients retriee through data access on-demand is not automatically added to the Analysis Repository. As with an ELP stage, howeer, users can add it manually when required. External data retrieal A data acquisition project that uses external data retrieal enables a lie link with third-party data. The approach allows simple retrieal of data (or related data) that was already identified through one of the other approaches. It is legitimate to implement external data retrieal alongside one of the other approaches on the same data proider. External data retrieal presents users with the latest information from the original proider of a piece of data. The Analysis Repository is not updated with new information unless the user opts to add it. Copyright IBM Corp. 2012,

8 Data load direct Data load direct is one of the approaches for making data from a proider outside the Analysis Repository aailable in the Intelligence Portal. The approach inoles loading data from an external proider into the Analysis Repository, where all of the platform tools for data analysis become aailable. A data load direct solution consists mostly of IBM-supplied components that you can customize to work with particular data proiders. The entry point is the data loader itself, which requires external data in a platform-compatible XML format that many data acquisition solutions consume. The data loader dries both the conersion of external data into items, and the generation of write-side commands that load those items into the Analysis Repository. The Intelligence Analysis Platform API includes a data loader that is intended primarily as mechanism for one-time importing of data, through its createitems method. Howeer, the loader also has updateitems, deleteitems, and purgeitems methods that you can call to perform the operations that their names suggest. These methods make it possible to reflect changes in the external proider, but if synchronization is your main requirement, a different approach to data acquisition might be appropriate. Note: Using the "update", "delete", and "purge" methods can conflict with changes that users made to the Analysis Repository since the original data was imported. The caller is responsible for ensuring that this functionality is used appropriately. 4 Data Acquisition Deeloper Guide

9 Data load direct diagram In a data load direct solution, calls to the createitems and updateitems methods of the bulk data loader add external data to the Analysis Repository. Calls to deleteitems and purgeitems hae the same effect on the repository as their namesakes on the command and purge serices. The diagram illustrates how a data load direct solution can change the contents of the Analysis Repository: 1. If appropriate, the Analysis Repository loader interacts with the XML conerter to create the items to be added or updated. 2. The Analysis Repository loader uses the command generator to create commands that drie the write side of the Intelligence Analysis Platform. 3. The loader passes the generated commands to the Jaa client, which is a standard Intelligence Analysis Platform extension point. 4. The Jaa client calls the standard command or purge serice to create, update, delete, or purge items in the Analysis Repository. Chapter 2. Approaches to data acquisition 5

10 Data load ELP stage Data load ELP stage is one of the approaches for making data from a proider outside the Analysis Repository aailable in the Intelligence Portal. The approach inoles loading data from the external proider into a semi-permanent store that has the same structure as the Analysis Repository. In a deployment of the Intelligence Analysis Platform, an ELP (entity, link, property) stage is a data store that is read-only for users. The data in an ELP stage supports the same searching and filtering operations as the data in the Analysis Repository. The purpose of an ELP stage is to present a large olume of relatiely static information for inestigation, without adding it to the Analysis Repository. A data load ELP stage solution uses the same mechanism for conerting and editing external data that the other approaches to data acquisition use. You must proide data to the loader as appropriately formatted XML. The loader conerts the XML into Intelligence Analysis Platform items, editing them to add extra information if necessary. After the data loader creates items, it uses an eent generator to turn the items into eents, and places the eents onto a message queue. The ELP stage retriees the eents from the queue, and reconstitutes the items that it stores from the information that the eents contain. 6 Data Acquisition Deeloper Guide

11 Data load ELP stage diagram In a data load ELP stage solution, calls to the loader result in the addition of eents to the message queue. An ELP stage receies the eents, and then creates or modifies the items that it stores in accordance with the methods that were called. The diagram illustrates how a data load ELP stage solution manages changes to its own data store: 1. If appropriate, the ELP stage loader interacts with the XML conerter to create the items to be added or updated in the ELP stage. 2. The ELP stage loader uses the eent generator to create the eents that drie an ELP stage in an Intelligence Analysis Platform deployment. 3. The loader places the generated eents in the standard Intelligence Analysis Platform message queue. 4. The ELP stage receies the eents from the queue, and uses their contents to populate its data store by creating, updating, deleting, or purging items. Chapter 2. Approaches to data acquisition 7

12 Data access on-demand Data access on-demand is one of the approaches for making data from a proider outside the Analysis Repository aailable in the Intelligence Analysis Platform. The approach usually inoles selecting and then analyzing the data in a subset of a larger body of information. Data access on-demand solutions are appropriate in situations where the external proider has its own, reasonably rich, search support. A typical solution uses the external proider to create the subset, and then enables analysis on the subset contents. Usually, the query to the external proider takes place when the user requests the data (rather than preemptiely, for example). A data access on-demand solution always requires one new read-side serice, and sometimes requires two of them. The precise structure of the solution depends on the relationship between the data proider and the subsets that the Intelligence Analysis Platform works with. In general, howeer, there are two phases: subset creation, and subset exploration. Solution types Conceptually, a data access on-demand solution for external data inoles a subset creation phase and a subset exploration phase. In practice, you might be able to meet the needs of your client or the data proider better by partially or completely merging those phases. 8 Data Acquisition Deeloper Guide

13 The following diagram illustrates the mechanisms for three subset creation options: using a direct query (A), exported files (B), or a creation serice (C). The "A" interactions inole a subset creation serice (SCS) and a subset exploration serice (SES). In this scenario, howeer, the SCS does not actually create a subset in response to a query. Instead, the SCS returns information about the query back to the client (A1), which forwards it to the SES. The SES then uses a locator to query the data proider directly (A2), or through an adapter (A3). Note: To integrate a data acquisition solution with the standard Intelligence Portal user interface for searching, you must implement an SES and an SCS. The "B" interactions inole an SES, but not an SCS. In this scenario, the data acquisition solution includes a custom user interface component that queries an external data proider, resulting in the creation of an exported subset. The client passes information about the subset to the SES, and the SES tells a locator where to find that subset. The "C" interactions show a request from the Intelligence Portal to an SCS to query the external data proider. The SCS creates the subset and returns a token that describes the location of the subset to the client. The interaction between the client and the SES then proceeds in the same way as the "B" interactions. When the client calls the SES for the first time, the SES populates the item cache that it uses to respond to future requests. The key to this operation is the information in the token that the client passes to the serice, which the serice then passes to the locator. Chapter 2. Approaches to data acquisition 9

14 The locator uses the information in the token to access the appropriate subset, or query the external data proider. After that, the following sequence takes place: 1. The locator retriees data from the subset (or proider), and returns an XML representation of the data to the connector. 2. The connector uses the conerter and the editor to turn the data from the subset into Intelligence Analysis Platform items. 3. The connector returns the items to the SES, which creates an indexed cache of those items. 4. Depending on which method the client called, the SES returns a modified identifier, item data from the cache, or both to the client. If the SES receies further requests that specify the same subset, it uses the item cache to respond to them. If the client makes a new query to the data proider, the result is a new subset and a second item cache. By default, all item caches remain aailable for 30 minutes from the time when they were last accessed by a client. External data subset creation The subset creation phase of a data access on-demand solution ends in the existence of some data that a subset exploration serice can consume. How the subset creation phase begins, and what it inoles, depend on how the external proider deliers its data. In data acquisition, an external data subset is an XML representation of some information that was drawn from an external data proider. In a data access on-demand project, there are three main ways in which a subset can be created: A subset exploration serice can query an external data proider directly. In this case, the subset is transient, and the exploration serice is responsible for all parts of the process. An external data proider can export XML data to a known location, in response to a user request. The subset exploration serice is then instructed to look for subsets in that location. A subset creation serice can retriee data from an external proider, and store an XML representation of the data. The caller of the creation serice receies a token that contains the subset location, which is later passed to the exploration serice. Manipulating the XML data from an external proider into Intelligence Analysis Platform items is also a two-phase process. The data is first transformed to a compatible XML format, and then conerted into items. In some projects, the XML in a subset is not in a compatible format, and the exploration serice has more work to do as a result. The implementation of the exploration serice is therefore tightly bound to the nature of the creation phase. 10 Data Acquisition Deeloper Guide

15 Query results as data subsets In some data access on-demand projects, there is neer a physical subset of data from the external proider. Instead, the subset exploration serice queries the proider directly, and uses the results as the "subset" that is presented to users. Note: It is relatiely unusual for an external proider to hae built-in support for the rich querying that a data access on-demand solution can request. For example, it is rare to find an external proider that supports "expand" or "network search" operations in a form that the platform expects. As an alternatie, consider one of the other data access on-demand strategies, and then implement an external data retrieal serice against the same proider. Depending on the intended user interface, a solution in which the exploration serice queries the proider directly might or might not include a subset creation serice. By implementing a creation serice in these circumstances, you allow users to run searches from the Intelligence Portal. The actual query against the proider still takes place in the exploration serice. Exported files as data subsets One way to create the subset for a data access on-demand project is to hae the external proider export data directly to a file. An exploration serice then reads the file and processes the data that it contains so that users can run further queries against it. A typical model for a project that uses exported file subsets is one where the external proider has an existing user interface for running queries. Each query that a user runs creates a separate subset, and the exploration serice makes the results of all the queries aailable in the Intelligence Analysis Platform. In some projects, the external proider can generate the exported file in the format that is required for conersion to Intelligence Analysis Platform items. In other projects, the subset exploration serice is responsible for transforming and conerting the exported data. Chapter 2. Approaches to data acquisition 11

16 Data subset creation serice In data acquisition, a subset creation serice can allow users to run initial searches against an external proider through the same interface as Intelligence Portal searches. On completion, the creation serice returns a token that the portal can use in subsequent calls to a subset exploration serice. A successful call to a subset creation serice can result in the creation (or the replacement) of an external data subset. Howeer, the detail of the interaction between the creation serice and the external data proider is highly dependent on the nature of the proider. For example, the Intelligence Portal might use a subset creation serice to query for data that matches a particular text string. In response, the portal receies a token that it can use to identify the created subset in future operations. Later, the Intelligence Portal can use a subset exploration serice to retriee data from the subset in the form of Intelligence Analysis Platform items. From the perspectie of the Intelligence Portal, a subset creation serice has a web serice endpoint that proides typical Intelligence Analysis Platform search methods. The portal can use a subset creation serice to call Search, DumbbellSearch, NetworkSearch, FullTextSearch, and Expand against an external data source. For a deeloper who is implementing an external data subset creation serice to perform subset creation, client requests are calls to the IExternalDataSubsetCreationSerice interface. The requests must be mapped to queries against the external data proider. 12 Data Acquisition Deeloper Guide

17 External data subset exploration In a data acquisition solution that uses data access on-demand, the external data subset exploration serice is equialent to the standard search serice. The subset exploration serice enables clients to browse into, query, and expand against data that is retrieed from an external data proider. From the perspectie of the client, the subset exploration serice has a web serice endpoint that proides typical Intelligence Analysis Platform search methods. Howeer, these methods also allow the client to identify the subset against which the searches take place. Clients use the subset exploration serice to call Search, DumbbellSearch, NetworkSearch, Expand, and Retriee against an external data subset. For a deeloper who is implementing an external data subset exploration serice, client requests are calls to the IExternalDataSubsetExplorationSerice interface. The actions of the exploration serice when it receies the first client call depend on the presence and nature of any preceding subset creation phase. 1. Receie a client request to search within a subset of data from an external proider, which is identified through a token. 2. Use the specified token to "locate" the subset. Depending on the nature of the data access on-demand project, this action takes one of three forms: Execute a query against an external data proider. Load files that were earlier exported from an external data proider. Access a subset that was created by a subset creation serice. 3. Optional: Conert the contents of the located subset into an XML representation of the data. (The subset might already be in XML format.) 4. Optional: Transform the XML contents of the subset to a format that is compatible with the Intelligence Analysis Platform. (The XML might already be in platform-compatible format.) 5. Conert the platform-compatible XML into Intelligence Analysis Platform items. 6. Populate an in-memory cache with the conerted items, which now represent the query results. 7. Execute the search that the client originally requested against the items in the cache, and return results to the client. The support that IBM proides for data acquisition includes a number of tools that make creating data access on-demand projects easier. For example, the in-memory cache is supplied by default, and there are implementations of the code for location and transformation that work in seeral common situations. Chapter 2. Approaches to data acquisition 13

18 Subset exploration serice diagram The first successful call to a subset exploration serice eentually results in the initialization of an item cache from an external data subset. For a particular project, the detail of initialization lies in the interaction between a data connector, a subset locator, and an XML conerter. Note: If a client does not initialize the item cache explicitly before it requests data, then initialization takes place implicitly during the first request. 14 Data Acquisition Deeloper Guide

19 The diagram reflects the ariety of choices aailable in a subset exploration serice, and especially in the behaior of the subset locator. In some deployments, a call to the locator must trigger the creation of an external data subset. In other deployments, the subset exists when the locator is called. The locator can use the identifier that it receies to determine its behaior. Some subsets contain data that was not retrieed in XML format from the external proider. In these cases, the locator is responsible for conerting the data to XML so that other parts of the serice can interact with it. Some subsets contain XML that the locator must transform to be compatible with the exploration serice code for creating Intelligence Analysis Platform items. In other subsets, the data is already in this form. Regardless of the process detail, the XML data that the locator returns to the connector must be in platform-compatible format. From that point on, the behaior of the exploration serice is common to all projects. External data connector The role of the external data connector in a data access on-demand project is to coordinate two processes. The connector manages retrieing data through an external data subset locator, and conerting that data into items for the Intelligence Analysis Platform. IBM supplies an implementation of an external data connector that is called by the supplied subset exploration serice. The connector has a single getsubset method that receies an external data subset identifier and then performs the following tasks: 1. The connector passes the subset identifier to the locator. The locator uses information in the identifier to find or create the subset, and then processes the subset data to return appropriately formatted XML. 2. The connector sends the returned XML to a conerter that turns the XML into Intelligence Analysis Platform items. In most circumstances, the supplied implementation of the IExternalDataConnector interface delegates its tasks in such a way that replacing it is an unusual requirement. External data subset locator In a data access on-demand project, an external data subset locator has two roles. It must find or create an external data subset, and then retriee well-formed XML data from it. Like an external data connector, an external data subset locator has a single method named getsubset that takes a subset identifier argument. The identifier contains all the information that the locator needs to retriee the appropriate XML data. The behaior of the locator depends on the identifier that it receies. The contents of a subset identifier hae meaning only in the context of a complete data access on-demand solution. For example, the identifier might contain one of the following artifacts: A query to execute against a data source, possibly through a data adapter The location of one or more files in the file system The name of a method to call on a "subset" object Chapter 2. Approaches to data acquisition 15

20 The nature of the data that an external data subset locator retriees depends on the proider. The locator might need to conert the data to XML, or to transform XML into the format that the conerter requires. If the data starts in XML format, then the transformation is relatiely straightforward, and IBM supplies tools that assist with it. If the data does not start in XML format, then the locator is responsible for this part of the processing. IBM supplies a simple implementation of an external data subset locator that delegates all its work to a data adapter. That locator implementation assumes that it can return the XML that it receies straight to the connector; it does no extra processing for itself. External data adapter In a data access on-demand project that queries the external data proider directly, a data adapter can act as the interface that other components call. In such projects, writing a data adapter is often the easiest way to enable the retrieal of information from an external proider. An external data adapter implements the IExternalDataAdapter interface, which has methods with the same names as IExternalDataSubsetCreationSerice: search, dumbbellsearch, networksearch, fulltextsearch, and expand. In a particular project, the adapter might be called by either the locator or a creation serice, depending on where responsibility for querying lies. Note: There is no supplied implementation of a data adapter because there is no standard external data proider. When you create a data adapter, you must call the API of the data proider appropriately from the methods that you implement. Supplied implementation IBM supplies an implementation of the IExternalDataSubsetExplorationSerice interface in Intelligence Analysis Platform Deeloper Essentials. This implementation ultimately relies on your implementation of a subset locator to interpret the token that it receies. The supplied implementation has the following features: An implementation of initializesubset that uses a data connector to conert data from a subset to Intelligence Analysis Platform format, and then caches that data. Implementations of getlatestitems, search, dumbbellsearch, networksearch, and expand that ensure initializesubset was called, and then operate on the contents of the item cache. Implementations of retriee and networkretriee that support further filtering and sorting on search (or similar) results, and then return items to the caller. The external data connector that the supplied implementation of IExternalDataSubsetExplorationSerice uses has its own dependencies, in particular the subset locator and the XML conerter. 16 Data Acquisition Deeloper Guide

21 External data retrieal External data retrieal is one of the approaches for making data from a proider outside the Analysis Repository aailable to the Intelligence Analysis Platform. The approach inoles querying the proider for the latest ersion of data that was already retrieed through one of the other approaches. An external data retrieal solution requires a single new read-side serice, and a proider that supports a basic querying function. In some circumstances, external data retrieal is the only way to access a proider directly from the Intelligence Analysis Platform. When you retriee data with one of the other approaches to data acquisition, the cards on each item in the data contain proenance information. This information connects the retrieed data with its source in the external data proider. Sometimes, the information can also connect the retrieed data to its origin in another data source. In the diagram, the subset exploration serice (from a data access on-demand solution) retriees data from a data warehouse. The data warehouse was originally populated from another pair of data sources. The cards on the items that the exploration serice retriees contain proenance information about the original source of their data. A client can use the proenance information to query the original source for the latest ersion of the data, through an external data retrieal serice. Like all data acquisition solutions, an external data retrieal serice conerts the data from its format in the proider to Intelligence Analysis Platform format. To enable an external data retrieal serice, the proider must support retrieal of an indiidual "item", gien some identifying information. The serice is more successful if the proider can also retriee "items" that are related to the identifying information. Chapter 2. Approaches to data acquisition 17

22 18 Data Acquisition Deeloper Guide

23 Chapter 3. Data conersion One of the problems that any data acquisition project must sole is that of data conersion. Regardless of the proider, the project must be able to conert external data into a format that the Intelligence Analysis Platform can understand. To help with the conersion of external data, projects use a pair of standard components: an XML conerter, and an item editor. To benefit most from the support aailable in data acquisition, projects can conert from the natie format of the external data in three steps: 1. Arrange for data that is entering the conersion process to be in XML format. 2. Conert the external data in XML format to Intelligence Analysis Platform items. 3. Add information to the items that the platform requires, but which the proider did not contain. Conersion takes place at different times in the different data acquisition approaches. In data load direct (and ELP stage), the input must be in XML format, and the loader starts by conerting the XML into items. In data access on-demand and external data retrieal, all of the conersion happens between the query to the proider and the creation of the cache. Copyright IBM Corp. 2012,

24 Conersion of external data to XML In any data acquisition solution, the XML conerter requires the XML input that it receies to be in a specific format. The XML input must alidate against an XML schema that is deried from the Intelligence Analysis Platform schema of the target deployment. It is your responsibility to ensure or arrange that the data proider can supply data in XML format. It is also your responsibility to transform the XML from the proider to the target structure. Howeer, IBM does assist with the creation of the XML schema that defines the target structure. To generate an XML schema from the Intelligence Analysis Platform schema, you use the supplied schema conerter. The tool has two distinct outputs: An XSD file that contains the generated XML schema A set of Jaa classes that map from the XML schema to the Intelligence Analysis Platform schema. To transform the XML from the proider to the XML that the conerter requires, you must create and apply a mapping from one to the other. There are seeral third-party applications that can assist with this task, through a range of different technologies. XML schema definitions It is not straightforward to conert arbitrary XML data into a form suitable for the Intelligence Analysis Platform. Deeloper Essentials makes the process easier by proiding a tool for expressing the Intelligence Analysis Platform schema as an XSD file. With a suitable schema definition, it is possible to transform records from the data source into a form that is compatible with the Intelligence Analysis Platform. During deelopment of a data acquisition solution, you use the schema conerter to create a target for the XML mapping process. To run the tool, issue the following command at a command prompt. python da-setup.py -t generate-mapping-jar -x IapSchema -z XsdFiles -o JarFiles IapSchema is the path to an existing Intelligence Analysis Platform schema file. XsdFiles and JarFiles are the paths to directories where the command saes its output. Note: The JAR files that the schema conerter creates alongside the XSD files are inoled later in the deelopment process. The XML conerter uses the classes in the JAR file to conert from the transformed XML to Intelligence Analysis Platform items. The generated XML schema definition contains a flattened representation of the Intelligence Analysis Platform schema. All of the item types and property group types in the schema are defined as complex types in the XML schema definition. All of the property types in the Intelligence Analysis Platform schema are defined as elements within the complex types. To use the support in Deeloper Essentials, all XML data from the data source must be transformed to match the XML schema definition. After transformation, the data is ready to be returned from the external data adapter. 20 Data Acquisition Deeloper Guide

25 XSD reference The XSD files that are generated from an Intelligence Analysis Platform schema ary depending on the schema contents. Howeer, the files all hae the same basic structure. This example represents a syntactically complete (but functionally useless) XSD file, as generated by the schema conerter. The following facts affect the structure of the file: The original Intelligence Analysis Platform schema contains a single link type named Employment. The Employment link type contains two property types, and a property group type named EmploymentDetails. The EmploymentDetails property group type contains four property types. <?xml ersion="1.0" encoding="utf-8" standalone="yes"?> <xs:schema ersion="1.0" xmlns:xs=" <xs:element name="queryresult" type="queryresult"/> <xs:complextype name="employment"> <xs:all> <xs:element name="_1_itemid" type="xs:string"/> <xs:element name="_2_fromendid" type="xs:string"/> <xs:element name="_3_toendid" type="xs:string"/> <xs:element name="_4_linkdirection" type="xs:string"/> <xs:element name="_5_linkstrength" type="xs:string"/> <xs:element name="employment_additionalinformation" type="xs:string" minoccurs="0"/> <xs:element name="employment_employmentdetails" type="employment_employmentdetails" minoccurs="0"/> <xs:element name="employment_uniquereference" type="xs:string" minoccurs="0"/> </xs:all> </xs:complextype> <xs:complextype name="employment_employmentdetails"> <xs:all> <xs:element name="employment_employmentenddate" type="xs:datetime" minoccurs="0"/> <xs:element name="employment_employmentstartdate" type="xs:datetime" minoccurs="0"/> <xs:element name="employment_employmenttype" type="xs:string" minoccurs="0"/> <xs:element name="employment_role" type="xs:string" minoccurs="0"/> </xs:all> </xs:complextype> <xs:complextype name="queryresult"> <xs:all> <xs:element name="employments"> <xs:complextype> <xs:sequence> <xs:element name="employment" type="employment" minoccurs="0" maxoccurs="unbounded"/> </xs:sequence> </xs:complextype> </xs:element> </xs:all> </xs:complextype> </xs:schema> Chapter 3. Data conersion 21

26 Implications The following features of the example are common to all XSD files generated by the schema conerter: There is always an element that is named QueryResult, with a type that is also named QueryResult. The QueryResult type is a container for all complextypes that represent item types. When a complextype represents an item type (rather than a property group type), the names of its first few elements begin with numbers. These numbered elements represent the mandatory property types that all item types must contain. The elements that represent the property types and property group types of an item type hae names that begin with the name of the item type. The elements that represent the property types of a property group type hae names that begin with the name of the containing item type. XML content that conforms to any schema generated with the schema conerter has a single root element named QueryResult. The root element can contain any number of elements that are named after item types in the original Intelligence Analysis Platform schema. XML data that leaes the external data adapter must alidate against the generated XSD file. Conersion of XML to Intelligence Analysis Platform items In a data acquisition project, the XML conerter receies data from an external proider that was transformed into platform-compatible XML format. The conerter is then responsible for turning the input into entities and links for the Intelligence Analysis Platform. The supplied components of Deeloper Essentials include support for conerting the XML that they receie into Intelligence Analysis Platform items. To perform the conersion, the supplied components use the Jaa classes that the schema conerter generates. To take adantage of this support, you must ensure that the JAR file that contains the classes is on the class path when you deploy the solution. Modification of Intelligence Analysis Platform items Before the XML conerter in a data acquisition project returns items to a caller, it sends those items to an item editor. The item editor proides an opportunity to augment or modify the content of the items that the XML conerter created. Data from an external proider does not always contain all the information that a typical item in the Intelligence Analysis Platform requires. In particular, external proiders often proide no alues for some or all of the following content: Information about the originator of the data, which would normally be displayed in the Source information section of a form in the Intelligence Portal Grading information for each of the property alues The security settings that determine what access and grant leels users receie if items are added to the Analysis Repository 22 Data Acquisition Deeloper Guide

27 The XML conerter calls your implementation of the IExternalDataItemEditor interface, where your code has full access to the contents of an item. In the edit method, you can supply any of the content from the list. Note: You can also use the edit method to modify alues that came from the external data proider. After the item editor, there is no further opportunity to change the results that the XML conerter returns. Items at this stage are likely to be presented to users, and might be added to the Analysis Repository. Chapter 3. Data conersion 23

28 24 Data Acquisition Deeloper Guide

29 Chapter 4. Deelopment process for data access on-demand Deeloping a subset exploration serice with Intelligence Analysis Platform Deeloper Essentials inoles writing some interface implementations, and calling or generating some others. Schematically, a subset exploration serice for the Intelligence Analysis Platform that was created with Deeloper Essentials looks like this diagram: The process of using Deeloper Essentials to deelop a subset exploration serice inoles the following stages: Determine which of the standard Intelligence Analysis Platform operations ("search", "dumbbell search", "network search", "expand", "get latest items") the data source supports. Implement the pieces of the external data adapter that query the data source on behalf of the client. Execute the schema conerter against the Intelligence Analysis Platform schema to create an XML schema definition and JAXB classes. Implement the pieces of the external data adapter that transform retrieed XML data to match the XML schema. Add the JAXB classes to the project so that the XML conerter can use them to create Intelligence Analysis Platform items. Implement an external data item editor to add source, security, and grading information to the created items. Copyright IBM Corp. 2012,

30 Supported operations Build the project and deploy the solution to an instance of the Intelligence Analysis Platform. The tools and code libraries that IBM supplies for data access on-demand proide maximum support when the external data source can return XML results. Howeer, the framework is still useful een when XML output is not aailable. A data source that is further from the assumptions just increases the amount of custom code that you must write. The Intelligence Analysis Platform proides seeral ways for clients to request information from an external data source. An external data subset exploration serice must support some, but not necessarily all, of these ways. Clients can perform fie different operations on an external data source through an external data subset exploration serice: GetLatestItems Returns the items with specified identifiers. Search Runs a search for items with data that matches some specified criteria, and returns a summary of the search results to the client. DumbbellSearch Runs a search for entity-link-entity structures with data that matches some specified criteria, and returns a summary of the search results to the client. NetworkSearch Runs a search for complex structures of items with data that matches some specified criteria, and returns a summary of the search results to the client. Expand Returns links and entities that are connected to the entities with specified identifiers. In Intelligence Analysis Platform Deeloper Essentials, the supplied implementation of the IExternalDataSubsetExplorationSerice interface has built-in support for "get latest items" operations. Howeer, the code delegates requests for the other operations to your implementation of IExternalDataAdapter. Before you can implement IExternalDataAdapter, you must determine which of "search", "dumbbell search", "network search", and "expand" the data source supports. To support "search", the data source must at least be able to retriee records with content that matches a string. To support "dumbbell search", the data source must be able to retriee simply related records, where each record in the relationship fulfills some criteria. To support "network search", the data source must be able to retriee collections of related records, where each record in the collection fulfills some criteria. To support "expand", the data source must be able to retriee records that are conceptually connected to a specific, identified record. 26 Data Acquisition Deeloper Guide

31 Data source queries Data transformation To retriee information from an external source in a data access on-demand solution, you must map from calls to IExternalDataAdapter methods, to calls to the data source API. In a complete implementation of an external data adapter method, your code must perform the following tasks: 1. Interpret the request from the client 2. Perform queries against the data source 3. Transform the results to a prescribed format 4. Return the transformed results to the caller Note: If you decide not to support one or more of the operations, then it is alid to throw a standard Jaa UnsupportedOperationException from the method implementation. You must also configure a settings file appropriately during deployment. Performing a data source query in response to a client request is the part of the process for which Intelligence Analysis Platform Deeloper Essentials proides the least assistance. The wide ariety of potential data sources means that there is no approach to mapping that works in the general case. To assist with later tasks, Deeloper Essentials places requirements on the data that you retriee from the source. The output of the querying phase must be in XML format, either because the data source proides it that way, or because you generated it. The data from the source is then ready to be transformed so that it corresponds with the Intelligence Analysis Platform schema. It is not strictly necessary for the XML retrieed from the data source to use the same schema for the results of eery supported operation. If the schema is consistent between operations, howeer, subsequent stages of implementing the external data adapter become easier. A data access on-demand solution must transform the XML data from an external data adapter to match the schema definition that is generated from the Intelligence Analysis Platform schema. This XML data must also be contained in the return alues of the external data adapter methods. The difficulty of transforming XML to the structure defined in the generated XSD file ("QueryResult format") depends in part on the consistency of the XML. If the data source returns data in the same basic structure for all possible queries, then you can use the same transformation in all cases. The difficulty of creating the transformation depends on the complexity of the returned XML, and the similarity of the XML to the target structure. You might choose one of seeral approaches, depending on your familiarity with the technologies inoled: Create your own XSLT translation from one form to the other Write XML DOM manipulation code as part of the implementation Use a third-party tool to construct the mapping and generate code Chapter 4. Deelopment process for data access on-demand 27

32 Creation and deployment The methods of IExternalDataAdapter all return the transformed data in Source objects that must contain a correctly formed QueryResult element. The size and precise contents of the element, howeer, are for you to decide: The question of how many QueryResult elements to create depends on the size of the result set and the characteristics of the data source. There is no requirement for a QueryResult to be self-contained. A link and its ends do not hae to be represented in the same element. The supplied external data connector reassembles separate QueryResult elements into a single result before it performs any further processing. Therefore, you can base your implementation decisions solely on the throughput of the external data adapter. Your choice does not affect the rest of the external data serice. When the components of an external data serice are all written or assembled, you can build the serice and deploy it to a running instance of the Intelligence Analysis Platform. When the external data serice is successfully deployed, the Intelligence Portal automatically detects the new serice and determines its capabilities. The external data source becomes selectable by name in the user interface of the Intelligence Portal. 28 Data Acquisition Deeloper Guide

33 Notices This information was deeloped for products and serices offered in the U.S.A. IBM may not offer the products, serices, or features discussed in this document in other countries. Consult your local IBM representatie for information on the products and serices currently aailable in your area. Any reference to an IBM product, program, or serice is not intended to state or imply that only that IBM product, program, or serice may be used. Any functionally equialent product, program, or serice that does not infringe any IBM intellectual property right may be used instead. Howeer, it is the user's responsibility to ealuate and erify the operation of any non-ibm product, program, or serice. IBM may hae patents or pending patent applications coering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drie Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such proisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are proided for conenience only and do not in any manner sere as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it beliees appropriate without incurring any obligation to you. Copyright IBM Corp. 2012,