Reconciliation and best practices in a configuration management system. White paper

Transcription

1 Reconciliation and best practices in a configuration management system White paper

2 Table of contents Introduction... 3 A reconciliation analogy: automobile manufacturing assembly... 3 Conflict resolution... 4 Identity resolution... 4 HP CMS reconciliation architecture Conduits... 6 Data providers... 6 Use cases... 8 Use case #1: reconciliation of DDM data... 8 Use case #2: reconciliation in a federated scenario... 8 Use case #3: reconciliation by an API... 9 Use case #4: key loading reconciliation via pre-established keys... 9 Recommended best practices...11 Conclusion...11

3 Introduction This paper illustrates key concepts of data reconciliation and the critical role it plays in the creation and enablement of a successful configuration management system (CMS). This topic is especially relevant for data architects who are responsible for the data layer of the solution including class models, reconciliation, mapping between these class models, and the governance process of changing the models. In a CMS, the integrated configuration management database () stores only core service data; extended service data is stored in various federated repositories of the CMS. The integrated functions as the central coordinator that drives decisions about data location, ownership, tracking, and updates to help the CMS determine when, where, and how to pull together the data on demand. How then does a CMS know what data to consume, which data providers to trust, which data to accept, and how to handle conflicting data? The process of reconciliation handles these complex tasks, which enables the to compile accurate configuration items (CI) on demand from different consumers. Reconciliation occurs in every federation, query, discovery, or synchronization process in the CMS. This process is a key technical enabler that must be well thought out and deployed for a successful CMS implementation to effectively manage the IT infrastructure, processes, and services to deliver desired business outcomes. This paper also highlights HP s approach and best practices in architecting and implementing reconciliation in a CMS. A reconciliation analogy: automobile manufacturing assembly An interesting way to understand reconciliation is to think of an automobile manufacturing plant. As the cars roll down the assembly line, automated robots assemble different parts of the car. If the robot has access to multiple suppliers providing different parts, it becomes the robot s task to pick the correct supplier for the correct part, as well as to ensure that the correct part is assembled at the different points of assembly. Similarly, to assemble a CI requested by a consumer application, the CMS reconciliation process must consider two aspects: 1. Where do I get the data from in the CMS? This question is resolved through conflict resolution. 2. How do I pick the correct data from the source I have selected and assemble it with my CI? This issue is resolved through identity resolution. Let s look at how conflict and identity resolution work to provide data reconciliation in a CMS. 3

4 Conflict resolution Conflict resolution is the process of selecting the correct data provider for a given data attribute when presented with multiple data provider options within the CMS. As IT organizations grow over time, overlap between applications becomes unavoidable and, as a result, the same data is often available in multiple repositories. Different consumers usually pick one of the repositories and start to trust that repository as authoritative. However, due to the introduction of newer technology, company policies, or better discovery tools, consumers get access to other sources of the same data. Each of these alternative sources may provide different data values, quality, or even different update cycles. The question becomes which source should the consumer use? There are two approaches to resolve this conflict: 1. Collect raw data from every possible source into one central repository, wait until all data is collected, and then feed all data into one central reconciliation engine to determine the correct data set. For the next required data set, repeat the same cycle. 2. Alternatively, determine one repository that is best suited to accurately provide the correct data value. When the data is required, go to the pre-selected provider to obtain it. HP recommends the second approach which is to ensure that each data attribute has its own authoritative data provider. This data provider is determined when designing the CMS architecture and is known as the single authoritative source of record (ASOR) for that particular data attribute. Often, the ASOR is the repository that owns the attribute and therefore is responsible for the accuracy and integrity of that attribute. This distinction is helpful when choosing between two apparently authoritative data sources. This approach not only reduces network traffic between the requestor and multiple providers for each data request, but also eliminates numerous computing cycles to compare several copies of the data to decide which one wins. Once the request has been made to obtain the data, reconciliation then ensures that the correct data is identified through identity resolution. Identity resolution Identity resolution is the process of selecting the correct data attributes for a given consumer CI request. This is achieved by matching the same CIs in different repositories. For example, a CI request for a particular service may require the Universal to pull together different attributes from multiple data providers. An identity context needs to be established for each of these repositories to pull together data that is consistent, accurate, and reliable for that particular service CI. Without this common context, there is no reliable mechanism to tie the different data attributes from multiple data providers together. Identity matching can be simple or complex depending on the nature of the data required for a given CI request. Some identity resolutions can be achieved by simple key-matching between unique identifiers think of a database JOIN concept between two tables to find the appropriate unique record. Other identity resolutions require complex computation in addition to key-matching. For example, a person s first and last names (which is not enough data), plus the person s employer s name (which still is not enough data), and the person s manager s name (which is probably enough data). If unique data is embedded in other data or split across multiple attributes, reconciliation must support transformations and conversions between data models. These conditions arise from the fact that different data stores may store the same data in multiple formats. For example, a person s first and last names may be stored as two separate attributes in one data model ( John, Smith ), and as a single attribute ( John Smith ) in another data model. In order to reconcile these, either the two names need to be combined, or the single name needs to be split into the first and last name. 4

5 HP s CMS reconciliation strategy supports both types of identity resolutions as illustrated in the use cases section. Reconciliation is a critical enabler of a good CMS, and poor capabilities in this component of the architecture can result in many inefficiencies and problems such as: Increased network and storage usage due to same data attributes stored in and transported across multiple repositories in the CMS Time wasted in investigating the accuracy and validity of data before using it Increased computing resources required to resolve frequent data integrity conflicts Let s now take a look at how the HP CMS reconciliation architecture enables the reconciliation strategy described above. HP CMS reconciliation architecture Reconciliation in the HP CMS can be characterized by describing the interaction between the, the, and the Data Providers, with the interactions being carried out by the Conduits. Core service data Extended service data DDM discovery Federation API ETL (Connect-IT) Conduits Project and portfolio Data providers Third-party integration Application quality LDAP Service desk Asset Business service automation BSM A consumer application initiates a CI request. For example, a service manager application requests the operational state of a service CI. The integrated acts as the broker for the CI request. It receives the CI request from the consumer application and executes the steps required to respond to the request. It knows the ASOR for each data attribute and the multiple protocols and methods to obtain this data. Its core reconciliation engine performs the reconciliation needed in addition to the distributed reconciliation processes. Finally, it puts together the final CI and responds to the consumer request. 5

6 Conduits Conduits are mechanisms that act as bridges between consumers and providers via the Universal. Conduits are built on specific technologies such as Discovery and Dependency Mapping (DDM), federation adapters, and APIs. Additional communication layers build on these as well, such as ETL (Connect-IT) and Java -based clients. Each of the conduits may themselves perform reconciliation at transportation time, in addition to the core reconciliation engine in, depending on the reconciliation use case. Data providers Data in an HP CMS is either stored in the integrated Universal itself or in other data providers in the CMS. Core service data is stored in the Universal or non-federated providers that replicate their data into the Universal on a periodic basis through DDM discovery, API calls, or ETLs (Connect-IT). Extended service data is stored and dynamically accessed from the federated providers at run time through federation adapters and services. Data providers Replication through ELT (Connect-IT) Asset center IT Financial Management SW/HW asset management Contracts, ownership, financial, licenses, etc. Federation Network management center Optimize network availability, performance, and bandwidth Network performance, configuration data Replication through DDM discovery Server Replication through API Third-party product Domain-specific metadata, process outcomes Let s now look at the selection process of authoritative sources for data attributes, the most critical choice point in the design phase of a CMS. Conflict resolution by selecting the appropriate authoritative source of record Going back to the automobile-assembly analogy, as discussed previously, let s assume that the decision to pick the correct supplier is left to the robot at the point of actual assembly. Each time the robot assembles a part, it becomes the robot s task to pick one of the suppliers and the correct part at the time of assembly. In this scenario, it becomes the robot s responsibility to implement quality control for the part. Some parts may fit well while others may be mismatched by color, size, or type. To find the correct part, the robot needs to sort through multiple parts, which slows or even stops the assembly line. However, if the robot knew before actual assembly time which supplier to get the part from, the operation would execute flawlessly each time. In case of a quality issue, it would be simple to pinpoint the offending supplier and rectify the issue quickly. For this to happen, a proactive, upfront approach in the planning phase of the automobile plant is imperative. At the time of designing the automobile plant, correct specifications, time, quantity, and other attributes of each of the parts need to be described to different suppliers. Depending on the supplier s ability, an agreement then needs to be documented which states: If you can give me the correct part at the right time in the right quantity, I will trust you as my authoritative source of car parts. If your error rates are above acceptable limits or if you are unable to supply the right parts at the right time, I will drop you as my preferred supplier. By planning ahead so that only the correct part arrives at the time it is required, we remove the need for the robot to sort through and choose the correct part from numerous suppliers at assembly time. 6

7 Similarly, a reconciliation strategy that picks every possible data attribute available and feeds it into one central über-reconciliation engine to make the appropriate decision each time, leads to degraded reconciliation performance and quality as the size and complexity of the CMS grows. Throwing more processing power at the problem may decrease the wall-clock time required but does not improve the quality of reconciliation. HP s approach to reconciliation is to determine the best ASOR for each data attribute when designing the CMS architecture. The real owner of each attribute is identified before adding it to the CMS. This approach avoids elaborate conflict-resolution processes at run time. Use cases are exercised to choose the appropriate ASOR for each attribute. Exceptions to this rule are treated as exceptions there should be exactly one authoritative source for any given attribute. If there are possible multiple sources, it is imperative to identify which source to defer to, and under what conditions. This approach ensures de-duplication and cleans up data sources at selection time. For example, between HP DDM and HP Network Node Manager (NNM), HP NNM would generally be identified as the ASOR for providing network devices information. As NNM provides additional depth and network management capabilities, it is the logical choice as the ASOR for networks and network devices. NNM patterns should be used to discover network devices and DDM default patterns for discovering network devices not activated. ASOR Federation Network management center Optimize network availability, performance, and bandwidth Network performance, configuration data DDM discovery Server An ASOR should be unique in scope and business accountability for the data it provides. If two provider data sources appear to be equal in all respects, the correct ASOR is the one that is accountable from a business perspective for maintaining accurate records. 7

8 Use cases The four use cases described below illustrate different aspects of reconciliation. As conflict resolution is alleviated at the time of designing the CMS, these use cases describe identity resolution at run time. Use case #1: reconciliation of DDM data Core service data in the Universal is updated periodically by DDM discovery. As changes occur in the CMS, these are discovered by the discovery probes and sent back to the Universal. Internal Universal reconciliation engine DDM discovery Server Data provider For example, assume that a CI exists for a server Hiawatha and that the host-name is the key for that CI. If an OS patch has been applied to the server since the last discovery, the new discovery cycle will report the update back to the Universal. Without any reconciliation mechanism, a duplicate CI would be created due to the difference in an attribute value (OS patch level). However, the mechanism of identity resolution ensures that the existing CI is matched with the new discovery through host-name key comparison and the existing CI is updated with the new attribute value. Use case #2: reconciliation in a federated scenario In the example illustrated above, the CI is put together by the Universal by federating data attributes from different data providers. Data provider Asset center Memory: 8 GB Hostname svr1 IP: OS: HP-UX Memory: 8 GB Status: Red Third-party product Hostname svr1 Status: Red IP: OS: HP-UX Memory attribute ASOR: asset manager Status attribute ASOR: third-party product IP attribute ASOR: network node manager Internal Universal reconciliation engine Hostname svr1 Memory: 8 GB Status: Red IP: Reconciled IP: Network management center Hostname svr1 IP: Mac address: xyz Hostname: Niles 8

9 If the ASORs for the attributes are not determined, the Universal needs to query every provider for each data attribute and process every returned value through the reconciliation engine to decide the winner. If there is a failure, further queries and reconciling cycles are needed. The cycle is then repeated for each attribute in the CI. Clearly, when scaled up, this approach generates an incremental load on network traffic, reconciliation cycles, and data providers. However, per HP s CMS reconciliation strategy, each attribute required for compiling the requested CI is pulled only from its own ASOR any other provider in the CMS containing the same attribute is ignored. For example, even though the IP attribute could be pulled from each of the three providers, it is only federated from NNM, its ASOR, determined at the CMS design phase. This bypasses the need for IP attribute conflict resolution at run time and reduces processing cycles in the core reconciliation engine. The reconciliation engine only needs to assemble the correct attributes together as opposed to spending several cycles trying to decide the best value for each and every attribute from every possible provider in the CMS. Only identity resolution is performed as the host-name matching is done with each provider to pull the right data set from each of the sources. Use case #3: reconciliation by an API A data provider itself may invoke an API to push a CI creation, update, or deletion into the Universal. In this case, reconciliation works in the same manner as discovery reconciliation based on matching data values. To maintain the integrity of the CIs in the Universal, the reconciliation process compares one or more attribute values, and if necessary, business logic is also executed. The CI is then appropriately created, updated, or deleted. 1. CI create, update, or delete PUSH 2. Universal reconciliation Third-party product Internal Universal reconciliation engine Data provider Use case #4: key loading reconciliation via pre-established keys Key loading is a method to speed up reconciliation at operational run time by establishing a pre-determined handshake between the integrated and a data provider. Normal reconciliation Internal Universal reconciliation engine 3. Identity reconciliation, business logic to retrieve correct data 1. CI query 2. Extended data request Third-party product 6. Query response 4. Extended data response 5. Universal reconciliation Data provider 9

10 This handshake is established by inserting unique Universal IDs into the data provider as foreign keys (foreign key population). Here, Connect-IT acts as the key loading engine, populating unique Universal CI IDs into the data provider. Key loading should occur periodically to ensure any data changes in the provider have up-to-date foreign keys. Key loading Internal U reconciliation engine Foreign keys inserted into federated data provider Foreign key: CI ID Network node manager Data provider At query time, the Universal only needs to use a single key the unique CI ID to retrieve the extended data from the federated data provider and bypass the latency of the reconciliation (identity resolution) process itself. Reconciliation at run time (after key loading has taken place) 1. CI query 6. Query response Internal Universal reconciliation engine 2. Extended data request 4. Extended data response 5. Universal reconciliation 3. Identity reconciliation, business logic to retrieve correct data Foreign key: Cl ID Network node manager Data provider Reconciliation is efficient and immediate, improving consumer response time and reducing load on the data provider. Note that an algorithm is still needed to match provider records with the identity of the CIs in the Universal. Reconciliation cannot be cheated through key loading; it simply provides a faster query at consumer query time. It is important to consider data latency tradeoffs when implementing key loading. As key loading is performed periodically, data in the provider may change between the key loading cycles. If a query is made before the next key loading cycle is run, the data returned may not be accurate or current. Therefore, though key loading provides faster query responses at run time, real-time reconciliation is the only option for real-time data needs. 10

11 Recommended best practices 1. Incorporate use-case exercises during the CMS planning phase to design your reconciliation strategy. Reconciliation works best when heavy emphasis is laid at the planning phase. This helps reduce reconciliation processing cycles at run time, as well as reduce the cost of maintaining reconciliation rules. 2. Determine one ASOR for each data attribute in the CMS. This enables data reconciliation to trust and connect only to that source for the required data avoiding conflict resolution during run time. 3. Every CI and attribute in the CMS should have an owner. This ownership should be specifically established at the attribute level and not at the CI level. Any attribute without an owner should be considered a nonauthoritative attribute. 4. Re-evaluate the validity of ASORs periodically. As the CMS matures, new data providers may be added or dropped out of the CMS. It is important to ensure that the required data is provided by true ASORs at any given point of time. Conclusion Technologies that accept data from every possible source and process it through batch-type reconciliation cycles in a central über-engine fail to scale up as the number of CIs grows in the or the CMS grows in complexity or size. Reconciliation loads tend to grow over time and it becomes an uphill battle to manage increasingly lengthier computing cycles to reconcile data. Even with this expensive approach, data may not be reliable, up to date, or readily available when needed by a consumer in the CMS. HP s CMS reconciliation strategy is distinctly geared to be a key enabler of a well-functioning and efficient CMS. First, HP CMS s reconciliation strategy strongly emphasizes the need to rationalize data providers that act as authoritative sources for configuration data. Most attention is paid to this aspect of the CMS architecture at the planning phase of the lifecycle through use-case exercises. This strategy acts as a gatekeeper for those data attributes into the reconciliation engines whose validity is ensured by the authoritative sources themselves. Second, the distributed architecture and flexible configurability of the reconciliation components enables real-time reconciliation in the core reconciliation engine and various conduits that connect the CMS consumers, Universal, and its data providers. It avoids overloading the core reconciliation engine and enables scalability as a CMS matures. Third, has a number of productized integrations with other HP Business Technology Optimization (BTO) software and third-party products for which the HP CMS reconciliation architecture is well defined. This allows data architects to implement these components without any additional efforts or investments on reconciliation other than making modifications for their unique situations. HP CMS reconciliation enables actionable federation for a real CMS whose reconciliation capabilities are scalable, efficient, and fast with a high quality of data integrity. These capabilities enable active reconciliation of data when CIs are added, updated, or deleted, which provides that data is always valid, current, and up to date when the business needs it. 11

12 Get connected Get the insider view on tech trends, alerts and HP solutions for better business outcomes Technology for better business outcomes To learn more, visit Copyright 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Java is a U.S. trademark of Sun Microsystems, Inc. 4AA2-8202ENW, December 2009