Data Validation for Smart Grid Analytics

Transcription

1 Key Concepts This white paper presents several key concepts behind quality and validation as implemented in the industry. Typical validation mechanisms and workflow targeting automatically and manually acquired as well as calculated are also described. Data is the most important asset in information systems. In order to enforce quality it is mandatory to implement rigorous validation procedures and mechanisms. In the scope of this paper, validation can be, on one end, a simple process as basic as bound checking or, on the other end, can involve statistical algorithms and sophisticated heuristics. Typically, validation is implemented through a set of business rules that includes, but is not limited to, the following key points: Detection of missing during acquisition; Detection of values greater or less than specified limits; Detection of variations (ramp) that are greater or less than specified limits; Assignment of a quality index, a status index, or any other indicators to every piece of allowing assessment of the quality and integrity of the. It is also very important that users have visibility on the level of quality. This is achieved by associating quality codes and status indices to every critical managed by an information system. This allows users to quickly assess the quality and to understand where specific is in its lifecycle. In this context, users can use with confidence or make informed decisions when faced with questionable. Data is the most important asset in information systems Users must have visibility on the level of quality Data quality is defined as a combination of accuracy and precision Data status represents the state of the in the diffusion and publication process Data Quality Data quality can be viewed as the combination of accuracy and precision. Accuracy is defined as the degree of veracity of a measurement while precision is the degree to which a measurement can be reproduced. For instance, precision for a water level can be / meters where the tolerance of +/ is the known, because the target value is unknown. For example, a measure of / meters is considered very precise. But if there is a bias of two (2) meters in the level gauge, then the real measurement should read In this case,

2 / is very precise but highly inaccurate. Data accuracy can be improved through a defined business process. Accuracy can be determined by taking the average of several measurements. In addition, the known bias of equipment can be compensated by calibration. Data precision comes from the physical equipment. In scientific measurements, the precision of every measuring device is known and therefore it is possible to determine the precision of measured or computed. Data precision is measured through a quantitative value (i.e. +/ ). Information systems generally do not store precision; precision is not visible to the end user. Therefore, another type of mechanism needs to be in place to propagate this variance from acquisition through all computation processes and make it available to the end users. Instead of measuring quantitatively the precision of, a qualitative index, the Quality Code, representing the accuracy is associated with all, allowing for implementation of rules to deduce the accuracy of any computed. Table 1 shows typical quality codes that could be associated with. They are presented in increasing order of significance. During implementation, a numerical value representing its relative weight is associated with each of these quality codes. Table 1: Typical Quality Codes Description of Qualities Codes Missing Data Doubtful Data Out of bounds Estimated Data with Quality corrected Valid Data The general rule is that any new incoming can replace already stored providing that its quality index is greater than or equal to the quality index of the stored. Thus at any time, missing can be replaced by doubtful that, in turn, can be replaced by out of bounds, and so on. Inversely however, missing cannot replace doubtful which could not replace out of bounds. It should be noted that valid can replace corrected and corrected can replace valid. A more detailed explanation of this process is given in section 0 (Data Quality and Validation Workflow). Data quality is also propagated to calculated. The general rule is that the resulting quality will always be equal to the lowest quality index of the used in the calculation. For example, given A with a quality code of Valid Data and B with a quality index of Estimated Data, then performing C = A + B, will give a quality index of Estimated Data for C.

3 Data Status In addition to the quality index, may have a status index that represents the state of the in the diffusion and publication process. Table 2 shows typical status indices that could be associated with. They are presented in increasing order of significance. Each of these indices has a numerical value representing its relative weight. The table also shows three main categories regarding the way could be entered in the system: automatic acquisition of new, calculation of new, and manual entry of new. Table 2: Typical Status Indices Description of the Status Index Acquisition - Raw Status Acquisition - Validated Status Acquisition - Manual Validated Status Acquisition - Published Status Calculated - Raw Status Calculated - Validated Status Calculated - Published Status Manual Entry - Raw Status Manual Entry - Validated Status Manual Entry - Published Status Independently of how is entered in the system, can have the following status: raw, validated or published. The general rule is that new incoming can replace already stored providing that its status index is greater than or equal to the status index of the stored. Thus, raw can be replaced by validated which could in turn be replaced by published. Inversely however, raw cannot replace validated which cannot replace published. A more detailed explanation of this process is given in section 0 (Data Quality and Validation Workflow). Data status is also propagated to calculated. The general rule is that the resulting status will always be equal to the lowest status index of the used in the calculation. For example, given A with a status of Acquisition - Raw Status and B with a status of Manual Entry - Published Status, then performing C = A + B, will give a status of Calculated - Raw Status for C. Many sources do not provide quality or status indices with their. Such sources are Web sites, OASIS systems, some external applications, etc. In the implementation of the quality and validation workflow, it will be mandatory to assign quality and status indices to these. Rules should be defined during the analysis phase of the project to perform this task. Data Quality and Validation Workflow A typical validation workflow for automatic acquisition and manual editing is described here to illustrate how quality and status indices may be used and processed. Figure 1 illustrates the typical workflow for processing the quality index and status index of acquired. Automatic acquisition involves incoming from various sources (SCADA, Web sites, manual entry, etc.) as well as validated coming from a validation process. Incoming can be new or it can be an already acquired that is being automatically sent again or manually re-entered.

4 In these examples consists of three fields: a value, a quality index and a status index. When is manually re-entered or automatically reacquired, any of the three fields may change. In the case of acquisition or validation, rules described in the previous sections apply as they are. Thus, when new is acquired, the following validation rules are automatically applied. Start Acquire new Same already exists Store new New status > New status = New quality < New quality >= Reject new Trigger an alarm Stop Figure 1: Quality index and Status Index Processing Rules for Data Acquisition

5 The status index of the new is compared to the status index of the existing. If the new status index is heavier than the existing one, the existing is replaced by the new one. If the quality index of the new is lighter than the quality index of the being replaced, an alarm is triggered. If the new status index is lighter than the existing one, the new incoming is rejected. In the case of two identical status indices, if the quality index of the new incoming is heavier or equal to the index of the existing, the existing is replaced. If not, the new incoming is rejected. Figure 2 illustrates the typical workflow for processing the quality index and status index of manually edited. Manual editing, involves manually entering new values for existing, following validation procedures. Start Manual edition of existing New status > New status = New quality >= New quality < User have clearance Trigger an alarm Trigger an alarm Reject new Stop Figure 2: Quality Index and Status Index Processing Rules for Manual Data Editing

6 In the case of manual editing of, the same rules for automatic validations apply differing only in that the operations are performed by humans. Thus, rules can be broken providing the user editing the has the necessary clearance. In any cases, alarms must be raised if general rules are broken. All having the status index Validated Status can replace other with an inferior quality index but an alarm must be triggered. All having a status index Published Status can replace other with an inferior quality index but an alarm must be triggered. All having a status index Published Status can replace other with a status index to Published Status but an alarm must be triggered Again, many sources do not provide quality or status indices with their. Such sources are Web sites, OASIS systems, some external applications, etc. In the implementation of the quality and validation workflow, it will be mandatory to assign quality and status indices to these. Rules should be defined during the analysis phase of the project to perform this task.