Integrated UV-Vis parameters for distribution network monitoring

Transcription

1 Techneau, 13. October 2009 Integrated UV-Vis parameters for distribution network monitoring & Redesigned Monitoring station and central station for monitoring of integrated UV-Vis parameters

2 Techneau, 13. October 2009 Integrated UV-Vis parameters for distribution network monitoring & Redesigned Monitoring station and central station for monitoring of integrated UV-Vis parameters 2006 TECHNEAU TECHNEAU is an Integrated Project Funded by the European Commission under the Sixth Framework Programme, Sustainable Development, Global Change and Ecosystems Thematic Priority Area (contractnumber ). All rights reserved. No part of this book may be reproduced, stored in a database or retrieval system, or published, in any form or in any way, electronically, mechanically, by print, photoprint, microfilm or any other means without prior written permission from the publisher

3 Colophon Title Integrated UV-Vis parameters for distribution network monitoring & Redesigned Monitoring station and central station for monitoring of integrated UV-Vis parameters Author(s) J. van den Broeke (scan Messtechnik) Quality Assurance C. Moldaenke (BBE Moldaenke) Deliverable number D & This report is: PU = Public

4 Summary For a long time operators of larger monitoring networks have suffered from the absence of software that allows quality-controlled and validated operation of sensors in order to reliably turn data into information. The need was firstly identified in process control applications where drifting sensors cause unnecessary costs. The need became urgent in event detection and water security systems that cannot work using unreliable, poorly-documented sensor data without constantly causing false alarms. Last, but not least, such software is required to provide wider acceptance of online sensors in the fields of compliance monitoring for drinking water, waste water and industrial environments. Previously, sensor and terminal operation software has not adequately addressed any of these needs. The deliverable describes the development of a software package that integrates sensor and station management, data validation and event detection in a modular software environment. This software tool has been optimised to utilise the high density information provided but spectral measurement but is open to include any type of instrument from any manufacturer. All events on a monitoring station are recorded (including service and maintenance), the produced results are validated online and then they are made available to a number of both simple as well as advanced event detection algorithms. The software described has been developed up to a prototype stage and initial results will be obtained in TECHNEAU Work package 3.6. TECHNEAU October 13, 2009

5 Contents Summary 2 Contents 3 Description of Workpackage 3.5, component Introduction General Introducton to this deliverable State of the art in December Progress in other Workpackages Progress towards Objectives 7 2 Background to Current Developments 8 3 Methods Sensor- and Stationmanagement Clean Data Data validation Reliable Alarming - Event Detection Initial test results 20 4 Future Work 22 TECHNEAU October 13, 2009

6 Description of Workpackage 3.5, component Development of a new alarm system for detecting quality changes during distribution The task of this component is the combination of the new methods on-line delta spectrometry and on-line integrated UV-Vis-Spectrometry and the evaluation of a new warning and alarm system for drinking water distribution networks. In order to prove practicability of this theoretical concept and advantages compared to conventional methods in real world case under diverse conditions the primary targets of this activity has to be: develop a monitoring system consisting of several monitoring stations located close to the points of water access ( tap ) building up a monitoring network develop monitoring stations being ready for operation within shortest time and providing automated data transfer to one central station develop one portable monitoring station building up an alarm system by linking up these monitoring stations to a central station develop a central station for centralised data storage, handling and access, for providing remote access to several monitoring stations for alarming and for calculating virtual parameters by comparison of parameters of several monitoring stations web application providing access to database of central station research for comprehensive virtual parameters providing distinct information needed for operating distribution networks with respect to drinking water security TECHNEAU October 13, 2009

7 1 Introduction 1.1 General Introducton to this deliverable The main goal of the s::can contribution within work package 3.5 is the development of all components required for setting up networks of UV/Vis spectrometry based monitoring systems. In such a network, independent stations provide their measurement data, automatically, to a central station. Using the information collected at all stations, an event detection system for quality monitoring in distribution networks will be set-up. In order to make such a network reality, a number of components needed to available. The development and / or optimisation of these components were addressed in the s::can activities within work package 3.5: 1) sensors and terminals for building the monitoring stations 2) suitable algorithms to analyse data and detect water quality changes in real time 3) a central station that collects the data from all monitoring stations and makes it accessible The current deliverable describes components 1 and 2. They are described in a single deliverable (i.e. this report); instead of in two separate deliverables. This is done because their conception and development was intertwined and discussing them in a single document is more efficient and prevents repetition. The knowledge and experience required for the developments described here should have been, partly, the result of operating a first version of a monitoring network at a demonstration customer. As the progress with installation at this customer was not quick enough to allow use of the data within the timeframe of this work package, it was decided to use information from other installations as the basis for the development described herein, knowing that this reflects experience from customers worldwide as well as experience gained in WP3.2, WP3.3 and WA7 of the TECHNEAU project. Component 3 was described in deliverable State of the art in December 2005 At the start of the TECHNEAU project, several components that are necessary for the realisation of spectrometry based monitoring networks were unavailable. First of all, there was no suitable data transfer method available in the s::can instruments that could transport the data amounts generated by the spectrometer probes. A further issue limiting the use of spectrometer probes in a monitoring network was the unavailability of software that can store, TECHNEAU October 13, 2009

8 handle and display multidimensional datasets. Central station software, for displaying time series and storing measurement results, were in existence, but only suited for handling small amounts of two dimensional datasets (the two dimensions being time and measurement value, for example ph). However, the s::can spectrometer probes produce spectra, which when stored and plotted against time present 3-dimensional datasets. Additionally, from these spectra parameters are calculated, using mathematical algorithms neither available nor implementable in existing visualisation software. Therefore it was decided to develop such a central station. This development was described in deliverable Secondly, despite the availability of a wide range of sensors for water quality monitoring, there was no efficient system on the market that composes an overall water quality image based on all these inputs. The state-of-the-art was the use of the data streams from all instruments separately and using only one or two main indicator parameters, if any, for online quality monitoring. These indicator parameters were typically monitored manually by the operators, and automatic alarm generation was practically limited monitoring static thresholds for single parameters. An important reason for this limited use of online measurement results was the low data quality often produced by online measuring instruments. This low quality has very diverse reasons, including outright instrument performance but equally important instrument installation issues, maintenance and service activities (or lack thereof) and many more. The overall result is a low confidence of plant and network operators in online measurement results. This is not helped by the legislative demands on quality monitoring, often prescribing grab sampling on a monthly basis. The fact that data are often of poor quality leads to frequent false alarms in any event detection system or control system that depends in this type of input. False alarm rates can be reduced by combining different parameters into one, virtual, alarm sensor. This allows detection of errors in single sensors through redundancy. Furthermore, assembling an overall picture using multiple complementary sensor technologies further reduces the chance of false alarms. Documenting the status of the monitoring stations and their sensors and feeding this information into the monitoring system will improve data quality and system reliability still further. Maintenance, calibration, installation issues, they all produce changes in measurement results. When such operations are not automatically taken into account, they will trigger false alarms. False alarms reduce the confidence in an alarm system and also mean that the operator will reduce the sensitivity of the system to reduce the frequency of false alarms, thus defeating the purpose of the alarm system, which by definition is meant to be a sensitive detector of changes. TECHNEAU October 13, 2009

9 The challenges of integrating multiple data streams into virtual alarm sensors as well as improving the quality of data coming from sensors and monitoring stations is addressed here Progress in other Workpackages First steps to evaluate data coming from multiple sources were described in deliverable This deliverable described the use of classification alarms where combinations of spectral parameters were used to recognise changes in source water. This work is expanded upon in this deliverable Progress towards Objectives In this deliverable a new approach towards ensuring data quality is presented. Furthermore, the use of multiple data sources as well as multiple evaluation procedures to detect water quality changes is evaluated. This includes both spectral data as well as data from any other sensor. In order to implement these processes in real products, the operating software for the spectrometer probe was completely redesigned and the new data analysis algorithms implemented. The focus in this deliverable was solely on operational concepts and the first steps in development of a software package that supports these concepts. Although the title might suggest hardware development is included, it was deemed that the current generation of sensors and industrial computers (optical and solid state sensors as well as biomonitors) are sufficient for the tasks demanded from them in a distributed early warning network if the software to support them and evaluate the data would be available. TECHNEAU October 13, 2009

10 2 Background to Current Developments For operators of drinking water treatment installations and distribution network, it should be a natural interest to know as well as possible the actual status of the installation as well as the processes taking place. To perform monitoring and control in an optimal way requires a huge number of measurement values that are continuously being refreshed. Until recently, the only way to get reliable online data was by the use of hugely expensive chemical analysers. As a result, real time measurements have remained limited to a small number of the most basic parameters. Over the last decade, more and more affordable online sensors have become available, leading to ever increasing acceptance of online water quality monitoring. The use of such instruments allows the evaluation, in real time, of concentrations of substances present in the water as well as the overall water matrix. This in turn allows control mechanisms that are optimised for and respond to the actual process conditions, instead of settings based on less than ideal outdated information or even purely based on operator instincts. Not only can the system be optimised, in this way it is also possible to detect failure of processes immediately, and thus reduce and completely prevent damage that would be caused by this. Because the amount of data that is available has increased dramatically, it is not possible to manually verify that all these data are plausible and of good quality. For automatic interpretation to function, the data in themselves must be of high quality. This will only be possible when procedures for automatic data validation and station sensor management are used. Without validation, any result will be fed to the data interpretation system (either alarm system or process control), and faulty results will be evaluated and will be awarded the same weight as good results (Figure 1). Figure 1: The effect of measurement noise on process control algorithms and alarm systems TECHNEAU October 13, 2009

11 The need for sensor and station management is directly linked to automatic data validation. The validation aims to detect incorrect results and prevents these from being fed to the data evaluation and interpretation processes (Figure 2). However, not all sensor issues that can occur can be detected and / or identified by validation alone. In many cases, it is necessary to directly include information on activities on the station. For example, calibration of an instrument can lead to a change in the results coming from that instrument, but a validation algorithm will need to be informed about the fact that the instrument was calibrated, as the resulting change could be due to any number of causes, not just calibration. Figure 2: Process control and alarm systems including sensor management and data validation. The depicted set-up represents the one developed here. Up to now, no integrated system for data validation, sensor and station management exist in the field of (online) drinking water monitoring. The market for online monitoring is the playing field of a very large number of very diverse companies. Most of the players have focussed on one particular part of that market. Some larger consulting companies have integrated multiple elements as mentioned above, but only in large scale projects e.g. during construction of new treatment plants. No solution able to deal with the processing and quality control of online measurement results exists that can be easily implemented on a monitoring station level, a monitoring station being anything from a single sensor and its controller to a network of distributed remote sensors. The development performed in this work package aims to provide the elements needed for such a system, including sensor and station management, data validation as well as data evaluation, the latter being focussed on detection of water quality events. In order to make this a universally applicable tool, it was designed with a particular range of instruments in mind, but remains fully open to integration of any sensor from any manufacturer in the future. TECHNEAU October 13, 2009

12 3 Methods The goal of this deliverable is a tool that will help increasing the availability and quality of measurement data. This is to be achieved through a modular software concept. Furthermore, this tool should be able to evaluate multiple data streams and use multidimensional data in order to evaluate water quality and changes therein. The concept looks at the following aspects, each aspect being represented in the software by a separate module: Module 1: Sensor- and Station management Module 2: Data validation Module 3: Event detection The separate modules are described in the following sections. 3.1 Sensor- and Stationmanagement The most important task of this module is to provide transparency. The sensor and station management software will make available to the user all the information necessary for operation of the measuring instruments. All events and actions on the station, either concerning the entire station or those concerning single instruments (such as maintenance, calibrations, configuration, etc.) is logged and stored in a database on the station. All the data is available for review to the operator, but is also available for the validation software (see section 3.2) where it will aid in evaluation the quality and reliability of measurement results. An important feature of the sensor and station management module is to enforce many procedures, such as entering of data on critical points as well as providing menu driven procedures such as calibrations, i.e. the user will need to enter specific information before he can proceed. As the software will recognise malfunctions (e.g. communication lost, signal out of range, membrane life expired) automatically, it will actively request information from the operator. This means that documentation of critical manipulations, from user identification and authorisation to logbook keeping, are not left to the individual users and their precision in documentation, but are determined by the system itself. Because comparable systems for quality management are not yet in use, it regularly happens that measurement results can not be interpreted as a result of very basic but untraceable causes. 3.2 Clean Data Data validation The data validation is the key functionality in the software concept described in this deliverable. Whereas event detection is an available feature of various commercially available products, online data validation has not been implemented in these systems. The data validation module has the task to TECHNEAU October 13, 2009

13 automatically detect, mark and (optionally) correct, untrustworthy data. This evaluation will provide information on the functioning of individual measurements / sensors in the system. This information is then utilised in various ways, for example to provide the user with indications that a sensor requires maintenance as well as automatic detection of malfunctions. This automatic detection will facilitate more rapid detection of problems with instruments, and when action is taken using the cues from the software the availability of the sensors as well as the data quality will be increased. Furthermore, through the marking of questionable results the subsequent data processing knows which results to use and which to ignore or allocate a lower importance. The use of automatically corrected results is available mainly for process control, where loss of signal can lead to incorrect settings in the process. Correction allows the controls to continue working properly. For the validation a number of simple but robust statistical methods have been applied, which analyse the data streams. The data are screened for four different characteristics: 1. Outliers An outlier is an observation that lies outside the overall pattern of a distribution. To detect such observations, a smoothed curve is projected on top of the last measurement values. Using the smoothed curve, it is possible to estimated the deviation of the actual reading from the previous results. This difference is then compared to the differences between the previous measurement points and the smoothed curve. Using the information from the previous differences a tolerance bandwidth is calculated, which follows the changes in the measurement values and which becomes narrower with higher measurement precision. When a value lies outside of this tolerance bandwidth it is classified as outlier (Figure 3). By definition, outliers are single events or events with a very short duration. Therefore the algorithm assumes that more than 2 subsequent measurements that are classified as outliers indicate a real change. In such a case only the first value is marked as outlier, the other data points are treated as an event (see also section 3.3). TECHNEAU October 13, 2009

14 Figure 3: Outlier detection in the data validation module. 2. Discontinuous Measurements In case of a discontinuous measurement series, the entire series shifts by a discrete value. This is something that for example can happen when a sensor is calibrated or when the water supply to the sensor is disrupted and it is measuring in air. In order to detect such discontinuities the results are screened for intervals where the average of the measurement values before a certain time differs significantly from the average of the measurement values after this point in time. The significance of the change in average is determined by measuring the signal to noise level of the results before and after the jump and verifying that the change is an order of magnitude larger than the noise level. The change in the average constitutes a jump height and using the average noise level a tolerance interval is calculated. If the height of the jump is larger than the tolerance interval, then a discontinuity in the measurements will be recorded (Figure 4). The jump is definitively recognised after a set number of values are at the new level. In case the values return to the original level within this timeframe, the change is not recorded as a jump. Jumps due to maintenance actions are filtered out by the input from the station management software, where calibration and maintenance actions are recorded. Jumps that occur but can not be traced back to maintenance actions can trigger an alarm (see also Paragraph 3.3) as they can trigger the Dynamic Alarm. TECHNEAU October 13, 2009

15 Figure 4: Analysis for discontinuities in data validation module. 3. Noise Noise is a measure for the random scattering of measurement results around a smoothed curve produced by using a moving average function on the same measurements. Such noise comes from instrument noise as well as slight fluctuations in concentration or activity of the analyte. In this data treatment no distinction is made between these two different types of noise. In the instrumentation for physical and chemical analysis, the instrument noise is typically higher than analyte noise in the timeframe used by the validation tool, i.e. over a time window of 5 30 minutes. In order to calculate the noise level, the average deviation of measurement results from this smoothed curve is calculated. This noise level is compared to upper and lower limits that are known to correspond with a particular sensor or application. In case the noise level is outside of these boundaries the measurement is marked as noisy (Figure 5). This means that an increase in noise is a trigger for rejecting data in the validation, which is most likely caused by poor functioning of the sensor evaluated. It also means, that a reduction in noise to practically zero indicates that something is not working properly. Concentrations in water always vary slightly and an instrument also always produces measurement noise. The absence of noise can indicate that the signal to the instrument has been lost or that the instrument is no longer submersed in/supplied with water. Therefore this will be used to trigger a change of the system status. TECHNEAU October 13, 2009

16 Figure 5: Noise detection in the data validation module. 4. Drift Drift is a long term continuous increase or decrease of the readings from a measurement device. To detect drift the readings are modelled using the Holt-Winters method. This produces a slope component and when this is significantly higher or lower than zero, a drift is detected. The time window for drift detection is significantly longer than that use for the other data validation tools; whereas tools 1 3 respond within minutes to changes in data, drift is only detected after increase or decrease of the values in a data stream is recognised over a period of 7 or more days. The statistical significance of the detected drift is then evaluated against the variations in the results and only drift substantially larger than the variations is considered as drift. Drift is the most difficult of the four validation parameters to analyses without human input. Long term changes in the water (for example seasonal effects on surface water, e.g. temperature or chlorophyll levels) can be impossible to distinguish from instrument drift. TECHNEAU October 13, 2009

17 Figure 6: Drift detection in the data validation module. 3.3 Reliable Alarming - Event Detection Only when data quality has been ensured and when information on system status and service and maintenance has been brought into the equation, it is possible to perform effective event detection. The software module for event detection that is part of the software concept introduced here evaluates measurement data that have been cleaned by the validation module, and determines the normality of these data and triggers an alarm when a significant deviation from normality is detected. It has been optimised for the utilisation of multi-dimensional spectral data, but will function just as well with single or multiple one-dimensional inputs from conventional sensors. However, the integration of spectral data provides a much more complete picture of water quality than can be obtained through single parameters, as the latter only provide a simplified projection of the multidimensional water quality onto a small number of simple parameters. For example: a completely normal change in water quality can have the same effect on ph as a toxic contamination, which should trigger an alarm. Single parameters have no way to distinguish between such causes for a change. The combination of multiple parameters into the event detection system, but especially the use of more advanced tools such as spectrometer data, substantially improves the resolution of changes into alarms and normal changes. By calculating derivative spectra across time and space it is possible to provide still further information that can be used as a basis for event detection. For event detection, the measurement results can be utilised in different ways; 1) to produce single alarm values based on single parameter for event. In this way a larger number of single alarm values is produced, 2) These single TECHNEAU October 13, 2009

18 values can be combined into pattern alarms which evaluate the overall status of the system instead of looking at single parameters. The methods for calculation of single parameter alarms consist of four different types: 1. Static Alarm: A static alarm detects where a single value or a group of values from a single parameter lie with the normal range of a parameter. This is done by verifying the value does not lie above or below static upper and lower limits. In case the readings leave this area of normality, the system produces an alarm (Figure 7). The range where a value is considered to be normal is defined during the configuration and runin period of the alarm system. The static alarm is the most basic alarm functionality available Figure 7: Static Alarm in the event detection module. 2. Dynamic Alarm: The dynamic alarm detects whether a value of a group of values shows a sudden change. Similar to the outlier detection described in section 3.2 a smoothed curve is calculated. Using this information a tolerance bandwidth is calculated and subsequently it is assessed whether the difference between this smoothed function and the actual measurements is within the tolerances or not. If the current measurement value is outside of these limits, a possible sudden change in measurement results has been detected. The selection rule TECHNEAU October 13, 2009

19 used to confirm a sudden change is complementary to the rule used to mark a result as an outlier: a single value outside of the tolerance limits is considered an outlier, multiple values outside the boundaries are considered to result from a sudden change. Such a change triggers the dynamic alarm (Figure 8). Figure 8: Dynamic Alarm functionality as implemented in the event detection module. 3. Pattern Recognition This alarm functionality uses the fact that in case of normality in the water matrix specific correlations between parameter values exist. This concept is best shown using a 2-dimensional example, as given in Figure 9 in practice more dimensions are used; a series of measurements is defined as reference points. This series should present an as complete as possible image of the normal state(s) of the system being monitored. Every parameter (including every wavelength in a UV/Vis absorption spectrum) constitutes one dimension in the space of measurement values. Every multiparameter measurement thus forms a single point in this space. As shown in Figure 9, the measurement points are not distributed homogeneously throughout space; they cluster around reference points which represent the normal operational conditions in the system. States that represent extraordinary situations are populated less densely. The normality of incoming data points is evaluated by calculating its distance to the nearest neighbour amongst the reference points. The closer a point is to a reference point, the more it resembles a known state of the system, and hence it resembles normality. A data point lies further away from known reference point is less normal. Again, the TECHNEAU October 13, 2009

20 accepted normal states are introduced in the system during a period of training. During the training the alarm limits are set based on the number of different states and the variability in the measurements. In order to eliminate effects of varying measurement ranges and scaling, all measurement values are first normalised on their standard deviation. The above mentioned training of the system can be performed in two different modes: either a dataset with a fixed time window can be used for initial training. Subsequently, new data collected can be added so that the database expands and different states of the water and different events are included in the database. The other possibility is training using a moving time window. This moving time window (e.g. a month wide) shifts constantly. Using this option the effect of long term changes, such as seasonal effects, are cancelled out and the alarm system can provide the highest sensitivity at all times. The disadvantage of this system lies in the fact that alarms that have occurred in the past are removed from the training dataset once they are outside of the time window. If these events were normal operational events and no real alarms, they will again trigger an alarm if they occur, because the system will not recognise them any longer. Figure 9 shows three different data points and their distance from normality as well as the resulting alarm value in a trained system. Figure 9: Pattern recognition TECHNEAU October 13, 2009

21 4. Spectral Alarms This alarm functionality is only applicable when at least one spectrophotometer probe is installed. If this is the case, this module analyses the UV/Vis spectra and checks for changes in the shape of the spectrum over time. A spectrum representing normality is defined during a training of the tool. Subsequently, the deviation from normality is assessed on the 1 st derivative of the spectrum. The first derivative is used as it reduces some of the noise on the data as well as removing practically all variability due to turbidity. This results in a focus of this analysis on dissolved substances. Deviations are then determined by comparing the values at each wavelength of the spectrum with the average value and the standard deviation in that value as acquired from the training dataset. If the sum of the deviations at any wavelengths is larger than the threshold value, this can be used to trigger an alarm. The algorithm will then indicate the size of the deviation and the region of the spectrum where the deviation occurred. Finally the results of all individual alarm algorithms, which separately operate (tools 1, 2 and 4) in individual parameters, are combined into a cumulative alarm. This cumulative alarm algorithm produces the final alarm value; the alarm functionality produces only one single alarm value. This value can be triggered by any alarm individually or by a combination of alarm values, each of which might not necessarily trigger an alarm on its own. The origins of the alarm, i.e. which parameters are responsible, can be established through status information that can be accessed via logbook functionalities. However, the single alarm value is the basis for any actions that might be triggered by an alarm status, e.g. switching of valves, sending of /sms messages...). The background information is available when requested but is not displayed so that it does not overload the operator with data. The weight of individual alarm parameters is established during the training of the system. The weights allocated are determined by the number of false alarms and the sensitivity of the parameters to changes in the matrix. The results of the default alarm settings for a parameter are multiplied by a factor, which corresponds to its weight in the composite alarm. The factor is optimised in such a way that the number of false alarms is minimised. An acceptable false alarm rate of 1 per defined time window, for example month, is defined and can be adjusted by the user. The weight follows from this optimisation. This ensures that the parameters that have the highest quality (measurement noise does not dilute the information) and with the highest significance (redundant parameters have a low weight) are used. This weight of the parameters can only be properly assigned when at least one test event is available for validation of the alarm settings. This can either be a real event that was captured by the monitoring system. Alternatively, an event can be artificially generated using mathematical tools. This allows the validation of TECHNEAU October 13, 2009

22 the training in case very stable water quality is monitored, and no true events are available for training. 3.4 Initial test results All the algorithms as well as the sensor and station management module were developed using real measurement data as well as user experience. The basic functionalities of the algorithms were evaluated using data previously obtained from various applications. The operation and integration of the three software modules is demonstrated here using a single example of water treatment plant. All data used in this example were collected using an s::can spectro::lyser submersible UV/Vis spectrometer probe. Evaluated were the raw spectral data as well as parameters calculated from this data, such as sum organics, dissolved organics, nitrate and turbidity. Assessment of the data as well as information from the plant operators indicated during the period evaluated here no events occurred that ought to trigger an alarm. In order to test the reaction of the event detection module to contaminants, it was decided to artificially add contaminants to the data. These was done by adding the spectrum of various concentrations of benzene (0 30 mg/l) to the spectral dataset and then recalculate the parameter results. The linearity between concentration and absorption (Lambert-Beer Law) allows this type of addition. Figure 10 summarises the results of this test. The starts of the artificial spikes are represented by the vertical dashed lines in the graph. In the middle field of the graph the measurement results of a single parameter are shown, and overlaid on this are the results of the data validation module. In the lower third of the figure, the response of the individual validation methods is presented, with dots indicating that the module has estimated that a measurement results is classified as outlier, noisy, discontinuous or subject to drift. Included as well is the evaluation of the sensor and maintenance module, which in this case assesses whether the instrument requires maintenance, in this case the maintenance requires being cleaning of the optical surfaces of the instrument. This conclusion is based upon the combined display of increased noise as well as drift, which is indicative of fouling of the instrument. It was later retraced to deactivation of the automatic sensor cleaning, which fits the observed symptoms. The upper part of the graph depicts the alarm value generated by the event detection module, with an alarm value above 1 representing an alarm state. The first four simulated contamination events are unambiguously recognised by the system, despite the fact that the fourth spiked events is superposed on a real event that had not been reported by the operators of the plant. This underlying event could be traced back to a short period where the instrument was being supplied with water of a different composition, possible coming from a different water source. As the system was not trained on this water type, it was classified as being abnormal. The fifth contamination event is no TECHNEAU October 13, 2009

23 longer detected, which can be related to the deterioration data quality which is visible in Figure 10 as increased noise on the signal and a visible upwards drift of the signal. This reduction in measurement quality is detected by the validation module, as illustrated by the increasing number of measurements being classified as noisy, outlier and / or drift affected. Figure 10: Example of Data validation and Event detection on data of less than perfect quality and with deteriorating sensor performance due to fouling of the sensor. TECHNEAU October 13, 2009

24 4 Future Work The tools described in this report are still in a prototype stage. At the time of writing a user software is being designed around this package, so that it will become available on the market as a user friendly instrument in the near future. Figure 11 shows the first impressions of how this data will be presented to the user. Most of the validation and alarm functionalities, however, will be embedded in the software at levels invisible to the normal user and is not presented to the user as in the figures above, because this type of information is unsuitable for interpretation by the average operator of online instruments. Experience has shown that the workforce in the field is mainly composed of relatively lowly educated personnel. Therefore, complicated features are reduced to minimum, and only the end result is presented in the form of cleaned data (output of the validation module) and alarm notifications. This is sufficient for proper maintenance and operation of instruments while still providing the advanced capabilities described above. In order to make this possible, the validation and alarm tools will come with default settings that allow use in typical applications and the software will include the intelligence to guide the user through simple optimisation procedures. In this way the user can work with a sensitive and reliable alarm system without needing to be an expert in mathematics and statistics. The system developed here will be further evaluated on real data as part of TECHNEAU Work package 3.6. In the deliverable that will be delivered after this evaluation, not only will further examples of results from the validation and event detection modules will be presented, but some focus will be placed on the actual benefits to the user of using a system such as the one presented here. TECHNEAU October 13, 2009

25 Figure 11: Upper picture 3D spectral data presented for quick evaluation of historic events. Lower picture An overview of alarm events detected by the system. Listed are date and time, alarm code, clear text description as well as a field where the user needs to acknowledge the alarm and report the actions taken. TECHNEAU October 13, 2009