EISCAT-3D WP8 Deliverable D8.2

Transcription

1 EISCAT-3D WP8 Deliverable D February 2007, v3.0, D.J.McKay, I.D. Finch, I.W.McCrea & 1. Introduction Report objectives Overview of the data and signal processing architecture Full-radar Receiver Beam Interferometric Data Intermediate archive content Final archive content Remote sites and data distribution architecture Data input and rates Computing hardware Data storage Data security Data Robustness Data Permission Enhanced-service archive Data formatting and association Network solutions Other Aspects not included at this stage User Interaction with the EISCAT-3D Data System Authentication and Data Security Data Visualisation Complementary Data Handling Post-processing software Dependences on Other Work Packages Summary and Conclusions References Appendix A: Data Rates Explained Summary Conventional ISR Mode Option Option Option Option Interferometry Appendix B: EISCAT3D Data Storage Specification Appendix B(1): Central Receiver Site - Permanent Archive (Single) Abstract Introduction Aim of the Information-Gathering Exercise Overall Requirements Phased Deployment Operating Modes Data Storage Specifications Networking Specifications

2 8. Software Specifications Operating System Total Cost of Ownership (TCO) Physical Size and Requirements Scalability and Migration Environmental Specifications Error Recovery Requirements Overall Downtime Documentation Training Warranty Appendix B(2): Remote Receiver Site - Remote Site Data Stores (x4) Abstract Introduction Aim of the Information-Gathering Exercise Overall Requirements Operating Modes Data Storage Specifications Networking Specifications Software Specifications Total Cost of Ownership (TCO) Physical Size and Requirements Environmental Specifications Error Recovery Requirements Documentation Training Warranty

3 1. Introduction This document forms the second deliverable from Work Package 8 of the EISCAT-3D Design Study, which relates to the design of the data archive and distribution system for the new radar. The current document summarises the progress made on this area of the Design Study up to January 2007, and sets out the current understanding of the dependences between this Work Package and the other packages of the study. In particular, we concentrate on the design of the data system topology and possible hardware solutions for implementing it. Our principal conclusion is that, in spite of the very high data volumes which will be delivered by the proposed EISCAT-3D radar system, a data archiving solution capable of preserving all of the scientifically important data products is achievable using existing technology. 1.1 Report objectives This report addresses the following description of work, as listed in Annex 1 of the design study contract [1] DoW-3 An initial low-level design relating to computing hardware, data storage, and network solutions for a secure archive will be undertaken, including evaluation of the relative merits of on-line and near on-line storage. The same document defines the objective of the current deliverable as follows: D8.2 Low-level design document: Networking and data storage requirements and favoured hardware solutions It should be noted that many of the issues relevant to WP8 depend upon the outputs of the signal processing work package (WP9), the control and monitoring work package (WP7), the network and time infrastructure work package (WP12) and the expectations of the end users (the EISCAT-3D scientific case). Hence at this stage of the study, there is still a degree of limitation imposed on WP8 because definitive outputs have not yet been produced from the other packages. That said, the following report attempts to summarise all the information currently available for this work package and explains the rationale and options for the currently envisaged implementation of the EISCAT-3D data archiving and distribution system. 2. Overview of the data and signal processing architecture Figure 1 shows the current conceptual design for the signal processing functions to be carried out at the central site of the EISCAT-3D system. The central site (probably to be located somewhere near Tromsø), will be much larger than the other receive-only sites, but the topology for all sites should be basically similar. Although it belongs within the remit of WP9, the signal processing functionality has profound implications for the design of the data archive and distribution system, so we review it briefly in this section. 3

4 Figure 1: Signal processing architecture [2], showing where the data archiving fits in with other components of the EISCAT-3D signal processing. The disk icons mark the areas which are addressed by this work package. 4

5 The central (transmitter) site comprises a large number of antenna elements, forming a phased array for both transmission and reception. Because the individual elements are likely to be crossed Yagi antennas, it will be possible to measure signals with two linear polarisations. (We shall discuss polarisation issues later.) The receiver array at the central (transmitter) site will be much larger than those at the receive-only sites. The elements of each array are divided into modules. These modules are sub-arrays and may therefore be used as interferometric elements. The output signals from the elements within each module are beam-formed to generate the actual receiver beams. Since multiple beams will be formed, multiple copies of all the input signals from the elements will be required. The module crossbar accomplishes this task. The output of the module crossbars are replicated sets of all the input signals from the elements. These will have already been amplified and digitised. These functions will probably be carried out close to the antennas themselves to prevent noise loss. The circuitry used as receiver protection (against the high power levels that are expected during the transmit phase of the radar duty cycle) is not relevant for this work package, since it does not modify the form of the signal, but merely time-slices it. However, it does not necessarily follow that the data rate is reduced, since it may be necessary to sample the transmitted waveform e.g. to obtain information on pulse shape for subsequent data analysis. The beam-formers take a number of inputs along with configuration information about the relative locations of the antennas that provide the input signal within the entire phased array not just the module. Thus each beam-former needs to be capable of full phase-correction and fractional sample addition, and must be capable of handling input signals from antennas over the full geometric extent of the array. This flexibility does not introduce additional complication, as it is simply a matter of scale. Furthermore, it means that any beam-former can be placed anywhere in the array; there will be no location-specific units, and this will ease manufacture, installation and maintenance. Combining modules for the final radar beam is then a matter of adding signals from the beam-formers, rather than having secondary beam-formers. It also means that in the interferometric case, the different interferometric elements have already had their delay/phase corrections introduced, and no further phase rotation or interferometric delay needs to be applied. A test array, consisting of a limited number of Yagi elements, is presently being constructed at Kiruna as part of WP9. It should be possible to build a fixed beam system for the test array very simply, thus creating a rapid prototyping environment for other parts of the system, including some aspects of WP8. It is anticipated that the outputs from the beam-formers will be handled in two different ways. The first is that the beam-formed input for each beam from each module will be added with the beam-formed input of the same beam from all other modules. The second is that each beam-formed input from each module will be input into the interferometric analysis system. In the following sections, we discuss the data handling for each of these two pathways separately, exploring the implications for the data archiving and distribution systems in each case. 3. Full-radar Receiver Beam The primary operating mode of the EISCAT-3D radar will be the formation of one or more full phased array beams, combining together the corresponding beams from all the modules of the array. This is simply a digital addition. Since the beam formers will be set up to accept inputs from the entire array, it is not necessary to apply additional corrections here. From the perspective of the data acquisition system, the output of the full-adders (one per beam) will be split. One part will be put into a cyclic buffer and then streamed to disk. These data will be analogous to the sample level data currently taken in some existing EISCAT experiments. The other branch of the beam-formed output will be put into a digital receiver and then auto-correlated and integrated. These data will be the equivalent of the autocorrelated data taken in most experiments on the current EISCAT system. 5

6 The decision on which of these two data types (i.e. beam-formed or autocorrelated data) is chosen for storage at any particular time should be made in the context of the science requirements for the experiment being undertaken. In the current EISCAT system, the archive consists of autocorrelated data, which have the advantage that they can be time-integrated. However, there are certain scientific advantages to storing the beam-formed samples so that they can subsequently be correlated by off-line software. This potentially allows a more flexible approach to auto-correlation and time integration, though at the price of requiring more powerful computing facilities at the central site. In Section 8, we show example calculations of data rate and archive size, based on long-term storage of beam-formed sample streams and autocorrelated data. Both scenarios are challenging: beam-formed data rates are of order GB/s, which means that the consequent data volumes are unacceptably large for permanent storage. This would not preclude the short-term storage of such data in cyclic buffers prior to subsequent re-processing, but even these would need to be relatively large, unless the sample streams were heavily decimated. Autocorrelated data are implicitly larger in size than sample level data, because of the need to store the full correlation lag matrix. However, this increase can be mitigated by integrating the data temporally and spatially, which is not possible for the sample level data stream. Our calculations in Section 8 show the effect of one such scheme, two more are detailed in Appendix A. This produces manageable data rates of as little as 2 MB/s, resulting in storage lifetimes on the order of 15 years for a Petabyte store. However, delivering autocorrelated data in this form will place stringent requirements on the signal processing work package (WP9) e.g. for decimation and correlation. It should also be remembered that storing autocorrelated data from both plasma lines as well as the ion line would increase the data rates and reduce the archive lifetimes by around a factor of 3 compared to the figures shown in Table 8.2. Because they can be integrated over range and time, autocorrelation functions probably offer the best solution for permanent storage. It is, however, also desirable to store data temporarily from at least some experiments as beam-formed, sample streams. Although of greater volume because they cannot be time integrated, such streams allow users to exploit the maximum flexibility in auto correlation and range gating. A scientific justification would be required to determine the autocorrelation and time integration strategies and to set the latency time of the cyclic buffer so that its duration can be varied according to the scientific importance of the data being taken. A strategy which combined short-term storage of sample streams with long-term storage of final autocorrelation functions would thus allow users to ensure that the contents of the permanent data archive represented the optimal measurement of the ionosphere which could be made from any given experiment. Regardless of whether the final autocorrelated data are being calculated at the time of the experiment, some form of real-time auto-correlation is required, in order to generate visualisation data, which will allow monitoring of the radar performance and first-look views of the data, using a visualisation system similar to that implemented in the existing EISCAT system. Aside from the issue of data volume, there are some other, less serious, problems associated with the storage of integrated data. For example, such data are often contaminated with cluttering signals from satellites and space debris. Such contamination is currently removed at the time of data analysis, but this is an inefficient method of clutter removal, since it generally takes place after the data have been pre-integrated. It should be possible to devise some intelligent software to eliminate such clutter in realtime, prior to data integration, but care would be needed to distinguish between genuinely unwanted clutter and unusual radar echoes which might have geophysical significance. Note that in Figure 1, the disks which store the beam-formed samples and the auto-correlated outputs of the normal radar signal path from each radar beam are shown as being separate. This is because the data streams are actually independent, and it may not be necessary to store all of the potential data products in order to achieve the required science objective. The disk symbols indicate those points in the EISCAT 3D system where WP8 systems would need to take over from the signal processing systems of WP9 with regard to data handling. 6

7 4. Interferometric Data In addition to the normal beam-formed data generated by the phased array, it is also intended that the radar should have a second type of data product, namely a set of interferometric data with subbeamwidth resolution. The availability of such data opens up an important area of new physics, not routinely accessible to the present EISCAT system. Under normal conditions, the ionosphere can be considered as a beam-filling target which, within the size of the EISCAT beam (a circle of radius a few kilometres at F-region heights) is effectively homogeneous. This means that, within this volume, no strong structure exists within the ionosphere, so that the whole area of plasma illuminated by the radar has the same basic properties (density, temperature, composition, velocity etc). However, it is known from existing EISCAT studies that under some conditions these assumptions can break down. For example, during certain types of aurora, the EISCAT volume can be dominated by sub-beamwidth scatterers known as NEIALs (Naturally Enhanced Ion Acoustic Lines), whose size is assumed to be of order tens to hundreds of metres, and whose backscatter cross-sections can be two orders of magnitude above that of the surrounding ionosphere. There is an ongoing debate about what these phenomena represent, and the nature of the processes which lead to their formation. Hence it is important to study them further, and this cannot be done with conventional radars. However, NEIALs occur very infrequently (much less than 1% of observations display this behaviour) and unpredictably, making them very difficult to study by specialised experiments. A better solution would be to ensure that every observation undertaken by the EISCAT-3D radar, at least from the central site, was suitable for interferometric analysis, so that when a suitable event is detected, the interferometric content of the returned signal was analysed and the size and number of sub-beamwidth scale scatterers determined. The methodology for making interferometric measurements on the EISCAT-3D system has already been outlined in the two reports produced by Work Package 5 [3], [4]. The reconstructed image is obtained by measuring the cross-phase and coherence of the signal measured by multiple pairs of subarrays within the overall antenna. For this reason, the central antenna array will probably not be in a neat geometrical form such as a square, rather the spatial distribution of the constituent modules will be chosen in a way which maximises the radar s interferometric performance. Because of the infrequency of sub-beamwidth scatterers and the high volume of interferometric data, it is not planned to store such data continuously. It should be obvious that interferometry data are only of interest when there is structure which needs to be resolved. At such times, the coherency of the signal between module pairs will rise to some finite value, while at other times it will be close to zero. Because of this, it should be sufficient to measure continuously the coherency between a few module pairs, with cross-phase and coherency data from all module pairs only being retained when the tested coherencies rose above a given threshold. However, in order to catch the onset of sub-beamwidth scatterers, there is a requirement to record the full interferometry data in a short-term cyclic buffer so that, once the coherence threshold is exceeded, the interferometric data from the period immediately preceding the start of the event can be reconstructed. Once the coherence threshold is exceeded, the full interferometric data will flow to disk until the coherence drops below the threshold again. The durations of such events seldom appear to be more than a few minutes. In order to catch the final disappearance of the scatterers, it is planned that data should continue to be recorded to disk for a fixed time after the coherences from the tested module pairs fall below the threshold, unless the threshold is once again exceeded during this period. This means that, as for the beam-formed data, it is proposed to use a buffered recording system for the interferometry data. The buffer is used for continuous recording in order to ensure data preservation while the coherence thresholds are tested. Normally, this buffer would be overwritten every few minutes, except when the coherences rise above the threshold value, at which point the buffered data will streamed to permanent storage, since they are assumed to correspond to the start of an interval of sub-beamwidth scattering. 7

8 5. Intermediate archive content As discussed above, there are two types of ring buffer in the EISCAT-3D data storage system. These will have limited-duration storage, before the data are discarded. Within that time period, the scientist or operator must determine whether or not the data are worth keeping or processing into some other form, though most aspects of the decision-making process can probably be automated. The first of these buffers is for standard radar use (marked RDCB in Figure 1), and will store beam-formed data in the form of sample streams for a short period of time. Longer-term storage of such data might not be a given, but could be decided in real-time by automatic detection/analysis software. Similar to the interferometric coherence threshold detector (see above), such software would constantly monitor the incoming data and, using the delay provided by the ring buffer, pre-emptively intercept and store data deemed to be of scientific interest. If not, the data will be overwritten after a while, as the buffer cycles. It is anticipated that any automated algorithms which decide whether data are to be stored, and for how long, will need to be refined at the early operational stage of the project, as experience is gained in their use. By putting a connection point to the ring buffer in place during the initial deployment, later additions of such automatic detection and storage software can be added. In the interim, the connection would serve as a pick-off point, should particular experiments require the storage of data in an alternate form. However, such storage would initially need to be provided by the end-user, and the EISCAT-3D system would only provide the connection interface. The interferometry component also has a cyclic buffer which, as described above, is used to buffer data which might be deemed useful for storing. The process of determining if the coherence threshold has been exceeded requires time for the correlation to be made, transforms to be performed, and the data integrated to a level at which signal can be detected for a decision to be made. Additionally, if the threshold limit is set high (to avoid spurious recordings of noise), then back-tracking the data somewhat is necessary to catch the rising edge of the event, which may be of scientific interest. This results in a cyclic buffer that needs to store data in the order of tens of seconds. Unlike the RDCB, it doesn t make sense to talk about picking off data from this buffer, since the final interferometric storage would be designed to do this job. Data storage in the RDCB will probably be in a low-level format (raw bit streams). If retrieved directly for experimental purposes, these data would need to be post-processed by the recovery system to convert them to a more usable format, and transferred to conventional media. If passed on directly to the next stage of the system (e.g. for autocorrelation), they would remain in the raw format, thus making them appear more or less as a transparent component of the system. In addition to their potential uses as storage and raw-data pick-off points, the ring buffers would also allow the system to cope with anomalies in the regularity of the data flow. By the time the data reach this point, they will be on conventional network systems, and delays can be expected owing to packet collision and the quantised nature of the packet transfer protocol. As a result, the ring buffers will serve to regulate the flow of information, and would allow a catch-up grace period for data which are temporarily disrupted. Such disruptions, in addition to the standard operation disruptions listed above, would also include more major incidents, such as rebooting a down-stream computer. The RCDB would provide sufficient latency to recover, with time, the computer outage, as it would be possible to process the backlog of data faster than the incoming data rate. This would not apply in the case of the interferometric system. 6. Final archive content As noted in Section 3, the contents of the permanent data archive, at least for the central site, will be data of two kinds, namely conventional ionospheric radar data and interferometric data. While storage of uncorrelated beam-formed sample streams is possible, an aggressive decimation strategy is needed to reduce the data input rates to values compatible with permanent storage. This will be achieved by storing time-integrated, atuocorrelated data as in the present EISCAT system, though this has the weakness that the integration and correlation strategies need to be preset. A short-term ring-buffer of 8

9 sample stream data however would allow an operator to vary the integration and correlation strategies post hoc while the data remained in the buffer. It should also be noted that, because of the data rates involved, it is necessary to provide each beamformer with a direct connection to the archive, rather than attempting to multiplex several inputs into the same connection. This is why Figure 1 shows both correlated and uncorrelated data stores separately for each beam. For the interferometry data, it only makes sense to store all interferometric channels in order to reconstruct the image-plane from all of the available visibility data. Furthermore, owing to the sizes of the intermediate interferometry products (such as cross-correlations), it does not make sense to store partially processed data. The storage problem for interferometry is thus equivalent to that for multiple streams of beam-formed data, in that it imposes a requirement to handle very high input data rates, but in this case only for relatively short time intervals. The full visibility data would be stored in the archive until they had been completely post-processed, at which point they could be deleted or moved elsewhere, and only the highest-level interferometry products would become part of the permanent archive (see Section 8). 7. Remote sites and data distribution architecture Since the EISCAT-3D system is planned to be multi-static, with up to four remote sites, a key issue for WP8 is the question of how the data are moved from the various sites back into the central archive and from there on to the users. This is an area that WP8 has explored, within the limits imposed by other work packages. The limitations are those of WP7 (Control and Monitoring), WP9 (Signal Processing), WP5 (Interferometry) and the scientific case. Without the scientific priorities and the proposed data rates from the input work packages, it is not possible to finalise the intra-site data rates, and thus to fully define the limitations of the data distribution architecture. That said, the following is brief discussion of some of the options that are currently available. It is assumed that the EISCAT-3D system will have some kind of master archive, ideally located at the central site, which will hold a copy of all the data which it was practicable to store centrally. The constraints on the size of the archive will include the large sizes of some of the data sets, especially for some of the interferometric data products, and the capacity of the network connections from the central archive to the remote sites of the EISCAT-3D system, some of which may be in remote locations. It is assumed that remote site data will be brought back to the central archive in the same form as the permanently stored data from the central site (see Section 3). In principle, the beam-formed remote site data are the same size as those from the central site. However, in practice it is intended that the radar will be run with an approximately 25% duty-cycle (i.e. the radar will only transmit for 25% of real-time with the central site receiving for the remaining 75% of the time). This means that the volume being observed by the remote sites is only illuminated for 25% of the time and data need only be recorded at those times. In addition, if transmitter sampling is not conducted at the transmitter site, then it will only produce 75% of the data compared with a site which is receiving for 100% of the time. For the remote sites, it would be impossible for the full beam-formed data at 30 MHz bandwidth to be moved to the central archive by any standard public link. Transfer of such data could only be undertaken for short periods with data being accumulated at the remote sites and transferred to the central archive during later periods of operation with lower data rates. Longer term operations requiring full-bandwidth data could only be handled using either proprietary networks or physical transport of media. It is anticipated that bandwidth will always be more than sufficient to transfer any auto-correlated and integrated data products in real-time. It is clear that some on-site storage will be needed at the remote sites, both because there needs to be some safeguard against data being lost in the event of a network outage and because data may need to be buffered on-site during times of high data rate operation (e.g. experiments with large numbers of 9

10 beams). There might be a requirement for such a buffer to have a latency time of a few weeks, given the possible logistical difficulties of repairing internet links in Arctic Scandinavia. Even in this case, the remote site archives would not require the Petabyte storage capabilities needed for a long-term archive at Tromsø, and a few tens of Terabytes would probably be sufficient, though the capacities for I/O of data to and from the remote archive should be the same as those for the central site. Analysis capabilities would obviously need to be provided at the central site, but since this also acts as the central archive, there is not envisaged to be a requirement to move large amounts of raw data off the central site. A large computing facility for data analysis needs to be located at that point. Provision of this computing system is not covered by the existing design study. A similar situation exists for interferometry, however this is more limited in scope, as all the interferometry data will be acquired from the central site (assuming no interferometry is done at the remotes). Data processing must be done on-site to keep the export data rates low, and experimenters who wish to take intermediate data away with them will probably have to do so using their own media. All other interferometry calculations will be done using local processing, provided as part of WP5, and only final results (figures, plots and animations) will be exported via conventional Internet links. 8. Data input and rates Some indicative data rates for the output of the EISCAT-3D radar are shown below. These have been derived from more detailed spreadsheets which we have constructed to calculate the data volumes which the archive system should handle [6]. They should not be taken as a firm indication of what is planned, merely as a demonstration of the difficulties inherent in the storage of the various kinds of raw radar data. Table 8.1 shows the total data rate for the whole central site, calculated by multiplying the sampling rate on each antenna element by the number of elements and the number of bits per sample. This represents the volume of data in the whole central site system, prior to beam forming. The table also shows the volume of sample-level data obtained after beam-forming, assuming that the data rate for each final beam is equivalent to the original data rate for each constituent antenna element. Receiver bandwidth 30 MHz Polarisation channels 2 Bits per sample 16 b Sampling rate 80 MHz Data rate per element 1.28 Gb/s polarisation Data rate per element 2.56 Gb/s Data rate from all elements Tb/s = 5.90 TB/s Storage at site TB Storage duration for all prebeam s formed data Assumed number of beams 1 Data rate per beam 2.56 Gb/s Beam-formed data rate 2.56 Gb/s = 0.32 GB/s Storage duration for all s beam-formed data = 36.2 days Table 8.1: Data rates and storage durations for pre-beam-formed and beam-formed data at the EISCAT-3D central site. 10

11 It is clear that the permanent storage of data prior to beam-forming is not practical because of their huge volume. Even after beam-forming, however, the volume of sample-level data is too large for long-term storage unless the beam-formed data can be decimated. This will be done by autocorrelating and time integrating the autocorrelation functions, as is done in the present EISCAT radars. Table 8.2 shows the anticipated data volumes which would occur assuming the storage of autocorrelation functions with a pre-integration time of 2 seconds. : Required minimum range 50 km Required maximum range 1500 km Range increments 500 Number of lags per range 200 lags No. of data per matrix 1.00E+05 data profile Data storage per datum 8 bytes Data per correlated matrix 0.80 MB Post-correlation integration 2 s Total data rate per beam 0.4 MB/s = 1.4 GB/hr Table 8.2: Data rates for auto-correlated data at the EISCAT-3D central site. The assumed number of range gates (500) arises from a rough calculation assuming range increments as follows: km: 100m, km: 500m, km: 1 km, km: 3 km, km: 5 km, km: 10 km, km: 50 km. Each of these range increments is about 25% of the characteristic scale length of the ionosphere at the relevant altitude, as calculated from either the plasma scale height or the wavelength of characteristic wave and tidal oscillations. Autocorrelation functions are assumed to be complex with 100 lags in each of the real and imaginary parts. Table 8.3 indicates the implications for the data archive of storing these auto-correlations, assuming decimated correlator output. It shows the expected lifetime of a buffer of 100 TB at the central site and for a permanent archive of 1000 TB. Number of beams 5 beams Total data rate per second 1.00 MB/s = 3.6 GB/hr = 30.8 TB/year Central buffer duration 3.24 years Archive duration years Table 8.3: Archive durations for auto-correlated data at the EISCAT-3D central site. The foregoing tables obviously only consider the data rate for the central site, and the overall data throughput will increase when remote sites are included. However, for these sites it should be possible to greatly reduce the volume of data because, for a pulsed radar, the range over which incoherent scatter return is received is limited to the subset of range gates in which the remote site beams intersect with those from the transmitter site. Although the precise ratio will depend upon the experiment geometry and the transmitter duty cycle, one might conservatively expect that the total amount of incoherent scatter data coming from all remote sites will be equivalent to, or less than, the volume of data coming from the central site. Hence it appears to be tractable to store the entire auto-correlated incoherent scatter data expected from the EISCAT-3D system in an archive of a size which is likely to be affordable at the time of project deployment, while still assuming 24 hour continuous operation. However, production of the autocorrelated data in this form may have significant implications for activities in the signal processing Work Package, WP9. 11

12 In the above tables, no consideration has been given to the interferometric data rate. We do not propose to make a separate provision for interferometric data storage, since this is assumed to use the same archiving resources as indicated for the standard incoherent scatter mode. We assume that users requiring the high volume intermediate products of the interferometry data, such as cross-correlation functions, will provide their own storage or that operational procedures will be altered in the allocation of storage resources if such products need to be stored. The most practical approach would be to limit the provision of permanent EISCAT archive storage for interferometry data to the highest-level derived products, such as images and movies, which are of relatively small volume, as discussed in Section 6. This means that the main WP8 requirement for interferometry is to allow temporary storage of the high bandwidth data from each module pair for subsequent post-processing, a problem equivalent to the short-term storage of decimated sample level data. 9. Computing hardware The computing hardware needed for the specification of WP8 can be divided into four main areas: Network infrastructure Data storage Data processing, and Visualisation facilities These hardware elements are discussed in subsequent sections. Although the network infrastructure is the first aspect that the WP8 components present to external interfaces, the specification of the network infrastructure is actually defined by the need to transport data for storage, so we shall commence our review with the data storage component. 9.1 Data storage In the context of this discussion, data storage refers to the long-term archiving of data from the EISCAT-3D radar system. The data collected will represent a major financial investment, so their storage should be secure (discussed below) in terms of access, reliability and longevity. Large data stores of this nature are commonplace in the commercial world, although the EISCAT-3D project will be pushing the boundaries of size to a certain degree. However, the most important point is that the provision of such an archive is already possible with commercial data storage products. Three potential candidate archive systems have been identified so far. One of these is the PetaBox system [7], supplied by Capricorn Technologies. PetaBox claims to offer a high density, highly reliable, highly scalable, and highly available data storage. The company claims to deliver the lowest cost per terabyte of any storage platform for that capacity range. It also claims that the system is completely scalable, from individual terabyte nodes to petabyte clusters. A single 19-inch rack can support up to 120 Tbyte of raw disk space. Best quoted energy consumption is 27 watts per terabyte. This is an existing system, which can be purchased off-the-shelf. The other two candidate storage systems are future projects. One of this is project BlackBox [8], by Sun Microsystems. This system is a data store, housed in a standard shipping container. The advantage is that the entire unit can be moved, and that it is possible to simply add additional units. Furthermore, the environmental condition requirements will not be as rigorous as for a standard system. (The storage could, for example, be located in a warehouse, rather than air-conditioned computer building.) The other system which we have explored is TotalStorage SAN File System [9] by IBM. This is a storage system designed to work over a distributed, heterogeneous environment. It is supposedly scalable, using Storage Area Network (SAN) protocols, and allows data to be distributed, by only maintaining metadata on the central server, with clients accessing the raw data directly from the different distribution sub-servers. The total storage requirement could therefore be built from a series of 12

13 smaller storage sub-systems. The TotalStorage SAN File System is planned to be used as part of CERN s next major data storage upgrade. As can be seen from Section 8, the assumed data rates for the EISCAT-3D system place strict requirements on the I/O performance of the selected storage hardware, especially if sample-level data are being streamed directly into the data store. Using currently available technology, the manufacturer s specifications discuss input data rates limited to around 4Gb/s. The data rate is therefore likely to be a critical factor in determining the way that the archive hardware is structured, imposing a requirement to configure the storage into a number of discrete units dictated by the maximum bandwidth available to each. For a multi-beam radar, it would be logical to imagine a structure based on the number of beamformers, so that the archive system for a radar capable of 8 beams might be split into 16 discrete units, one for each beam polarisation. While input data rates of Gb/s would not be required at all times, since it would be impossible to store a long-term data set taken at this rate, it is assumed that there will be some occasions where storage of data taken at such rates will be required, even if only for short intervals intervals when the coherence threshold for interferometry is exceeded may be an example of this. Likewise, it is assumed that data access from the store will potentially require Gb/s rates, e.g. in order to supply users post-processing the products discussed above, or for the mirroring needed to ensure data robustness (see Section 9.2). Because of this, we have framed a specification test which has been sent to potential data store suppliers, asking them to verify that their data storage systems are capable of handling two simultaneous Gb/s input streams at the same time as two simultaneous Gb/s output streams. We have also suggested that we should participate in such a test to verify that the required specification is being achieved. Documents setting out the required specifications for the central archive and the remote site data stores are included as Appendix 1. When discussing specific hardware solutions, it is necessary to inject a word of caution. The technology of data storage systems is a fast-changing field, with continuing rapid improvements in storage capacity and I/O speed. It is therefore almost certain that the technology in use today will have been succeeded by more capable systems by the time that the EISCAT-3D system begins to be deployed. The discussion of specific systems above is not so much aimed at identifying the actual system to be used in the final radar, rather to establish whether the required performance is feasible with current devices and to give some indicative costs for the data storage hardware, since it is clear that this will be an expensive component of the EISCAT-3D system. 9.2 Data security There are two aspects to data security: Ensuring that data are not lost which will be referred to as Data robustness Ensuring that data may only be accessed by those permitted to access them referred to here as Data permission 1. Data Robustness There are two methods to ensure that data is not lost: replication and back-up. Replication is more convenient, but requires more maintenance. There is also the issue of concurrency that is, keeping the copies synchronised. Machines that are physically close have the advantage of faster, more reliable interconnectivity, but the risk is that both might be damaged by the same catastrophe. Remote mirroring, on the other hand, may be constrained by interconnectivity limitations the size of the data pipe between the two locations. However, modern transfer software (for example bbftp [10], GridFTP, etc.) is able to increase performance with multiple-connections [11][12]. This may also allow us to overcome problems such as the finite transfer limits for individual connections which might exist between the master archive and the chosen mirror sites. This also raises the question of the file system 13

14 structure and file-size quantisation (the current EISCAT system groups data into directories each of which contains one hour of data). Another problem with mirroring is how to ensure that the corruption of one mirror does not propagate to the others, which will require a degree of intelligent software. Given the large size and high data input rate of the data set being considered here, the optimum solution is probably for the mirror to be housed in a physically separated building on the same site as the master archive, so that the two can be connected by proprietary network links. The mirror archive does not necessarily need to employ the same hardware as the master. Although using common hardware and software would make maintenance easier, employing different solutions for the two systems might make the overall archive more resilient to faults. Since the mirror archive is less likely to be accessed for data output it could also include a large component of off line or near-on-line storage in which much of the data were held on storage media such as robotically accessed tape cartridges, rather than on spinning disks. The ability to transfer data to such media could also facilitate the transport of further data copies to off-site archives, which it might be impractical to update using public networks. 2. Data Permission Controlling access to the archive is something that will require consideration, but this issue is not being addressed by the current phase of the design study. However, it is appropriate to mention it at this stage, so as to ensure it is kept in mind for future discussion. For example, we will need to consider whether the EISCAT-3D data architecture should implement a full authentication/accreditation system, or instead rely on users observing the so-called rules of the road. 9.3 Enhanced-service archive In addition to a simple archive, which accepts incoming data, stores it, and serves it to users on request, there is also the possibility of configuring the archive system to provide additional services. For example, such services might include preliminary data processing, thereby accomplishing the following objectives: Flexible auto-correlation of decimated beam-formed sample streams. Initial data analysis (which would normally have to be done anyway). As with auto-correlation, this processing would be performed on the server at the central site, which would probably have more CPU capacity than any individual user machine. Reducing the data volume, thus limiting the impact on the network, since users would then transport the analysed data to their own machines, rather than moving the whole raw data set and analysing it locally. Keeping the correlation and data analysis programs in a central, well managed facility, rather than having multiple copies of (potentially) outdated reduction software scattered throughout the user community. Initial data processing would also include functions such as integration and satellite screening. In this scenario, the archive would have three modes of operation: accepting/storing raw data, serving raw data, or processing the raw data and serving the results. Developments in technology (network bandwidths or CPU/software) between now and the time of system deployment may tip the balance one way or the other in favour of serving raw or partially-processed data, but at this stage it would be prudent to plan for all three forms of data serving. 9.4 Data formatting and association In addition to the actual storage of the data, there is a degree of processing which needs to be performed. This processing is at a lower level than that required for functions such as data analysis or satellite-screening (see above). It is primarily, but not solely, the computing power required to perform two closely-related tasks: 14

15 Data formatting Data association The data formatting is the conversion of the data from raw bit-streams to a format suitable for storage on long-term media and subsequent retrieval. The data association task is that of combining the stream data coming from the primary radar output with the auxiliary data coming from other areas of the system, in particular the data produced by the Control and Monitoring Work Package (WP7), which will be required by the data reduction software. 9.5 Network solutions The network requirements of the overall project are generally being addressed under WP12. However, the data rates required by WP8 will be an important input to that work package. As indicated in Section 8, these data rates are extraordinarily high, and this will require a high degree of parallelisation for the various data streams. This is certainly possible, as the individual data streams are independent and may be channelled separately. For this, conventional fibre channel communications may be used. Initial input from WP12 suggests that preferred solutions may involve network protocols such as FPDP-II (Front-Panel-Data-Port) or Infiniband. 10. Other Aspects not included at this stage For the current deliverable, (D8.2 in the project plan [1]) we have concentrated on specifying the topology of the EISCAT-3D data system, explaining the form that we expect the various data products to take and how these could be accommodated. We have also set out some options for hardware solutions which will make these requirements achievable. In addition to the various aspects of Work Package 8 which have been discussed above, there are a number of important issues which have not been considered in this report. This is either because the project plan calls for these topics to be addressed at a later stage of the design study, because the issues concerned are dependent on the resolution of questions relating to other work packages, or because the topics are not within the scope of the design study. Although we cannot discuss these issues in detail in the present report, it is worthwhile to list them here, and to indicate how they fit into the timeline of future WP8 activities User Interaction with the EISCAT-3D Data System In addition to specifying the EISCAT-3D data system at the hardware level, it is also necessary to consider the types of interaction which the user might wish to have with the data system. In the original project plan [1] we envisaged a set of access layer software, which would provide the interactive interface between the user and the data holdings. Although the actual development of such software is not within the scope of the design study, the specification of the functionality required by the user, and suggestion of how this might be achieved in software is an important function of the design study. Some of these issues have already been alluded to in Section 9.3 above (the Enhanced Service Archive ) and a more detailed outline of the required functionality will be the subject of the next deliverable (D8.3) due in July Authentication and Data Security This issue has been briefly discussed in Section 9.2 above, under the heading of Data Security. EISCAT data are, in principle, proprietary in the sense that Special Programme data are reserved for the proposing experimental team for the first year after the data are taken, normal Common Programme data are reserved for scientists in the EISCAT member countries, while only data from World Day Common Programme operations have unrestricted availability outside the member countries of the EISCAT association. Currently, access to EISCAT data is not handled by an authentication system, but relies on users respecting the rules of the road. However, a number of comparable projects have adopted more formal authentication facilities, based (for example) on the exchange of digital security 15

16 certificates. This issue could be considered as an aspect of user interaction, and will be discussed further over the coming months, with outlines of a possible system again being included in D8.3 (July 2007) Data Visualisation Visualisation of the data (both system data and ionospheric measurements) being produced by the radar is a key aspect of user interaction, since the ability of a scientist or operator to monitor the ionospheric conditions or system performance will be key to controlling radar operations and influencing the choice of experimental modes. This part of Work Package 8 will assess the optimum methods for visualising the EISCAT data at various levels of processing, from sample level to derived data products. This will build on the heritage of EISCAT s existing visualisation utilities, as well as facilitating the adequate representation of the types of user interaction defined by D8.3 (see above). Documents outlining the design of the visualisation systems for raw and analysed data will be produced as deliverables of WP8 (D8.4 and D8.5) due in February Complementary Data Handling The handling of complementary data is an important issue for the EISCAT-3D project, and can be split into three distinct categories. The first is metadata, which should be understood as data which describe the data themselves. This includes the ability of the data to be self-describing, such that software reading the data can use relatively generic interfaces, such as standard APIs, exploiting the fact that the data file contains descriptors specifying the size and type of the data which it contains. This is a well-known issue within the scientific community and a number of different approaches (e.g. CDF, HDF, XML) exist to address this problem. The second issue concerns the type, volume and format of the data which describe the state of the radar system. The specification of such data largely falls within the remit of the Control and Monitoring Work Package (WP7), but this is a very significant issue for Work Package 8, because such system data need to be stored in a similar way to the ionospheric data, and need to be formally associated with them, because aspects of system configuration such as transmitter power, number of beams, beam pointing direction and signal processing status will be required by any subsequent data processing algorithms. The third issue regarding complementary data concerns inter-operability with other instruments, either those specifically taking data in conjunction with the EISCAT-3D radar, or those operating independently in a wider geophysical context. A number of geophysically important parameters, e.g. Joule heating rates and particle energy maps, can only be produced by combining EISCAT data with data from other instruments and would add significant value to the scientific data from the EISCAT-3D radar. The data formats and archive systems selected for such instruments therefore need to be considered with this inter-operability in mind, and consideration of all these issues is scheduled for the final phase of the design study, leading up to the last deliverable, D8.6, due in August Post-processing software It is worthwhile to note in this section that the design of post-processing software (i.e. analysis programs equivalent to the present GUISDAP program currently used at EISCAT) it is not a specific part of the design study. However, there is a clear link between WP8 and such software. The external interface, and anticipated user-computing power and capacity will place limitations on what we can reasonably expect to export from the archive and these will be important factors in the eventual design of the data analysis software. 16

17 11. Dependences on Other Work Packages As noted in several places above, the progress of WP8 depends upon the outputs of several other work packages of the design study. In this section, we have collected together these dependences to make clear exactly what inputs are anticipated from the various other work packages of the study. Work Package 1 (Management of the design study) provides the overall organisational framework within which the project is carried out, consisting of financial and technical oversight, distribution of funds, and operation of the project web site. Work Package 2 (Evaluation of design performance goals) has already delivered a baseline specification document describing some of the performance goals to be achieved [13]. Work Package 3 (Evaluation of options for the active element) concerns the transmitter hardware for the EISCAT-3D radar, and has no obvious implications for WP8. Work Package 4 (Phased array receivers) concerns the evaluation of hardware options for the receiver side of the EISCAT-3D system, including the design of the arrays for the central and remote sites. Although it has no direct interaction with WP8, the details of the array design may have implications for the way in which the system will work, such as the number of possible simultaneous beams and the number of sub-arrays to be used in interferometry, which affect the size of the various data products. However, we would expect these implications to manifest themselves to WP8 through our interfaces with WP9 (signal processing) and WP5 (interferometry). Work Package 5 (Interferometry) has close links to WP8, since it will define the number and size of any interferometry data products and the lengths of time for which these need to be stored. These two work packages have already been interacting very closely, and the design of the data system for interferometry, shown in Figure 1 and discussed in Section 4, reflects detailed discussions between the teams involved. Work Package 6 (Active Element) is concerned with the detailed design of the transmitter side of the EISCAT-3D system, building on the choices made in WP3. It has no direct interfaces to WP8. Work Package 7 (Distributed Control and Monitoring) is responsible for developing the control and monitoring system for the EISCAT-3D radar, mainly involving the identification of suitable software systems used elsewhere in the scientific community. We would expect this work package to have an important interface to WP8, and some effect on the data rate, since it will be responsible for establishing the amount and type of system status data to be collated. These data should then be archived by WP8 and the system status data need to be associated with the corresponding ionospheric data, since both will be required by subsequent data reduction software. The first requirement from WP7 is a list of the system status data to be provided and an indication of their size, format and frequency of provision. Work Package 9 (Signal Processing) is probably the most important work package for WP8, since it will provide the specification of the standard ionospheric data products, such as sample series and/or auto-correlation matrices, together with important scaling factors such as the number of possible beams. We anticipate a close interaction between WP8 and WP9 over the coming months, as we refine the nature and volume of the products which the EISCAT-3D signal processors will deliver to the storage systems of WP8. Work Package 10 (New Techniques) deals with the investigation of novel uses for incoherent scatter radar data. This package has no immediate interaction with WP8, though might possibly have some implications for the data rates if it resulted in radically new experimental concepts. Work Package 11 (Implementation Blueprint) is the package under which the final detailed implementation plans will be prepared, using the output from other Work Packages. Although the bulk 17

18 of the effort is concentrated at the end of the project, some earlier work will be needed to develop and iterate the overall system design. This is likely to have implications for WP8, particularly in clarifying its interfaces to other Work Packages. Work Package 12 (Networking and reference time and frequency) deals with issues of data transport within the EISCAT-3D system, at the level of network hardware and protocols, as well as the distribution of time and frequency standards. This clearly has implications for WP8, as it will determine the format and protocols used for streaming data to the archive systems. 12. Summary and Conclusions This report has comprised deliverable D8.2 of the EISCAT-3D design study. In this document, we have summarised the status of WP8 at this stage of its development and identified a number of potential hardware solutions for the primary cost component the data storage facility. At this stage, we have just begun to approach commercial suppliers with an indication of our performance requirements (see documents in Appendix 1), and it is still premature to identify a specific hardware solution. Rather, our aim has been to outline the relevant considerations for the hardware level of the EISCAT-3D data system, and to demonstrate that, even for the large data volumes under consideration, an archive solution is feasible using currently available technology. References [1] Annex-1, FP Infrastructures-4 [2] DJM, Interferometry and IS Signal Processing, v1.1, [3] Fundamentals of radar interferometry I: One baseline Stage 2 report, April [4] D5.1, EISCAT_3D Radar Imaging Arrays Configurations Report, submitted September 27, s_report.pdf [5] Lind et al., Advanced techniques for incoherent scatter radar, Session G3, URSI GA, Maastricht, 2002, [6] DJM, Data rates analysis, 30-Aug-2006, [7] [8] [9] [10] BBFTP Homepage, [11] DJM, RAL/ESC Network Performance Report #1, 31-Aug-2006, [12] DJM, RAL/DL Network Performance Report #1, 10-Nov-2005, [13] D2.1, EISCAT_3D radar performance specification document, version

19 Appendix A: Data Rates Explained Summary All scenarios in this summary assume that transmitter sampling is conducted on a single transmitter beam from the central site which is illuminating the sky for 25% of real time, and that each of the four remote sites is sampling this transmitter beam with 10 receiver beams. Raw data no bandpass filtering (see Option 1) 1 Central Beam + 10 beams per remote site 3.52 GB/s 12.7 TB/hour 304 TB/day 8.5 PB/month 111 PB/year Auto-correlated data (see Options 2-4) High-resolution (see Option 2) Medium-resolution (see Option 3) Low-Resolution (see Option 4) MB/s MB/s 2.09 MB/s GB/hour GB/hour 7.52 GB/hour 3.67 TB/day 1.84 TB/day 0.18 TB/day TB/month TB/month 5.06 TB/month 1340 TB/year TB/year TB/year High resolution data option has a temporal resolution of 1s; medium- and low-resolution data options have a temporal resolution of 2s. High and medium resolution data options have a range resolution of 300m; low-resolution data option has a range resolution of 3km. Interferometry (average rate): Central Site Only 0.64 MB/s 2.30 GB/hour 55.3 GB/day 1.55 TB/month 20.2 TB/year 19

20 Conventional ISR Mode 3. Option 1 This is essentially the option of storing the results coming from the beamformers without any filtering. This is our worst case scenario since it results in the largest possible data rate, excluding the possibility of recording data from the individual antenna elements. The digital data rate from an individual antenna element is: Receiver bandwidth 30 MHz Polarisation channels 2 Bits per sample 16 b Sampling rate 80 MHz Data rate per element 1.28 Gb/s 0.16 GB/s polarisation Data rate per element 2.56 Gb/s 0.32 GB/s Since beam-formed data is created by combining per element data, the data rate of each beam is the same as that for a single element. a) Full 30MHz bandwidth beam-formed data without transmitter sampling, 25% transmitting duty cycle (i.e. receiving for 75% of real-time): Burst data rate at all sites. We give these in turns of bits and bytes per second, though the average data rate in a second may be less than this Gb/s 12.8 Gb/s 25.6 Gb/s 0.32 GB/s 1.6 GB/s 3.2 GB/s 1.15 TB/hour 5.76 TB/hour 11.5 TB/hour 27.6 TB/day 138 TB/day 276 TB/day Average data rate at the central site: 0.24 GB/s 1.2 GB/s 2.4 GB/s TB/hour 4.32 TB/hour 8.63 TB/hour 20.7 TB/day 104 TB/day 207 TB/day Average data rate at the remote sites (sampling for 25% of real-time): Average data rate at a single site: 0.08 GB/s 0.4 GB/s 0.8 GB/s TB/hour 1.44 TB/hour 2.88 TB/hour 6.91 TB/day 34.6 TB/day 69.1 TB/day Total average data rate at all (4) remote sites: 0.32 GB/s 1.6 GB/s 3.2 GB/s 1.15 TB/hour 5.76 TB/hour 11.5 TB/hour 27.6 TB/day 138 TB/day 276 TB/day 20

21 Total average data rate for central site and remotes sites: 0.56 GB/s 2.6 GB/s 5.6 GB/s 2.01 TB/hour 10.1 TB/hour 20.1 TB/hour 48.4 TB/day 242 TB/day 484 TB/day 1 Central Beam + 10 beams per remote site 3.44 GB/s 12.4 TB/hour 297 TB/day One hour of such operation, using 1 central beam and 10 beams at each remote site, would use 6% of the annual budgeted archive capacity of 200TB. b) Full 30MHz bandwidth beam-formed data with transmitter sampling, 25% transmitting duty cycle: Average data rate for central site is the same as the burst data rate option 1(a), total average data rate for remote sites remains unchanged from option 1(a). Total average data rate for central site and remotes sites: 0.64 GB/s 3.2 GB/s 6.4 GB/s 2.31 TB/hour 11.5 TB/hour 23.1 TB/hour 55.3 TB/day 276 TB/day 553 TB/day 1 Central Beam + 10 beams per remote site 3.52 GB/s 12.7 TB/hour 304 TB/day One hour of such operation, using 1 central beam and 10 beams at each remote site, would use 6% of the annual budgeted archive capacity of 200TB. 21

22 4. Option 2 Auto-correlated data as used in the current EISCAT system, with higher temporal and spatial than current standard operations. Required minimum range 50 km Required maximum range 1500 km Required range resolution 300 m Range increments 4833 Number of lags per range 100 lags No. of data per matrix 4.83E+05 data profile Data storage per datum 4 bytes Number of Polarisations 2 Data per correlated matrix 3.87 MB Post-correlation integration 1 s Total data rate per beam 3.87 MB/s a) Auto-correlated 1s-integrated beam-formed data without transmitter sampling, 25% duty cycle: Burst data rate at all sites: Mb/s Mb/s Mb/s 3.87 MB/s MB/s MB/s GB/hour GB/hour GB/hour GB/day GB/day GB/day 9.35 TB/month TB/month TB/month TB/year TB/year TB/year Average data rate at central site: 2.90 MB/s MB/s MB/s GB/hour GB/hour GB/hour GB/day GB/day GB/day 7.02 TB/month TB/month TB/month TB/year TB/year TB/year Average data rate at a single remote site: 0.97 MB/s 4.83 MB/s 9.67 MB/s 3.48 GB/hour GB/hour GB/hour GB/day GB/day GB/day 2.34 TB/month TB/month TB/month TB/year TB/year TB/year Total average data rate for all (4) remote sites: 3.87 MB/s MB/s MB/s 22

23 13.92 GB/hour GB/hour GB/hour GB/day GB/day GB/day 9.35 TB/month TB/month TB/month TB/year TB/year TB/year Total average data rate for central and remote sites: 6.77 MB/s MB/s MB/s GB/hour GB/hour GB/hour GB/day GB/day GB/day TB/month TB/month TB/month TB/year TB/year TB/year 1 Central Beam + 10 beams per remote site MB/s GB/hour 3.59 TB/day TB/month 1.31 PB/year One month of such operation, using 1 central beam and 10 beams at each remote site, would use 50% of the annual budgeted archive capacity of 200TB. b) Auto-correlated 1s-integrated beam-formed data with transmitter sampling, 25% duty cycle: Average data rate for central site is the same as the burst data rate option 2(a), total average data rate for remote sites remains unchanged from option 2(a). Total average data rate for central site and remotes sites: 7.73 MB/s MB/s MB/s GB/hour GB/hour GB/hour GB/day GB/day GB/day TB/month TB/month TB/month TB/year TB/year TB/year 1 Central Beam + 10 beams per remote site MB/s GB/hour 3.67 TB/day TB/month 1.34 PB/year One month of such operation, using 1 central beam and 10 beams at each remote site, would use 51.5% of the annual budgeted archive capacity of 200TB. 23

24 5. Option 3 Auto-correlated data as used in the current EISCAT system, with higher temporal and spatial than current standard operations. Required minimum range 50 km Required maximum range 1500 km Required range resolution 300 m Range increments 4833 Number of lags per range 100 lags No. of data per matrix 4.83E+05 data profile Data storage per datum 4 bytes Number of Polarisations 2 Data per correlated matrix 3.87 MB Post-correlation integration 2 s Total data rate per beam 1.93 MB/s a) Auto-correlated 2s-integrated beam-formed data without transmitter sampling, 25% duty cycle: Burst data rate at central site: 15.2 Mb/s Mb/s 152 Mb/s 1.93 MB/s 9.67 MB/s MB/s 6.96 GB/hour GB/hour GB/hour GB/day GB/day GB/day 4.68 TB/month TB/month TB/month TB/year TB/year TB/year Average data rate at central site: 1.45 MB/s 7.25 MB/s MB/s 5.22 GB/hour GB/hour GB/hour GB/day GB/day GB/day 3.51 TB/month TB/month TB/month TB/year TB/year TB/year Average data rate at a single remote site: 0.48 MB/s 2.42 MB/s 4.83 MB/s 1.74 GB/hour 8.70 GB/hour GB/hour GB/day GB/day GB/day 1.17 TB/month 5.85 TB/month TB/month TB/year TB/year TB/year Total average data rate for all (4) remote sites: 1.93 MB/s 9.67 MB/s MB/s 24

25 6.96 GB/hour GB/hour GB/hour GB/day GB/day GB/day 4.68 TB/month TB/month TB/month TB/year TB/year TB/year Total average data rate for central and remote sites: 3.38 MB/s MB/s MB/s GB/hour GB/hour GB/hour GB/day GB/day GB/day 8.18 TB/month TB/month TB/month TB/year TB/year TB/year 1 Central Beam + 10 beams per remote site MB/s GB/hour 1.8 TB/day TB/month TB/year One month of such operation, using 1 central beam and 10 beams at each remote site, would use 25% of the annual budgeted archive capacity of 200TB. b) Auto-correlated 2s-integrated beam-formed data with transmitter sampling, 25% duty cycle: Average data rate for central site is the same as the burst data rate option 3(a), total average data rate for remote sites remains unchanged from option 3(a). Total average data rate for central site and remotes sites: 3.87 MB/s MB/s MB/s GB/hour GB/hour GB/hour GB/day GB/day GB/day 9.35 TB/month TB/month TB/month TB/year TB/year TB/year 1 Central Beam + 10 beams per remote site MB/s GB/hour 1.84 TB/day TB/month TB/year One month of such operation, using 1 central beam and 10 beams at each remote site, would use 26% of the annual budgeted archive capacity of 200TB. 25

26 6. Option 4 Auto-correlated data as used in the current EISCAT system, with higher temporal and spatial than current standard operations. Required minimum range 75 km Required maximum range 1500 km Required range resolution 3000 m Range increments 475 Number of lags per range 100 lags No. of data per matrix 4.83E+05 data profile Data storage per datum 4 bytes Number of Polarisations 2 Data per correlated matrix 0.38 MB Post-correlation integration 2 s Total data rate per beam 0.19 MB/s a) Auto-correlated 2s-integrated beam-formed data without transmitter sampling, 25% duty cycle: Burst data rate at central site: 1.52 Mb/s Mb/s 15.2 Mb/s 0.19 MB/s 0.95 MB/s 1.90 MB/s 0.68 GB/hour 3.42 GB/hour 6.84 GB/hour GB/day GB/day GB/day 0.46 TB/month 2.30 TB/month 4.60 TB/month 6.00 TB/year TB/year TB/year Average data rate at central site: 0.14 MB/s 0.71 MB/s 1.43 MB/s 0.51 GB/hour 2.57 GB/hour 5.13 GB/hour GB/day GB/day GB/day 0.34 TB/month 1.72 TB/month 3.45 TB/month 4.50 TB/year TB/year TB/year Average data rate at a single remote site: 0.05 MB/s 0.24 MB/s 0.48 MB/s 0.17 GB/hour 0.86 GB/hour 1.71 GB/hour 4.10 GB/day GB/day GB/day 0.11 TB/month 0.57 TB/month 1.15 TB/month 1.50 TB/year 7.49 TB/year TB/year 26

27 Total average data rate for all (4) remote sites: 0.19 MB/s 0.95 MB/s 1.90 MB/s 0.68 GB/hour 3.42 GB/hour 6.84 GB/hour GB/day GB/day GB/day 0.46 TB/month 2.30 TB/month 4.60 TB/month 6.00 TB/year TB/year TB/year Total average data rate for central and remote sites: 0.33 MB/s 1.66 MB/s 3.33 MB/s 1.20 GB/hour 5.99 GB/hour GB/hour GB/day GB/day GB/day 0.80 TB/month 4.02 TB/month 8.04 TB/month TB/year TB/year TB/year 1 Central Beam + 10 beams per remote site 2.04 MB/s 7.35 GB/hour GB/day 4.94 TB/month TB/year One year of such operation, using 1 central beam and 10 beams at each remote site, would use 32% of the annual budgeted archive capacity of 200TB. b) Auto-correlated 2s-integrated beam-formed data with transmitter sampling, 25% duty cycle: Average data rate for central site is the same as the burst data rate option 4(a), total average data rate for remote sites remains unchanged from option 4(a). Total average data rate for central site and remotes sites: 0.38 MB/s 1.90 MB/s 3.80 MB/s 1.37 GB/hour 6.84 GB/hour GB/hour GB/day GB/day GB/day 0.92 TB/month 4.60 TB/month 9.19 TB/month TB/year TB/year TB/year 1 Central Beam + 10 beams per remote site 2.09 MB/s 7.52 GB/hour GB/day 5.06 TB/month PB/year One year of such operation, using 1 central beam and 10 beams at each remote site, would use 33% of the annual budgeted archive capacity of 200TB. 27

28 Interferometry The interferometry system makes use of the existing samples coming from the beam formed modules. However, instead of the standard incoherent scatter receivers, there will be separate "interferometric receivers" from each of which the data must be recorded. Fortunately only the ion-line frequency is required as in Option 3. Input data rate to receivers 80 MSamples/s Required interferometry bandwidth 1 MHz Receiver data output 2.67 MSamples/s Resulting decimation factor 30 Data rate of interferometry (over assumed 19 modules) MSamples/s Number of polarisations 2 Number of bits per sample 16 b Data rate of interferometry Gb/s = MB/s = GB/hour = 17.5 TB/day = 490 TB/month However, the interferometric system will make use of a threshold detection system. This means that data will only be recorded when a certain level of coherence has been exceeded. This will result in greatly reduced data rates. It is extremely difficult to estimate the amount of data that will exceed this threshold as a percentage. We have assumed a threshold which is triggered 5% of the time, which significantly exceeds current expectations, but allows for any novel possibilities. Above-threshold samples 5 % Burst-rate MB/s Average rate 10.1 MB/s 36.5 GB/hour GB/day 24.5 TB/month TB/year One month of interferometry data (available at the central site only) represents 12% of the annual budgeted archive capacity of 200TB. 28

29 Appendix B: EISCAT3D Data Storage Specification Appendix B(1): Central Receiver Site - Permanent Archive (Single) Abstract This document describes the requirements for the central storage archive of the proposed EISCAT3D phased array incoherent scatter radar system. The system, which is planned to become the most advanced research radar in the world, will become operational by 2015, subject to securing the necessary funding from the various international partners. A design study into the EISCAT3D radar system is currently funded by the European Union s Sixth Framework Programme for research and technological development (FP6). The design study has been running since 2005, and finishes in May It will provide the material necessary to prepare a full proposal for the funding of the construction of the radar system, with proposals being addressed to national funding agencies as well as the 7 th Framework programme of the European Union. It is anticipated that the data storage component of this system will consist of one large central store deployed alongside the system s transceiver site and four smaller archives deployed at the remote receiver sites. 1. Introduction EISCAT3D is currently a design study for a project that aims to build a next-generation multistatic incoherent scatter radar (ISR) system for European scientists which will be superior to any other current or planned ISR system in the world. The planned system will produce a substantially greater volume of data and advanced networking and storage technology will be required to handle the transport and storage of this data. The dataset is anticipated to grow at the rate of approximately 200TByte per year (in multiple files which may vary in size from MBs to GBs in size). It is therefore planned that the initial size of the central archive should be on the order of 1 PetaByte (PByte) to allow the radar to operate for 5 years before upgrading may be required. A growth rate of 200TByte per year implies an average data rate of approximately 6MByte/s. Any system archive must be capable of operating with this input and output rate continuously. However, the system will also be required to operate as a short term buffer for the radar system, to allow for post-processing of data not required for permanent archiving. In this operational mode the data storage system will be required to handle sustained simultaneous input and output data rates of 4 GByte/s each on a continuous basis. 2. Aim of the Information-Gathering Exercise It should be stressed that the present document is not a formal invitation to tender (ITT), since we currently have no funding for system construction, only for system design. The aim of the present design phase is, however, to produce a substantially detailed blueprint for the construction of an advanced radar system, sufficient to give us a firm basis for estimating the full costs of construction when applying for funds, and to provide a detailed set of technical criteria for discussions with potential suppliers. Thus, although we are not officially asking for tenders, we hope that suppliers will make an effort to respond seriously to the requirements set 29

30 out here, on the basis that companies which can demonstrate that their system meets our specifications willl have a very good chance of being included in the eventual ITT. Any response to this document should, therefore, include material that convincingly demonstrates that the supplier understands the issues associated with extremely large datasets and the high input and output rates required of the archive. Responses should indicate in detail where the supplier is unable to comply with the specification issued. 3. Overall Requirements The complete system should have: A minimum of 1PByte of usable storage, composed of: o A minimum of 400TBytes of usable online storage o A maximum of 600TBytes of usable near-online storage (e.g. provided by automated tape system) or additional online storage. o Capable of accepting and storing data at a rate of at least 264 MBytes/s over all input channels and reading and outputting 695 MBytes/s over all output channels. (But must be capable of expanding to an output bandwidth of 1320 MBytes/s.) o Capable of 11 MBytes/s sustained on each input or output channel and with a minimum of 24 such input channels and 63 such output channels. (But must be capably of expanding to 120 such output channels.) A minimum of 56 TBytes of short-term (24 hours or less) storage available at all times o Capable of accepting and storing data at a rate of at least 1 GBytes/s over all input channels and reading and outputting 1.5 GBytes/s over all output channels. o With a minimum of 4 input channels and 8 output channels each capable of 0.16 GBytes/s or better on any one input or output channel. o With a minimum of 19 input channels and 19 output channels each capable of 6 MBytes/s or better on any one input or output channel. No more than 300 kw total power draw for the storage system. System and subsystem management software Monitoring and system analysis tools 36 months of hardware and software (all parts and labour inclusive) support Further detailed specifications are given below. The EISCAT3D central data store is not intended as a computing facility and as such no CPU or memory requirements are specified in this document. The supplier should however include such details in their response. The archive system is required to operate on a continuous basis (see sections 13 and 16 for support requirements), and with minimal operator intervention. Although support staff will be available on-site, secure remote IP based network access and control is considered essential. The archive system will be located at or near to the current EISCAT facility at Tromsø, Norway but a representative system or subsystem may be required to undergo evaluation for performance, stability and final documentation at Rutherford Appleton Laboratory, UK. 30

31 4. Phased Deployment Suppliers may suggest a single deployment archive system meeting the specifications of this document or a multi-phase deployment. In the case of a multi-phase deployment, the supplier must ensure that the initial deployment meets all networking specifications, and must deploy at least 300TB of the required online data storage in the initial phase. Phased deployment must ensure that 200TB per year TB of spare capacity are available. Major equipment installation in a phased deployment may occur no more frequently than once per year, and downtime caused by such deployment will count towards overall system downtime, as specified in section 13. If phased deployment is to exceed the 36 month warranty period or support contract period, as specified in section 16, the supplier must include the cost of extending said warranty or support contract until full deployment of the final phase. The warranty or support contract must cover the complete system and not just phases deployed after the required 36 month period. If phased deployment may lead to the deployment of heterogeneous subsystems, then the supplier is required to guarantee their satisfactory interoperability. The supplier must specify the maximum cost of each stage of phased deployment, after the initial deployment, but may estimate expected costs. (Estimated costs may include supplier s predictions of future reduced component costs.) Such estimated costs of later deployment stages must be justified by the supplier and deemed reasonable by the customer. If justified and reasonable they will be used to evaluate the cost of the system. 5. Operating Modes The central data archive will have a number of distinct operating modes, all of which will be in simultaneous operation. These operating modes have a wide range of bandwidth and archiving duration requirements. It is not required that a single homogeneous system should address all of these requirements. The supplier may, if desired, supply a number of interconnected heterogeneous subsystems capable of handling these requirements though such subsystems must meet the other requirements (such as support, uptime and operating environment requirements) specified in this document. 1. A number of ring-buffers are required to store data from the radar system. Each ring-buffer will be configured as a first-in first-out (FIFO) queue and will continually overwrite data previously stored. Data will be written once but it may be read multiple times. In this operating mode the central data store will be required to guarantee simultaneous sustained data read and write. a) 19 ring-buffers each with 2 input channels and 2 output channels of 6 MBytes/s each. Each ring-buffer must be capable of storing the input data for a minimum of 30 minutes, requiring backing storage of 20 GBytes each (19*20 = 380 GBytes total). Each ring-buffer must be capable of simultaneously allowing writing and reading of data at the total maximum input and output bandwidths. b) 2 ring-buffers each with 2 input channels and 4 output channels of 0.16 GBytes/s each. Each ring-buffer must be capable of storing the input data for a minimum of 24 hours, requiring backing stores of 27.7 TBytes each (2*27.7 = 55.4 TBytes total). Each ringbuffer must be capable of simultaneously allowing writing and reading of data at the total maximum input and output bandwidths. Note the requirement for twice the total output bandwidth. 31

32 2. Permanent archiving of incoherent scatter (IS) data products. This will be heavily write and read biased, with a low frequency of updates to previously written data expected. This is a low bandwidth mode, requiring 55 MBytes/s input and 275 Mbytes/s output bandwidth. A total of at least 30 channels will be required: 5 input and 25 output channels each capable of 11 MBytes/s. All numbers given are sustained data rates. 3. Semi-permanent archiving of interferometry scientific data products. Interferometry data products will be substantially larger than those produced by the IS radar system. For this reason it will not be possible to store all scientifically interesting data. Instead each data record input into the archive will be labelled with a priority number. The interferometry data shall receive two overlapping allocations from the archive administrator. The first of these will be a yearly quota of storage, which will allow it to co-exist with the IS data products. The second allocation will be for a shorter period, for example two months, and will generally be the relevant fraction of the long period allocation (i.e. 1/6 of the yearly allocation if the shorter period is 2 months). This shorter period allocation will operate as a priority buffer. When this allocation has been filled new data with a priority number greater than the lowest priority number already in the buffer will overwrite that data. Data older than the shorter period (e.g. 2 months) will be transferred to the long period allocation and permanently archived. This system will require 209 MBytes/s input bandwidth over 19 input channels (each of 11 MBytes/s). Provisions must be included for eventually expanding the output interface as required, up to a maximum of 95 (5 x 19) 11 Mbyte/s channels; however, in the initial deployment phase only 38 (2x19) of these will be populated. Initially therefore the subsystem will require a bandwidth of 418 MBytes/s output bandwidth rising to a maximum of 1045 Mbytes/s output bandwidth. All numbers given are sustained data rates. 6. Data Storage Specifications The total permanent archive storage requirement for the fully-deployed archive system is 1PByte. Usage patterns by current EISCAT users indicate that demand is greatest for the most recent data. For this reason 2 years of data (equivalent to 400TBytes) must be available online with only networking delay. The remaining data capacity may be online or may be near-online, with user requests for data taking no more than a maximum of 1 hour. If a near-online solution is chosen the supplier should detail the software designed to track and notify users of data availability and the size of the additional online capacity for storing recently requested data. The archive system will have to migrate older online data into the near-online system fully automatically, deal with user requests for data in the online and nearonline systems transparently and retrieve data from near-online storage fully automatically without any manual intervention. Although the archive system will have backup in the form of offsite replication, integrity of the data is an important concern. No single component failure may result in loss of data. The archive system s file system must be capable of dealing with files ranging in size from megabytes to tens of gigabytes. 32

33 7. Networking Specifications All bandwidth figures quoted in this section are for data only. Suppliers should therefore include overhead for traffic control and transfer protocols. Data are expected to be fed into the central data archive as rapidly as they are collected at the central and remote receiver sites. This means that the archive must be capable of storing about 4 MBytes/s from the central site and 10 MBytes/s from each of the four remote sites at all times. Input to the data store is expected to comprise multiple 10 Gbit/s Ethernet links or equivalent, e.g. FibreChannel links, for data from the central site and a single 1 Gbit/s Ethernet from each of the remote sites. Simultaneously, the archive must be capable of accommodating at least two external 1 Gbit/s Ethernet connections, each accessing and reading out already stored data at MBytes/s. The readout operations must not reduce the sustained storage rate. Latency in the order of some seconds between the storing of data and the reading out of the same data is acceptable. Simultaneous to the mentioned permanent archiving of data and its networking requirement, the data store will be temporarily storing data at much higher data rates. In this mode the store will have to manage a total input data rate of 4 GByte/s. This is expected to consist of 3.2 GBytes/s from the central site and 0.1 GBytes/s from each of the remote sites. (Some remote sites may be capable of more than one connection of 0.1 GBytes). Individual data channels will have a bandwidth requirement of 0.16 GB/s. The supplier should include some provision for redundancy in the networking fabric of the data store. 8. Software Specifications 8.1. Operating System The underlying operating system, if accessible or required, should be one for which it would be reasonable to assume that experienced personnel would be readily available, or in which existing EISCAT staff would require only minimal retraining. (Variants of the UNIX operating system such as Linux or Solaris would meet this latter requirement.) If the underlying operating system is not accessible and/or if personnel experienced in administering it are likely to be rare or expensive then this must be justified by the supplier Clustering/Node Management Software A simple and easy tool or tool suite should be used to install, manage, update, and monitor all routine aspect of the system and any important subsystems. In particular a good system analysis tool should be provided to aid fault-finding. The supplier should also outline the software and operating system s ability to recover from corruption or major system failure Network Monitoring Software The software system should be able to provide instantaneous and historical overviews of inbound and outbound network bandwidth usage and latency at channel resolution Offsite Backup Solution It is intended that the system s main backup will be provided by either another single large data store in one of the EISCAT Associations member nations, or by a series of smaller 33

34 stores spread amongst the Associations members. These solutions may already be in place at the time of construction or may be placed out to tender by the members. The supplier should indicate if offsite replication software is included with the system. If there is a choice of such software the supplier should supply details of each. The supplier should also indicate such software s compatibility with other storage solutions as well as its ability to achieve offsite replication with heterogeneous data stores of mixed size. 9. Total Cost of Ownership (TCO) The supplier should, if available, provide any studies indicating the systems total cost of ownership (TCO). Running the data storage system would become one of the EISCAT Associations three largest expenses (along with personnel and radar power) so it is important that the supplier provide estimates of the ongoing operating cost of their system beyond the immediate capital cost. 10. Physical Size and Requirements The supplier should specify the physical dimensions of the system. This should include the dimensions of significant subsystems (e.g. racks) as well as the total size of the system. The supplier should specify the recommended operating environment for the system. This should include full specifications for power, temperature, humidity and cooling requirements at a minimum. If there are any other requirements or recommendations for housing the system the supplier should indicate them. Preference may be given to systems which are compact, low-power and/or tolerant of variable climates as this will become part of the systems overall capital and operating costs. Operating noise is not considered a substantial constraint but should be noted. 11. Scalability and Migration At estimated data rates it is anticipated that a one PByte store will meet user requirements for approximately 5 years. However, the EISCAT Association is a research organisation and as such its requirements can change rapidly with the ongoing progress and requirements of its users. For this reason it may be necessary to expand the central data archive within the 5 year period. Suppliers should indicate the scalability of their solution. They should in particular note the additional physical requirements of a doubling in size (to 2 PByte). The EISCAT Association has existed for more than 25 years and has no future date for its dissolution. Part of its responsibility is to ensure that data it has collected is available to future scientists. Suppliers should therefore supply details of any potential difficulties in migrating data out of their proposed storage system. 12. Environmental Specifications The system will be sited inside the Arctic Circle in Northern Norway and, although the system will be housed in a climate-controlled facility, this places some additional constraints on the design of the archive system. Extreme environmental conditions can make the operation of cooling and air-conditioning impossible. For this reason details should be provided on the 34

35 system s environmental tolerances in a shutdown state, and on the time and any manual intervention required to place it safely in such a state. (It is not required that the system operate in an arctic environment!) The supplier should specify the time required to bring the system from online status to a fully shutdown state, and from such a shutdown state back to online status. The supplier should indicate whether either operation can be safely performed entirely remotely or whether manual intervention is required. If remote shutdown and start is possible the supplier should indicate the methods available. If manual intervention is required the supplier should indicate the nature of this intervention and the skill level required by any operator(s). 13. Error Recovery Requirements Overall Downtime Reliability of the archive system as a whole is expected to be on the order of two nines (99%), or unavailable for 4 days or less per year. Individual components or subsystems may be offline but the system as a whole is expected to be substantially available. It should be possible to perform routine maintenance, such as replacement of failed parts, without bringing the system as a whole offline. Scheduled maintenance that brings the entire system down should not exceed 1 day per year. It is essential that the supplier provide suitable fault analysis tools and good documentation on how to handle faults. It may benefit the supplier to provide secure remote diagnosis tools Hardware Failure It is expected (though not required) that the system will be made of a large number of identical subsystems, each comprised of standard components. In such a case the supplier will be required to provide adequate hot spares on site. The supplier will also be required to guarantee an adequate supply of spare parts on all systems for the period of the warranty and to dispatch components for which hot spares are not available on site within 2 working days. If the system is not configured as above, or if some parts are not redundant, the supplier will be expected to justify their reliability Response Time Any fault that disrupts the normal operation of the system should be resolved within 2 days. It is acknowledged that the systems location at Tromsø, Norway may make such a response time difficult but a best effort would be required. Preference may be given to suppliers who can guarantee a good response time. It is expected that there will be some local support available during working hours. It is acceptable for the supplier to await confirmation that any fault cannot be dealt with locally before escalating it further Responsibility on Faults The eventual ITT will be made on the basis of a complete working solution covering all hardware and software (excluding the fabric of the building in which it is to be sited). We would expect suppliers to be responsible for any software and/or hardware faults that may 35

36 arise under normal operation throughout the warranty period. All such faults should be resolved fully and normal service restored within 2 working days.if this requires a support contract then the cost of such should be included, as it will be evaluated as part of the overall cost of the system. 14. Documentation This should detail the exact specification of the equipment supplied. It should also detail the system setup, management, user administration, software updates, hardware maintenance and replacement, and error recovery. 15. Training Adequate training must be provided by the supplier to ensure that the local administrator can operate and take over the system correctly and safely. The supplier should indicate the (longterm) availability and cost of training additional local administrators. 16. Warranty The supplier should indicate the duration of any hardware and software warranty, including technical support. See section 13. The supplier should indicate whether longer warranty periods are available, and if so the cost of such extensions. (A sufficiently comprehensive support contract may be considered in lieu of a warranty). The cost of additional years of support may be considered when evaluating the value of the system. 36

37 Appendix B(2): Remote Receiver Site - Remote Site Data Stores (x4) Abstract This document describes the requirements for the networked remote site data stores of the proposed EISCAT3D phased array incoherent scatter radar system. The system, which is planned to become the most advanced research radar in the world, will become operational by 2015, subject to securing the necessary funding from the various international partners. A design study into the EISCAT3D radar system is currently funded by the European Union s Sixth Framework Programme for research and technological development (FP6). ). The design study has been running since 2005, and finishes in May It will provide the material necessary to prepare a full proposal for the funding of the construction of the radar system, with proposals being addressed to national funding agencies as well as the 7 th Framework programme of the European Union. It is anticipated that the data storage component of this system will consist of one large central store deployed alongside the system s transceiver site and four smaller archives deployed at the remote receiver sites. 1. Introduction EISCAT3D is currently a design study for a project that aims to build a next-generation multistatic incoherent scatter radar (ISR) system for European scientists which will be superior to any other current or planned ISR system in the world. The planned system will produce a substantially greater volume of data and advanced networking and storage technology will be required to handle the transport and storage of this data. The remote receiver sites will be located in a potentially hostile environment, at the extreme end of communication and power links which will probably have to be partially purpose built. Although it is intended that in standard operations data collected from these remote radar receiver sites will be immediately forwarded to the central archive by network link, there are two scenarios in which short-term storage would be required. In the case of the failure of the network connection between the central and remote site it may be possible for the remote site to continue collecting data to a pre-programmed schedule. Alternatively a redundant low-bandwidth network link may remain available to receive control commands and send status information, but the link would be unsuitable for transmission of scientific data. In these cases the remote archive would be required to store data until such time as the high-bandwidth network link can be restored and data can be forwarded to the permanent archive. The system will also be required to operate as a short term buffer for the radar system, to allow for post-processing of data not required for permanent archiving. In this operational mode the data storage system will be required to handle sustained simultaneous input and output data rates of 4 GByte/s each on a continuous basis. 2. Aim of the Information-Gathering Exercise It should be stressed that the present document is not a formal invitation to tender (ITT), since we currently have no funding for system construction, only for system design. The aim of the present design phase is, however, to produce a substantially detailed blueprint for the 37

38 construction of an advanced radar system, sufficient to give us a firm basis for estimating the full costs of construction when applying for funds, and to provide a detailed set of technical criteria for discussions with potential suppliers. Thus, although we are not officially asking for tenders, we hope that suppliers will make an effort to respond seriously to the requirements set out here, on the basis that companies which can demonstrate that their system meets our specifications willl have a very good chance of being included in the eventual ITT. Any response to this document should include material that demonstrates that the supplier understands the issues associated with hardware reliability in hostile (cold) environments. Responses should indicate in detail where the supplier is unable to comply with the specification issued. Suppliers should note that four identical remote site data stores will be required. 3. Overall Requirements The complete system should have: A minimum of 70TBytes of usable online storage Capable of accepting and storing data at a rate of at least 1 GBytes/s over all input channels and reading and outputting 1 GBytes/s over all output channels. At least twenty inputs channel capable of at least 0.16 GBytes/s each and at least one output channel capable of at least GByte/s. No more than 30 kw total power draw for the storage system. System and subsystem management software Monitoring and system analysis tools 36 months of hardware and software (all parts and labour inclusive) support Further detailed specifications are given below. The EISCAT3D remote site data stores are not intended as computing facilities and as such no CPU or memory requirements are specified in this document. The supplier should however include such details in their response. The data store is required to operate on a continuous basis (see sections 13 and 16 for support requirements), and with no operator intervention apart from periodic maintenance. Support staff will not be available on-site, so no solution without secure remote IP based network access and control will be considered. The data stores would be located at a number of sites in Norway, Sweden and Finland but a representative system might have to undergo evaluation for performance, stability and final documentation at Rutherford Appleton Laboratory, UK. 4. Operating Modes The remote site data store will have two distinct operating modes which will be run simultaneously. The principal operational mode is as a ring buffer to allow for data flowing directly from the radar array. The data collected in this operating mode will not be permanently archived, but instead will be overwritten within 24 hours. However, it may be read multiple times. In this operating mode the remote site data store will be required to guarantee simultaneous sustained data read and write. 38

39 This operating mode will require a significant overall bandwidth and a large number of wide channels. Total input bandwidth over the required 20 input channels is 1 GBytes/s and each channel must be capable of sustaining a minimum of 0.16 GBytes/s. Only one output channel capable of GBytes/s would be required. The second mode will temporarily archive scientific data when transfer bandwidth to the central data store is either insufficient or unavailable. This will be heavily write and read biased, with a low frequency of updates to previously written data expected. This is also the lower bandwidth mode, requiring no more than 1 MBytes/s on one channel each of input and output bandwidth. 5. Data Storage Specifications The total storage requirement for each fully-deployed data store is 70 TBytes. This must be in the form of immediately accessible online storage. (Scientific users may co-locate equipment capable of processing and/or reducing the stored data, so recent data must be immediately available over the site s local area network.) Although the remote site data store is intended only as a staging post before permanent archiving at a central site, integrity of the data is an important concern. No single component failure may result in loss of data. The data store s file system must be capable of dealing with files ranging in size from megabytes to tens of gigabytes. 6. Networking Specifications All bandwidth figures quoted in this section are for data only. Suppliers should therefore include overhead for traffic control and transfer protocols. Data is expected to be fed into the central data archive as rapidly as it is collected at the receiver sites. The remote site data stores will not therefore, under normal operation, be required to store data. However, irregularly network connectivity may be lost and data will have to be stored locally until the connection is restored. Under these circumstances the data store will have to deal with input data rates of 10MBytes/s. At all times sustained data rates will significantly exceed these levels and the data store will have to manage a total input data rate of 1.6GByte/s continuously. Individual data channels will have a bandwidth requirement of 0.16GB/s. Simultaneously, the archive must be capable of accommodating at least one external 1 Gbit/s Ethernet connection, accessing and reading out already stored data at MBytes/s. The readout operations must not reduce the sustained storage rate. Latency in the order of some seconds between the storing of data and the reading out of the same data is acceptable. The supplier should include some provision for redundancy in the networking fabric of the data store. 39

40 7. Software Specifications 7.1 Operating System The underlying operating system, if accessible or required, should be one for which it would be reasonable to assume that experienced personnel would be readily available, or in which existing EISCAT staff would require only minimal retraining. (Variants of the UNIX operating system such as Linux or Solaris would meet this latter requirement.) If the underlying operating system is not accessible and/or if personnel experienced in administering it are likely to be rare or expensive then this must be justified by the supplier. 7.2 Clustering/Node Management Software A simple and easy tool or tool suite should be used to install, manage, update, and monitor all routine aspect of the system and any important subsystems. In particular a good system analysis tool should be provided to aid fault-finding. The supplier should also outline the software and operating system s ability to recover from corruption or major system failure. 7.3 Network Monitoring Software The software system should be able to provide instantaneous and historical overviews of inbound and outbound network bandwidth usage and latency at channel resolution. 7.4 Offsite Backup Solution The data store at the remote site will forward data to the permanent central archive. It must confirm that the data has been correctly received and stored at that central archive before deleting it locally. Suppliers should detail any network backup or offsite replication software which may be supplied with the archive system to achieve this goal. If there is a choice of such software the supplier should supply details of each. The supplier should also indicate such software s compatibility with other storage solutions. 8. Total Cost of Ownership (TCO) The supplier should, if available, provide any studies indicating the data store s total cost of ownership (TCO). The maintenance and running of the data storage system will be an important part of the overall cost of running the remote sites, so it is important that the supplier provide estimates of the ongoing operating cost of their system beyond the immediate capital cost. 9. Physical Size and Requirements The supplier should specify the physical dimensions of the data storage system. This should include the dimensions of significant subsystems (e.g. racks) as well as the total size of the system. The supplier should specify the recommended operating environment for the system. This should include full specifications for power, temperature, humidity and cooling requirements at a minimum. If there are any other requirements or recommendations for housing the system the supplier should indicate them. 40

41 Preference may be given to systems which are compact, low-power and/or tolerant of variable climates as this will become part of the systems overall capital and operating costs. Operating noise is not considered a substantial constraint but should be noted. 10. Environmental Specifications The remote site data stores will be located at sites inside the Arctic Circle in Northern Norway, Sweden and Finland. It is intended that these sites will remain unmanned. Although the system will be housed in a climate-controlled facility, these locations will place constraints on the design of the archive system. It is most important for suppliers to note that, due to their remote locations and the nature of the environment, it will be impossible to guarantee that the sites will be electrically powered at all times. Even if power supplies can be maintained extreme environmental conditions can make the operation of cooling and air-conditioning impossible. For these reasons details should be provided on the system s environmental tolerances in a shutdown state, and on the time required to place it safely in such a state. (It is not required that the system operate in an arctic environment!) The supplier should specify the minimum non-operational temperature and humidity requirements of the system. The supplier should specify the time required to bring the system from online status to a fully shutdown state, and from such a shutdown state back to online status. The supplier should indicate whether either operation can be safely performed entirely remotely. If remote shutdown and restart is possible the supplier should indicate the methods available. No system requiring a manual start will be considered acceptable. 11. Error Recovery Requirements 11.1 Overall Downtime Reliability of each data store is expected to be on the order of 97%, or unavailable for 10 days or less per year. Individual components or subsystems may be offline but the system as a whole is expected to be substantially available. It should be possible to perform routine maintenance, such as replacement of failed parts, without bringing the system as a whole offline. Scheduled maintenance that brings the entire system down should not exceed 1 day per year. The target of 97% excludes environmental circumstances which would require the system to shutdown, such as loss of power or an operating environment outside the supplier specifications due to climate control failure. Incidents which occur while the site is inaccessible, and which bring the system offline for 5 days or longer will be deemed to have lasted 5 days for the purposes of calculating the supplier s downtime requirements. It is essential that the supplier provide suitable fault analysis tools and good documentation on how to handle faults. It will benefit the supplier to provide secure remote diagnosis tools Hardware Failure It is expected (though not required) that the system will be made of a large number of identical subsystems, each comprised of standard components. In such a case the supplier 41

42 will be required to provide adequate hot spares on site or at a designated EISCAT Association facility. The supplier will also be required to guarantee an adequate supply of spare parts on all systems for the period of the warranty and to dispatch components for which hot spares are not available on site or at the designated facility within 2 working days. If the system is not configured as above, or if some parts are not redundant, the supplier will be expected to justify their reliability Response Time Any fault that disrupts the normal operation of a data store should be resolved within 5 days, if the site is accessible. It is acknowledged that the locations in Northern Scandinavia may make such a response time difficult but a best effort would be required. Preference may be given to suppliers who can guarantee a good response time. No staff will be present on-site during normal operation, but staff from the EISCAT Association may be available to assist the supplier s engineers if notified in advance Responsibility on Faults This eventual ITT will be made on the basis of a complete working solution covering all hardware and software (excluding the fabric of the building in which it is to be sited). We would expect suppliers to be responsible for any software and/or hardware faults that may arise under normal operation throughout the warranty period. All such faults should be resolved fully and normal service restored within 5 working days with no charge to the client. If this requires a support contract then the cost of such should be included, as it will be evaluated as part of the overall cost of the system. 12. Documentation This should detail the exact specification of the equipment supplied. It should also detail the system setup, management, user administration, software updates, hardware maintenance and replacement, and error recovery. 13. Training Adequate training must be provided by the supplier to ensure that the local administrator can operate and take over the system correctly and safely. The supplier should indicate the (longterm) availability and cost of training additional local administrators. 14. Warranty The supplier should indicate the duration of hardware and software warranty provided by the supplier, including technical support. See section 13. The supplier should indicate whether longer warranty periods are available, up to 5 years, and if so the cost of such extensions. (A sufficiently comprehensive support contract may be considered in lieu of a warranty). The cost of additional years of support may be considered when evaluating the value of the system. 42