HERSCHEL/PACS On-board Reduction Flight Software

Transcription

1 HERSCHEL/PACS On-board Reduction Flight Software Roland Ottensamer and Franz Kerschbaum University of Vienna, Department of Astronomy, Türkenschanzstr. 17, A-118 Vienna, Austria ABSTRACT PACS, the Photodetector Array Camera and Spectrometer for the HERSCHEL Space Observatory (HSO) has a sophisticated on-board software performing data reduction and compression to reach a datarate that is compatible with the downlink requirements. For this purpose, highly specialized lossy and lossless techniques are combined to retain a maximum of the original signal quality. The FM generation of detector hardware for the HSO has given rise to adaptation of the already qualified flight software. In response to changed detector characteristics and observational needs the reduction/compression scheme has undergone substantial modifications, such as an additional quantization step in photometry. In spectroscopy, on-board deglitching has been sacrificed in favour of higher temporal resolution, thereby freeing CPU resources that were utilised for an improved semiadaptive arithmetic compression model for lossless compression. The modular concept allows for upgrades during the verification phase to increase in-flight performance. A detailed overview of the reduction/compression software and its capabilities is given, along with lessons learned from the FM instrument level test campaigns that had to be taken into consideration as well as demands from ground segment infrastructure to guarantee a sound operational phase. Keywords: compression, reduction, software, Herschel, PACS, on-board 1. PACS RAW SCIENCE DATA DILEMMA The Herschel satellite will be launched in early 29 together with Planck and being the biggest space telescope ever built it will explore the far-infrared universe with unprecedented sensitivity. Three instruments are carried on board the satellite, HIFI for frequencies ranging from GHz, SPIRE for wavelengths between 2 and 67 micron and PACS for the micron range. 1 PACS with its novel bolometer arrays for photometry and the two Ge:Ga spectrometer arrays has by far the largest number of pixels and also yields the highest datarates. In photometry, the array for the shorter wavelengths has 248 pixels, whereas the array for the red end has 512 pixels. Both are read at 4 Hz with bit precision. This leads to a raw science data rate of kbit/s. In addition to that, a header consisting of sixteen 32-bit parameters is attached to each science data frame for each array, adding another 4 kbit/s. In spectroscopy, the two matrices are made up of 26 stacks of 18 pixels for each of the two arrays and they are read non-destructively at 256 Hz also with bits precision, yielding 3744 kbit/s. We also have the header parameters in spectroscopy, yielding 256 kbit/s. Now the problem with these datarates is that operational and power budget constraints severely limit communication with the satellite and result in an allowed sustainable data rate of 12 kbit/s. That is typically times smaller than the raw science output of the PACS photometer and up to 4 times for the PACS spectrometer. Even the raw header data from one single spectrometer would already be too much for transmission! For this reason, relatively drastic measures have been included in the on-board software of our instrument, involving lossy steps like averaging and ramp fitting as well as lossless data compression techniques. Further author information: (Send correspondence to R.O.) R.O.: roland.ottensamer@univie.ac.at, Telephone: +43 () F.K.: franz.kerschbaum@univie.ac.at, Telephone: +43 () We use 124 as base for quantities in kbit.

2 Figure 1. Data flow inside the PACS warm electronics. From left to right: either the bolometer arrays or the Ge:Ga arrays are used at a time, depending on the mode of observation. Their data are read by the DMC, which structures them into dataframes by attaching a 64-byte header. Then the reduction and compression is done by the SPU units. The SPU produces a compressed data stream sliced into packets that get wrapped by the DPU to become standard telemetry packets. Aside from the nominal science data flow, each subunit is obliged to send Housekeeping (HK) a set of diagnostic counters and values at time intervals of 2 seconds. 2. ON-BOARD COMPUTERS PACS is a highly computerized instrument. 5 out of 8 processor boards are involved at any time in the data flow. The remaining 3 boards are redundant units. These boards all come with an 18-MHz ADI 212 Digital Signal Processor (DSP) and a SMCS 332 chip for communication. The detector arrays are read by the Detector and Mechanism Controller (DMC) built by Centre Spatial de Lige. The data are structured into frames, attached with a header consisting of 32-bit words and sent to the Signal Processing Unit (SPU). This header has parameters like observation identification and information about the instrument setup. Also the compression mode that should be used by the SPU is set as a header parameter here. The SPU reduces and compresses the data according to the data type, compression mode and the contents of on-board parameter tables, which are loaded during instrument set-up. After that, the data are shipped in packets to the Data Processing Unit (DPU), which has an interface to the spacecraft. The DPU hardware is built by Gavazzi under Istituto di Fisica dello Spazio Interplanetario (IFSI) contract and the DPU Software is also contributed by IFSI. Figure 1 shows the 5 on-board computers that are involved in nominal dataflow. Regardless whether the instrument is operating in photometry or in spectroscopy, the dataflow of the long-wavelength arrays (the red channel) and the short-wavelength arrays (the blue branch) goes separate ways through dedicated DMC and SPU units. Only the DPU, where everything comes together before storage in the solid state mass memory of the spacecraft, handles both channels. Each SPU and the DPU have back-up units on board, in case that the hardware of the nominal units becomes damaged in any way. 3. THE SIGNAL PROCESSING UNIT PACS has four independent SPUs built by CRISA under contract by the Instituto de Astrofisica de Canarias (IAC). Our Spanish colleagues also provide the startup software and low-level drivers. For all the data processing 4 MB of data memory and 3 MB of program memory are available. This is sufficient to buffer a few seconds of incoming data and still have space for processing. Due to the noisy character of the data there is not much redundancy to be expected within a few seconds and due to the drift in the detectors there are no longer periodicies to be exploited as well.

3 3.1 Task Management and Communication To handle the task management especially with all the simultaneously operating high speed communication links the compression software runs under the Virtuoso TM real-time operating system from WindRiver (previously developed by EONIC). It is linked with the executable during compilation. The drivers to access the SMCS and other hardware are provided by the hardware manufacturer. The operating system supplies the necessary functionality for multitasking, interrupt handling, signaling, semaphores and FIFOs. A number of tasks are running in parallel, most notably the input and output data buffering tasks, the compression task which processes the data in chops of a few seconds, the housekeeping task, which reports the software status every two seconds and the watch process, which monitors the command link with respect to high priority commands. The command link is also used for initialization of software relevant tables such as principal compression parameters or the detector selection tables used for masking bad pixels. It is also possible to upload a new software to the EEPROM via that link. One SPU communicates via three links: raw data are permanently received from DMC, compressed data are sent to the DPU and commands can also be received any time from the DPU. The warm electronics units use a Scalable Multichannel Communications Subsystem (SMCS) 332 chip which implements the Spacewire standard. It is nominally configured to handle up to 1 Mbit/s on each link. To meet the real-time requirements it was necessary to implement the input and output buffers as circular buffers, that are continuously filled and drained by independent tasks. 3.2 SPU Software structure PACS operates either as a photometer or as a spectrometer, and each detector type comes with two independent detector arrays for the red and the blue channel. Normally, we concentrate on reaching the necessary compression factors yet retaining as much original data as possible. 2 For some observations, temporal resolution is preferred over spatial resolution, for others the raw data of a selected sub-area are required. Other restrictions apply if PACS and SPIRE are operated in Parallel mode, sharing the available bandwidth. On the other hand there is Burst mode, which more than triples the available bandwidth if continuous operation is not scheduled. All these things have to be considered during on-board data processing leading to a variety of sequences and algorithms, commandable and/or adaptable to the data input and uploadable sets of parameters and tables to ensure a maximum of flexibility. Each SPU has a full copy of the data processing/compression software, no matter if it is installed in the red or in the blue unit. A SPU is not configured to its channel, the distinction is made only by looking at the metadata in the frame headers. In general, once the SPU software is booted and has established connection with the other warm electronics units, it is waiting to receive a start command from the DPU. Once this has been acknowledged, the compression threads are signaled to be started and data from the input link are buffered. The data are identified by their headers and the software branches accordingly into the necessary processing steps Compression Modes The full nominal sequence as illustrated in Figure 2 is not the only mode of operation. All processing steps are optional it s even possible not to do anything at all with the data, though that case is most unfavourable, as the price to be paid for a few seconds of raw data from all arrays is a 1-minute time-out, because the data cannot be written fast enough to the spacecraft and need to be buffered in the warm electronics units. What the compression software is meant to do is determined by: the header parameters, indicating detector type, channel and compression mode direct commands, used for starting and stopping the data flow, for updating software and tables and for switching to test and diagnostic modes the on-board tables, controlling the bad pixel masks as well as various parameters for fine-tuning, such as choosing the lossless compression algorithm

4 Figure 2. Default reduction and compression steps. Modules where raw data are irreversibly reduced are depicted with dark grey colour. From top to bottom: incoming data are tested for their integrity, the headers are separated from the data and passed to the header compression subroutine. The science data are filtered with the detector selection table mask and passed on to the reduction module. A few pixels are routed outside the normal data processing for inclusion of additional raw data in the downlink stream. The reduction module is where averaging or ramp fitting are performed, optionally including an additional rounding step. After that, the data are prepared in the decorrelation stage for the lossless compression. At the bottom of the scheme, the data streams that went different paths are reunited and sent to the DPU in packets ready for transmission. If all parameters are left at their initialization values, the software operates in default mode, which has the minimum of lossy operations for transmitting the data of all pixels. This mode has been used during most of the instrument level tests and will also be the standard in space. Default mode in photometry means averaging frames by 4 and lossless compression of the results. In spectroscopy we have generation of sub-means of 8 samples length and lossless compression. However, if the arrays are driven in very noisy settings, it is necessary to include an optional rounding step. Figure 2 gives an overview of the modules involved in the standard data processing of the SPU Metadata and Header Compression The origin and type of the data are reported in the frame header, as is the compression mode. This can resemble something like Photometry, red channel, default compression. Some of the header parameters are important for on-board data reduction, others such as the position of the filter wheel are needed on ground for pipeline processing and data product generation. As the header parameters have the same structure for all observations, the header compression is the same for all of them. Although the headers have to be compressed in a lossless way and in the worst case their data

5 rate already exceeds the available downlink rate by a factor of 2, we manage to yield a typical factor of 4 owing to the structured content of the headers, which contains parameters that stay the same for several frames or counters which increase or decrease in a steady manner. Only a few parameters, such as the chopper position or the grating position, are affected by a 2-bit noise. The principal algorithm used for header compression has been published 3 and is one of the few examples for a subroutine that has survived from the very beginning of the development until the flight model of the software without major modification needed. The frame headers have another very important purpose. In photometry, the data frames are synchronized with the movement of the chopper via a flag in the header, signaling the start and stop of a sequence. A start of a sequence marks the first frame of a number of frames to be averaged. That way it is guaranteed that averaging does not occur during a chopper transition. In spectroscopy, the thing is a little bit more tricky. Two header parameters are used for synchronization. One indicates the length of a ramp, the other one counts the sample index on the ramp. That way the compression software can synchronize onto the start of a ramp On-board Tables The on-board software has a compression parameter table for spectroscopy and one for photometry, plus detector selection tables for each detector array which are used to exclude any individual pixel. Most of the compression parameters deal with choosing different options for the decorrelation and lossless compression modules. By default, all pixels from the arrays are transmitted, but there are cases where it may not be needed to use the full array for observation. In addition to that, the detector will degrade over time and individual pixels will become defunct. So it is most reasonable to apply a pixel selection mask on the detector arrays. Depending on the number of selected pixels, the raw data could even be transmitted without reduction. Detector selection is handled by the SPU with uploadable tables. These tables have a unique identification number that allows to reconstruct the array geometry on ground. Detector selection tables are stored within the ground segment software, where the tools for preparation and upload are available. In the end, the datarate has to be met, no matter what parameters are tuned: (De)Selecting pixels. If raw data are to be transmitted, the compression factor of (photometry) and 4 (spectroscopy) have to be achieved by selecting only that fraction of total pixels from the arrays. Changing the number of frames to be averaged: instead of the default number of 4, any other number can be commanded, such as 8 in the photometer Parallel mode. In addition to averaging, the results obtained can be rounded by 1-2 bits to discard oversampled noise. In addition to the nominal data stream, a few pixels raw data can always be added to the downlink stream, mostly for verification purposes. Lossless compression of these so-called additional raw data pixels typically yields a factor of 2 here. 4. IMAGING PHOTOMETRY When PACS is operated as a photometer, 4 the two bolometer arrays are in use. The many stage readout electronics of the bolometers allow for different bias and gain adjustments, but most importantly, the gain applied at the warm electronics control box determines whether a low gain/low noise or high gain/high noise setting is used. The purpose of these configurations is of course to have a setup for observing weak and one for observing bright targets. Different ways of reading the detector elements are also available, leading to different output data types that are essentially distinguished whether the datatype is signed or unsigned. The gains have been readjusted for the FM model, leading to a substantially different noise magnitude if compared with the qualification model. Now, the readout noise typically varies between 4-8 bits at signal levels between 2-6 ADU. Figure 3 depicts a typical dataframe from the blue array. Even after bias subtraction, the variation from neighbouring pixels is slightly bigger than the readout noise owing to varying sensitivity. Therefore, it is better to The read-out electronics of the spectrometer generate 26 columns of 18 pixels although only 25 columns are equipped with Ge:Ga crystals. That 26 th ghost column is always deselected by the pixel mask, as it contains no useful information.

6 Figure 3. Appearance of a single frame from the blue bolometer array. Above all, it is dominated by the bias pattern. Different submatrices that constitute the full array are easily distinguishable. The level of brightness maps the -bit range to black= and white= Signal of pixel at x=24, y=22; frames Signal of pixel at x=24, y=22; frames ADU ADU Sample # Sample # Histogram of samples from pixel at x=24, y=22 on section ➀ Histogram of samples from pixel at x=24, y=22 on section ➃ 12 Frequency [1] Frequency [1] ADU ADU Figure 4. Typical pixel signal for a low noise (left) and a high noise setting (right). Below are the histograms, fitted with a Gaussian distribution of the same variance. The entropies for this pixel are 4.56 bits/sample in the left and 5.15 bits/sample in the right plot. concentrate on the temporal redundancies rather than on cosmetic aspects within the frame. The plots in Figure 4 show how a pixel signal typically looks like. The drift is in part due to 1/f noise and in part to instrumental effects. Histograms of the noise reveal its Gaussian nature and facilitate estimates for entropy encoding. 4.1 The effect of chopping To account for the high background signal in the observed wavelengths and to compensate instumental effects, the standard way of observing with the bolometer in point-source mode and raster observations is to have a chop/nod pattern at 1 Hz, leading to sections of 2 raw samples on the same chopper plateau. Although this

7 ADU Signal of pixel at x=24, y=22; frames Sample # Figure 5. Typical example of a chopped signal. On top of the chopper pattern, the drift and the readout noise are still visible. Because the data are processed in slices of a few seconds, only few chops fall into the same processing buffer. creates a regular pattern in the data, it is difficult to take advantage of during compression. First of all, the size of the chop is also subject to noise and secondly, the chop rate is not a parameter in the frame header and therefore not known to the compression software. Figure 5 has an example of a chopped signal for the same pixel as in Figure 4. Chopping is one of the main problems to consider in designing a proper decorrelation step. 4.2 The devised reduction scheme In reaction to the noise behaviour of the FM matrices, the reduction scheme, especially the decorrelation stage and the lossless compression had to be adapted. The result from the averaging process is still a -bit signed datatype. Averaging 4 frames reduces the noise by a factor of 2 which is 1 bit, but still the averages span over the whole dynamic range and the noise varies from 3-7 bits. The next step is to separate signal and noise, with signal comprising the bias offset including any form of drift, the chopping pattern and the actual source signal. This is approached in the decorrelation stage, which ought to produce a dataset as sparse as possible, with more or less the readout noise remaining around zero. The particularities of the data had to be considered in the design of the decorrelation, but keeping the limited processing resources in mind. The easiest operation with yet great effect is a simple differentiation of the data, with the sideeffect of increasing the noise by a factor of 2. In terms of datarate, 1 bit of noise means 25 kbit/s that cannot be compressed. This can be avoided by keyframing or offsetting from an average frame, but this is a hopeless attempt for data that are drifting and even chopped. Several predictor/corrector algorithms ranging from a running average to a full linear prediction of second order have been considered, but although there is a benefit for staring data they offer no match for chopped data. Tests with block-sorting transforms were exceeding the available CPU resources. Other decorrelations like orthonormal transforms had the same problem. So a simple differentiation for each pixel was chosen, followed by reformatting the data to an order allowing frames differentiation. These 2 differentiations increase the noise by 1 bit, but their combination eliminates very well the signal redundancy, any drift and even the chopping, with operations that all work in place and barely need any processing power. The result is that only 5-1% of the decorrelated values exceed an 8-bit range even for chopped observations, which is a good starting point for the lossless compression module (described in section 6). 4.3 Additional Rounding The default datarate after modification of the lossless compression to the new data is still too high for the high bias setting. As the reason for this is that the noise is actually oversampled by the high noise setting, we

8 Comparison of Averaging with and without Rounding on a 3s buffer for Pixel (24,22) raw data "perfect"average integer average additional 1-Bit rounding additional 2-Bit rounding ADU Sample Deviation from "perfect" Average Figure 6. Additional rounding compared with standard averaging. The spiky line is the pure raw data from that pixel. Four samples are now averaged in different ways: a perfect average has float datatype. In PACS the averages are integers, so a rounding already occurs in the default reduction as well. Additional rounding still deviates from these values in a barely noticeable way as compared with the original noise. have reconsidered to include a rounding option back into the software. That way, even the highest noise setting fits well into the telemetry budget. 2 bits of rounding are already sufficient, but this additional quantization noise amounts to only 1-4% total noise increase. The tricky thing about it is, that statistically correct rounding involves random number generation. As the rounding is applied to integers, the question was, what the best way of implementation would be. We decided to include the rounding process already in the averaging step. That way we saved precious CPU cycles and also ensured that the digitization noise of the nominal averaging process and the extra rounding does not cumulate sporadically. In Figure 6 the effect of additional rounding is made obvious. Compared with the savings on the datarate, the additional quantization noise is a small price to pay. 4.4 Performance and Further Improvements An analysis of the data entropy showed that the new lossless compression comes as close as 15% to the theoretical limit, which is satisfying given the tradeoff that had to be made in all the involved operations. However, a few desirables improvements are left in photometry CPU: decorrelation, fully adaptive model If more processing resources had been available, a fully adaptive model for the arithmetic compression would have saved another 5%. Even more could be saved in the decorrelation stage, since the differentiation used increases the noise by a factor of 2, i.e. half a bit. Therefore, Linear Prediction seems to be a better candidate, but it is a costly one and its performance on chopped data is not any better Compressed Sensing The reduction part of the data processing consists of averaging of the frames on board by groups of 4 to 8. Although this operation increases the signal-to-noise in the down-linked images, it introduces a significant degradation in both spatial resolution and signal quality of scanned observations, which severely impacts the science capabilities of the mission. Recent advances in mathematics and information theory have given rise to a technique termed Compressed Sensing (CS), 5 which offers the potential for restoring the full capacities of the PACS photometer while achieving the high compression factor required to downlink the data. The principle behind is to already combine the compression step during measurement. Compressed Sensing uses the mathematical concepts of sparsity and incoherence to accurately reconstruct data that have been undersampled by the measurement process. The sensing step in such a CS scheme consists of a projection onto incoherent measurement ensembles. For this purpose we have recently shown that the noiselet transform fulfills the specific requirements of the Herschel/PACS mission. 6 The sensing algorithm is the smallest component, yet it needs careful attention as it

9 Figure 7. The cosmetic aspect of the spectrometer arrays. On the left, a frame from the red array is shown and on the right is one from the blue spectrometer. The line on top and on the bottom are dummy pixels that could be used for calibration and therefore have to be included in the telemetry. The ghost column is plotted to the right side of an array, although it has no physical pixels. It is always excluded from telemetry by the detector selection mechanism. has to be added as a separate compression mode to the flight software. The advantage of CS is that it is simple yet very efficient and it runs with minimal CPU load and memory requirements. Reconstruction of the original data can be achieved by solving a convex optimization problem. Although CS is a very young field of research, a number of algorithms are already available for this purpose. We have successfully developed an approach based on iterative thresholding, in which we can take additional properties of the detector and the observed target into account, without needing to modify the sensing algorithm. That way we would still be able to improve the reduction scheme even beyond Herschels expected lifetime. 5. IMAGING SPECTROSCOPY When PACS is in spectroscopy mode, the two Ge:Ga arrays (a stressed one for the red part and an unstressed one for the blue part) are generating data at even a higher rate than in photometry. The reason for this is in the way the photoconductors are operated. On the one hand the photoconductors suffer from quite a big reset noise, which is circumvented by using non-destructive readouts. The measure for the intensity is then the slope of the ramp. On the other hand these ramps are not perfectly linear, so they are read at a higher rate to get a finer description of the nonlinearity. Very similar to the bolometer, a single frame of a spectrometer array as shown in Figure 7 has a quite inhomogeneous appearance. Each column is an integrated module of 18 photoconductor pixels, stacked to form a 25x18 array. The signal from an individual pixel is substantially different to the photometer, as it is read non-destructively. Only after a number of samples are gathered, the capacitance is reset and the integration starts anew. Some features of photoconductors can be found in the plots of Figure 8. At the top is a sequence of raw data from a single pixel. Here, a reset occurs every 64 samples. The keyramp in the plots is made by averaging the ramps. It reveals a good linear shape. After subtracting the keyramp and taking some more measures accounting for sensitivity drift and other effects, the reset noise can be extracted to get an estimate of the data entropy for consideration in the reduction and compression scheme. 5.1 Capacitance, filter wheel settings Similar to the different gains in photometry, different integrating capacitances can be selected to cover the range from faint to bright sources. Again, by increasing the dynamic range, the noise is as well increased. For the spectrometer we have 3-6 bits of noise per readout. Different settings of the filter wheel and the grating order have no effect on the readout noise and therefore affect the data reduction only slightly.

10 5324 raw data 53 raw data noise histogram keyramp slope: offset: keyramp sigma: 4.36 entropy: sigma: 4.72 entropy: noise histogram slope: offset: Figure 8. The signal of the spectrometer, easily identifiable by the ramps. The so-called photoconductor hook is nicely represented in the left plot (pixel from red array) of the keyramp. The reset noise can be seen in the plot of the blue pixel (right side). In both examples the readout noise is between 4-5 bits. The ordinates of the plots are in ADU except for the histograms. 5.2 Ramp fitting vs. submeans For the lossy part, two options exist slope fitting and averaging. For reasons of nonlinearity, the effect of glitches and memory effects such as the so-called photoconductor hook, it is not desirable to reduce a ramp to only one slope value. As one ramp is normally 64 raw samples in length, i.e. 1/4 of a second, we make more than one slope by chopping the ramp into smaller sections that are fit with subramps. Already in early test phases it has become obvious that averages of such ramp sections (we call them submeans ) as short as the datarate allows with later fitting on ground leads to even better results. The shorter the submean, the better the resolution of the ramp. The shortest submean that is still compatible with the bandwidth is 8 samples/submean, or 1/64s. However, if a noisy detector setup is chosen, the datarate is too high and either longer submeans have to be chosen, or additional rounding of the submeans. In our studies we have shown that the final precision of a slope generated of 4 -sample submeans has the same precision as one made of 8 8-sample submeans with 2 bit additional rounding. Obviously, the latter option is preferable, as it has a better temporal resolution. 5.3 The devised reduction scheme After ramp fitting or submean averaging has been performed, the data need to be decorrelated again before they can be passed to the lossless compression. This is done in a different way as in photometry, as each processing buffer has 4 complete ramps per pixel. First of all, each ramp is differentiated. After that, the first ramp is subtracted as a keyramp from the remaining ones. So far the offset noise of the ramps and the nonlinearity in the keyramp remains. The keyramp is then differentiated another time. As the first sample of each pixel s keyramp is a -bit value, we subtract a mean first sample as the last step. In analogy to the photometry scheme, decorrelation is achieved with subtractions, in this case with an increase of the noise by slightly more than half a bit. 5.4 Performance Comparing the data entropy with the achieved datarates shows that the overall compression also comes as close as 15% to the theoretical limit, which is also satisfying in spectroscopy. That length has been determined during the various test phases. Shorter ramps are tainted by the hook, longer ramps are prone to glitches and to random walk effects The subtraction of the keyramp does not increase the noise by 2.

11 6. LOSSLESS COMPRESSION Due to the noisy nature of science data, compression algorithms that aim at recurring patterns are seldomly of any use. No matter how the data are arranged either spatially or in a temporal way there are hardly any repeating segments. For this very reason a whole class of compression algorithms can be excluded right away: dictionary-based compression methods such as the LZ77-family. The best way to deal with noise is by not trying to compress it but by efficiently encoding it to the entropy limit. That is why we chose a decorrelation/encoding approach. Both reduction schemes produce a data stream of 3-6 bits signed noise around with certain peaks that take the full bit range. This had to be considered in designing the backend encoding algorithm. We decided to implement arithmetic compression to have an encoding whose efficiency depends on the data model, but is relatively stable when the model does not fit well. Before passing the data to the entropy coder, we map the input values to positive (unsigned bit). The limited CPU resources excluded a fully adaptive (i.e. that the whole probability table is updated after every encoding symbol) model. On the other hand, static models are not flexible enough for the variety of effects that impact the probabilities. Another driver in the design of the model was that the full bit range had to be encoded, yet 9-95% of the encoding symbols are within the 8 bit region. The trick was to handle values -254 with a classical cumulative probability table and use the index 255 for values , indicating that they are put aside for separate compression with a variable block word length encoder. The semi-adaptivity of our model stems from the probability table, which we replace whenever 8192 symbols have been encoded. In terms of CPU power, the lossless compression uses up to 5% of what is available. This is of course the biggest consumer of available resources, yet this is well justified: the more efficient the lossless compression works, the less drastic reduction has to be made. 7. GROUND SOFTWARE AND MAINTENANCE The decompression framework is a part of the PACS Common Science System PCSS, which in turn is a part of the Herschel Common Science System HCSS. 7 This huge framework (exceeding 1 files) is written in Java TM, which is very beneficial for collaboration of the many dozen of developers spread all over the globe. For decompression backwards compatibility to old on-board software versions had to be considered. Although it is not anticipated to revert to any older version any more, the test and calibration measurements that were made with outdated on-board software still have to be readable. Aside from the delivered FM software version, which has been used during the ILT phase with no undesired effects, a few more improvements are considered for preparation, such as a dedicated Compressed Sensing mode for photometry, a better control mechanism for the additional raw channels and a mechanism for interleaving frames. During early operation of PACS we will learn if the expected glitch rates were estimated correctly and if any unexpected instrumental effect or even damage gives rise to a bigger software update. Thanks to the compact format of the executable it is possible to upload a new software version to the SPU within a few minutes. 8. LESSONS LEARNED AND FUTURE PROSPECTS The design and implementation of an on-board data reduction and compression system are processes that are tightly bound with the development of the individual subunit hardware and with the kind of observations to be performed. Naturally, we will further improve our algorithms and try to extend and apply them beyond PACS. There are a few key elements that we will include in any future design from the very beginning, such as relatively intense multitasking, which has proven very advantageous. Although it complicates finding certain types of software bugs, it also helps tracking down bugs, as several tasks such as the Housekeeping task usually stay alive and can be used for debugging when a software problem occurs. The biggest restriction for the SPU software was the CPU power. In our opinion, this is definitely more important than memory. Although there are various ways to gain speed through the use of memory, this compensation does not go very far in the end. Various test modes were also very helpful in identifying the source of a problem. In a warm electronics system as complex as PACS it is not always clear where the source of a problem is and how this can be tracked down.

12 Attaching checksums to different data sets that are subsequently repackaged by other subunits is a quick way of identifying subunits as the source of a problem. The experience we gathered and the on-board software we developed for PACS can be easily applied to similar projects, such as the Japan-led SPICA mission. Depending on the selection of the type of detector that will be used in the European SAFARI instrument proposed for SPICA, 8 a reduction/compression system similar to PACS will have to be implemented. For studying the necessity and possible ways of implementation we have joined the SAFARI consortium. 9. ACKNOWLEDGMENTS This work was supported by the Austrian Federal Ministry of Transport, Innovation and Technology within the project FIRST/PACS Phase I and the ASAP project of the FFG/ALR. REFERENCES [1] Poglitsch, A., Waelkens, C., Bauer, O. H., Cepa, J., Feuchtgruber, H., Henning, T., van Hoof, C., Kerschbaum, F., Lemke, D., Renotte, E., Rodriguez, L., Saraceno, P., and Vandenbussche, B., The photodetector array camera and spectrometer (PACS) for the Herschel Space Observatory, in [Space Telescopes and Instrumentation I: Optical, Infrared, and Millimeter. Edited by Mather, John C.; MacEwen, Howard A.; de Graauw, Mattheus W. M.], Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference 6265 (July 26). [2] Bischof, H., Belbachir, A. N., Hoenigmann, D., and Kerschbaum, F., Data reduction concept for FIRST/PACS, in [Proc. SPIE Vol. 413, p , UV, Optical, and IR Space Telescopes and Instruments], Breckinridge, J. B. and Jakobsen, P., eds., Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference 413, (July 2). [3] Ottensamer, R., Belbachir, A. N., Bischof, H., Feuchtgruber, H., Kerschbaum, F., Poglitsch, A., and Reimers, C., Herschel/PACS on-board reduction/compression software implementation, in [Optical, Infrared, and Millimeter Space Telescopes. Proceedings of the SPIE, Volume 5487, pp (24).], Mather, J. C., ed., Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference 5487, (Oct. 24). [4] Billot, N., Agnèse, P., Auguères, J.-L., Béguin, A., Bouère, A., Boulade, O., Cara, C., Cloué, C., Doumayrou, E., Duband, L., Horeau, B., le Mer, I., Lepennec, J., Martignac, J., Okumura, K., Revéret, V., Sauvage, M., Simoens, F., and Vigroux, L., The Herschel/PACS 256 bolometers imaging camera, in [Space Telescopes and Instrumentation I: Optical, Infrared, and Millimeter. Edited by Mather, John C.; MacEwen, Howard A.; de Graauw, Mattheus W. M.], Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference 6265 (July 26). [5] Candes, E., Romberg, J., and Tao, T., Robust Uncertainty Principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. on Information Theory 52, (26). [6] Bobin, J., Starck, J.-L., and Ottensamer, R., Compressed Sensing in Astronomy, IEEE Journal on Selected Topics in Signal Processing: Signal Processing for Astronomical and Space Research Applications (Oct. 28). [7] Wieprecht, E., Brumfit, J., Bakker, J., de Candussio, N., Guest, S., Huygen, R., de Jonge, A., Matthieu, J. J., Osterhage, S., Ott, S., Siddiqui, H., Vandenbussche, B., de Meester, W., Wetzstein, M., Wiezorrek, E., and Zaal, P., The HERSCHEL/PACS Common Software System as Data Reduction System, in [Astronomical Data Analysis Software and Systems (ADASS) XIII], Ochsenbein, F., Allen, M. G., and Egret, D., eds., Astronomical Society of the Pacific Conference Series 314, (July 24). [8] Swinyard, B. M. and Nakagawa, T., European contribution to the SPICA mission, in [Proc. SPIE Vol. 71, Optical, Infrared, and Millimeter], Oschmann, J. M., de Graauw, M. W. M., and MacEwen, H. A., eds., Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference 71 (28).