ICT-2010 Evaluation, October 2003

Transcription

1 ICT-2010 Evaluation, October 2003 Evaluation committee: Thomas Ericson, Linköping, Sweden Kaisa Sere, Åbo Academy University, Finland Seif Haridi, KTH and SICS, Sweden 1. Introduction The purpose of the ICT-2010 programme is to build up basic knowledge at Norwegian universities, colleges and research institutes within the areas of communication technology, distributed systems, and large information and software systems. The programme is currently funding 17 projects, 11 of which are considered in this evaluation. The evaluation is based upon the following material: i) Progress reports, ii) Application forms, iii) Oral presentations. The oral presentations occurred on Tuesday October 28 at the SAS Radisson Hotel at Gardermoen airport. The progress reports and the application forms were distributed in advance to the evaluation committee. 2. General remarks A general conclusion is that the 11 projects taken together provide an adequate coverage of the area in question. In general the quality is high and the technical and social relevance is undisputable. The balance is somewhat in the favor of information systems, while communication technology is represented in just two of the eleven projects. The character of the activities varies between the projects, but generally speaking practical matters are emphasized ahead of pure academic aspects. In what follows we give first a few remarks on the individual projects. After that we make a few comments, including a few suggestions for possible changes. None of our comments should be understood as a serious criticism. The purpose of our remarks is to provide an input to the steering committee for discussions.

2 3. Individual projects 3.1 INCO The project focuses on practical methods for incremental and component-based software development. The aim is to propose, refine, and validate improved technologies. The ambition is to reduce time and cost for development and to increase software quality. The problems studied are generally very relevant to software industry right now. However, as mentioned also in the project summary, the technologies under study are very immature. There exist today only very few principles that are industrially employed. We believe that it might be difficult to reach the project goals. The project is very industrially oriented, carrying out empirical studies in different companies. Even though the problems are important and a study of them certainly is interesting, it is unclear what parameters are used in the evaluations and whether possible results are at all verifiable and reproducible. In other words: it is unclear whether any general conclusions can be obtained. We did not observe any feedback from the results to the component technology. It would be interesting to study how results from the study could influence the design of components or the software process based on the components. 3.2 INSTANCE II The project can be seen as a continuation and a merge of two earlier projects, INSTANCE and MediaNode. The goal is to develop new solutions for the next generation of multimedia-distribution infrastructures that minimize response times, resource requirements, and costs, and that are scalable. The project moves from clientand-server architecture to a large scale architecture based on overlay networks. There are three students involved. One of them studies QoS in overlay networks. A second student addresses caching and replication in overlay networks. A third student studies the impact of memory scheduling and allocation in the operating system. The group is well connected in the European community through the network of excellence E-Next, and the proposed STREP project COALA. These facts assure that the Ph.D. students will work in a stimulating environment. All students involved were recruited just about half a year ago. As a consequence, until now there is little concrete progress in the project. However, there is good hope that this will quickly change as soon as the students start being more active in their research work. One related topic that we recommend the project group to look at is the recent work on structured peer-to-peer overlay networks based on distributed hash tables (DHT), and how these can used in media distribution.

3 3.3 WIWIC The project concerns radio design principles for low-cost terminals in wideband wireless communication. Almost all key electronic components are considered: antennas, oscillators, amplifiers. Impairments of various kinds are studied with an ambition to provide a better understanding of the phenomena and with the hope to be able to reduce the effects of undesirable imperfections. Power consumption is a key issue. The topics under study are at the core of modern wireless communication. Cutting edge research in this area is essential for any community main-taining ambitions to contribute to or to efficiently utilize this technology. At the same time the area in question is a very mature one, where intensive research and development activities have occurred during a very long time. Therefore only marginal improvements can be expected. The group under professor Bakken has a longstanding reputation as a leading group within this field. The current project seems to be carried out with skill and competence. Attention has to be brought to the fact that several of the main supervisors are relatively close to retirement, with the project manager, professor Bakken, retiring on November 1, As a consequence certain changes in the management structure of the project will occur in the near future. What changes this will lead to in terms of research content and direction remains to be seen. 3.4 MOWAHS The project studies work processes in virtual organizations. The goal is to understand and assess these processes as well as to provide an environment for executing and sharing work processes. The use of different electronic devises within virtual organizations is a main issue. The area of the project is new and therefore the project is difficult to judge on scientific grounds. One of the main contributions seems to be a definition of so called requirement indicators, giving hints concerning the complexity involved in implementing different mobile-work scenarios. The indicators might be a useful device. However, some of them seem to be very much depe ndent on the state-of-the -art in technology. Therefore as technology advances they might rapidly become obsolete. Nevertheless, the research agenda is good, although the perspective is a bit short and very much dependent on current technology.

4 3.5 BEATS The topic considered is transmission principles in wireless digital communication systems. In contrast to the WIWIC project implementation and electronic components are not considered, and in contrast to all other projects within the ICT-2010 programme protocols and software issues are also disregarded. However, basically all possible topics in between are considered. Specifically the project addresses coding and modulation, multiplexing, diversity, multi-access, and channel estimation. A key idea is to develop adaptive schemes so as to obtain a close-to-optimum performance regardless of possible variations in the channel. The area covered is quite broad and the project is the most generously funded one within the programme. The activities are subdivided into a number of sub-projects. Although high-quality research is conducted within each one of these the interaction between them seems to be rather weak. We believe that a more natural way to support these activities would be by individual funding of each activity as a separate project. In that way an evaluation of the activities would also be easier. Generally speaking, though, the project seems to be running well. The research activities occur within a well established context and are performed from a solid scientific platform. A large number of graduate students are involved. The progress seems to be satisfactory in all cases. International contacts are frequent and publication rate is satisfactory. 3.6 UMA The UMA project focuses on the access of multimedia content through future generation mobile and wireless communication devices and systems. Moreover, QoS issues are addressed when aiming at ensuring e.g. reliability even in mobile environments. As such the topic of the project is important and relevant, although very much dependent on the technological advancements. The project goals as described in the project report are very vague, as a matter of fact, there are no real goals w.r.t. research tasks given so it is difficult to judge if the project has been successful. Moreover, it is unclear, how the group plans to attack the issues raised. No measurable deliverables are given, only issues to be addressed. The research group seems to be active in standardisation work within the field and many of the reported publications are part of this. A test bed for delivery of streaming media has been developed partly based on the standardisation work. For the QoS issue, a new traffic mechanism has been proposed and simulated.

5 3.7 Artic Beans The project aims at the design, implementation, and evaluation of reflective component technology. The ambition is that all components should be dynamically configurable and allow effective monitoring. The project considers issues related to the associated architectures and models for security and transactions in this framework. The ambition is to provide a component-architecture for a reflective transactional system, in which different transaction services can be deployed and modified. An additional problem under study is the construction of a transaction processing monitor (TP monitor) based on reflection techniques. A third direction is the design of a language, Obol, for the programming of security protocols. The group cooperates closely with other research groups, especially one in Lancaster University. The cooperation concerns development of reflective middleware with support for QoS, and mobility. The project seems to be running well. 3.8 MoSIS-IKT The goals are to develop modularization techniques for large software systems and to develop a conceptual and formal framework for the composition and interaction of modules. The aim of such techniques is to reduce the cost and effort of developing and maintaining software. The problems investigated are interesting and important. The design of large software systems based on modules is a difficult task. Moreover, the interactions between the modules provide interesting design challenges. The investigation is subdivided in two sub-projects, one for each one of the two Ph.D. students employed. We are concerned about the apparent lack of cooperation and cohesion between the two parts of this project. It is not clear how they will support each other even in the long run. 3.9 ADIS The project is concerned mostly with multi-agent technology and applications. The activities are divided into sub-projects, which are conducted by the individual students. One subproject investigates web-service composition using linear logic and its associated theorem-prover as the underlying technology. Another activity concerns the utilization of agents for the dissemination of medical knowledge by means of adaptive distributed information service. A third activity is directed towards the use of agent modeling for defining so called virtual enterprise. The project combines basic research with application scenarios. We think this is a suitable project within the ITK-2010 programme.

6 3.10 WebSys The goal is to develop guidelines for industrial development of Web-based systems. The project aims at meeting this general goal via two sub-goals. One of them focuses on software processes and how to establish a trade-off between TTM and reliability. The other concerns the design of methods for constructing Web-based systems. The trade-off between TTM and reliability is very import and quite common a problem. The general form is studied within many fields, especially within economics. From the material given to us it is unclear what web-specific aspects arise in this area. Attention should be brought to that question. Web-based system development certainly needs better methods than the ad-hoc methods currently in use. The project aims at developing certain patterns which would take reliability and TTM aspects into account. This is an interesting goal. However, we are somewhat skeptical regarding the means for meeting this goal. The project is very industrially oriented. The main activity seems to be empirical studies of Web-based development in some Norwegian companies. We are not convinced that this approach is the best possible way to attack this problem ERCIM The project concerns Norwegian membership in the European Research Consortium for Informatics and Mathematics. In contrast to all other projects within the programme this one is not directed towards any specific research problem. Instead the purpose is to provide general support for interaction with other participating institutions and to offer fellowships within the consortium. The goal is to enhance cooperation between Norwegian institutes and corresponding institutes in other countries. The project proceeds according to plan and seems to be quite successful so far. An impressive number of post-docs are already involved. The degree of activity seems to be high.

7 4. Appropriateness Among the specific questions to be addressed in the evaluation is whether the projects are appropriate in order to fulfill the programme goals. A clear-cut answer cannot easily be given. The area is quite wide, spanning from fundamental hardware ( WIWIC), over transmission principles (BEATS), protocols, and services (MOWAS, UMA, ADIS) to design principles for large soft-ware systems (INCO, Artic Beans, WebSys). There is no way such a wide area can be fully covered in a few projects and within a relatively restricted budget. One word of criticism that can be directed towards the programme as such is that it is a bit too wide. One consequence is that it becomes difficult to achieve a reasonable coverage. Another consequence is that the programme will house projects of very different characters, having different ambitions and representing different scientific paradigms. This phenomenon is very apparent in the ICT-2010 programme. A third consequence is that a coherent evaluation of the various projects becomes very difficult. With these remarks made and taking the difficulties mentioned into account we do feel that the projects taken together provide a reasonable coverage of the area defined in the programme. Communication technology is slightly underrepresented. At least this is true in terms of the number of projects. In terms of economic resources the unbalance is less striking, as both of the telecommunication projects belong to the best financed ones of the eleven projects within the programme. One important area which we believe would adequately belong to the programme is Data security. Problems related to this area are certainly included in severa l of the projects, but only as side interests. We feel that security is such an important aspect of internet and information technology that a separate project specifically directed towards this topic would be very well motivated. Another topic, also partially included in several of the projects, but maybe important enough to deserve a specific project of its own, is Large scale distributed systems. Problems we have in mind include distributed algorithms, routing, congestion, robustness, analysis of large scale systems, peer-to-peer architectures, and scalable GRID-services.

8 5. Quality of projects Quality is a complicated thing that can be measured and judged in many different ways. In our evaluation we have tried to evaluate each individual project with respect to its own ambitions and based upon its own specific circumstances. With such a view in mind our conclusion is that the general quality is high and that the programme as such proceeds in a satisfactory way. Nevertheless a few minor remarks can be made. In most of the projects industrial relevance is given priority before academic aspects. This seems to be well in accordance with the ambition that ICT-2010 should provide a platform for development of business and society, but less so with respect to the other ambition of supporting basic research. The desired balance between industrial relevance and academic content is of course a question for the steering committee to decide upon. It is a striking fact, though, that several of the projects are conducted more in line with industrial values than in accordance with basic academic principles. We also feel that in many cases there is a lack of clear and crisp measurable parameters by which the success or failure of the projects can be measured. Another observation is that in several cases publication of reports and production of PhD:s seem to be regarded as goals per se. We very much disagree with such a view. Publication and education are very important aspects of academic activities, but they are not in themselves adequate motivations for a project. The goals of a project should be operative and defined in terms of desired scientific, technical, or social achievements. When interesting results are obtained it is an important duty to publish. The process naturally leads to dissertations. But the order of matter is important: a thesis or a paper should be a report on the results achieved, but neither should be the reason for the project. 6. Steering committee The evaluation group is hardly in a position to evaluate the steering committee. The people involved seem to be highly qualified. They represent a reasonable mix of industrial and academic experience. Again communication technology is slightly underrepresented. Apart from this minor defect the composition seems to be well balanced.

9 7. Possible improvements We cannot see any urgent need for major changes in the programme. Our remarks should be understood as suggestions for discussions. All of them concern possible minor modifications. 8. Conclusion Our general conclusion is that the programme is running well. We believe that the purpose of generating useful basic knowledge within the area defined by the programme is well met by the projects involved. November 2003 Thomas Ericson Kaisa Sere Seif Haridi