EUROPEAN COMMISSION DG RTD SEVENTH FRAMEWORK PROGRAMME THEME 7 TRANSPORT - SST SST.2011.RTD-1 GA No

Transcription

1 EUROPEAN COMMISSION DG RTD SEVENTH FRAMEWORK PROGRAMME THEME 7 TRANSPORT - SST SST.2011.RTD-1 GA No ASPECSS Assessment methodologies for forward looking Integrated Pedestrian and further extension to Cyclists Safety Deliverable No. ASPECSS D2.1 Deliverable Title Test target specification document, including sensor response requirements Dissemination level PUBLIC Written By Paul Lemmen (ASPECSS, HP2, Humanetics) Seiniger (ASPECSS, HP2, BASt) Johan Stoll (vfss, HP2, AUDI) Thomas Schaller (ASPECSS, vfss, HP2, BMW) Eugen Schubert (ASPECSS, Bosch) 17/08/2012 Checked by Thomas Schalled (BMW) 17/09/2012 Approved by Monica Pla (IDIADA) 07/10/2012 Issue date 07/10/2012

2 Executive summary The AsPeCSS project is aiming at an improved protection of vulnerable road users, in particular pedestrians and cyclists by developing harmonized test and assessment procedures for forward looking integrated pedestrian safety systems. A suite of tests and assessment methods is under development as input to future consumer rating protocols and regulatory testing procedures. Implementation of such procedures / protocols will enforce widespread introduction of the technologies in the vehicle fleet, resulting in a significant reduction of fatalities and seriously injured among these road users. To realise these project goals a structure of five work packages has been defined. WP1 deals with the development of the overall Assessment Methodology including the socio-economic aspects, WP2 will develop test methods and tools for accident avoidance and mitigation and WP3 will study the testing related to injury assessment. Dissemination and Management are performed in WP4 and 5 respectively. This report presents specifications defined for the targets used in the tests for accident avoidance. The targets are objects that mimic humans for different sensing systems. To arrive at a technology-independent test procedure they should represent relevant physical properties for the most common sensors like radar, camera and PMD. Moreover they should be incorporated in the various test set-ups available resulting in specifications for mass, crash forgivingness, stiffness etc. Specifications defined in this report have been defined in close cooperation with input from the vfss and AEB project. Coordination of the input was arranged via the Harmonisation Platform on Test Equipment. 2/47

3 Contents 1 Introduction ASPECSS WP Harmonisation Platform 2: Test Equipment Approach and Contents Target specifications w.r.t. sensor technology Introduction Dimensions Posture Clothing and surface Camera sensors PMD and IR sensors Radar based technologies Other requirements Test set-up Test configuration Test track and asphalt Ambient conditions Workshop / Testing event evaluating target specifications Dummies tested Propulsion systems Test vehicles Test set-up and execution Discussion directly following testing Summary Optical Summary Radar Summary PMD Remarks from each test drivers Outcome of assessment sheets Introduction Composition of database Results - Dummy Results Propulsion System Additional comments filled on assessment sheets Conclusions from the workshop Reference data for radar sensors Measurements on pedestrians Daimler Toyota MOSARIM Project /47

4 5.2 Expert input Discussion and future activities Conclusions Risk Register Acknowledgment References APPENDIX A Boxplots of assessment sheets APPENDIX B Overview of Test Facilities APPENDIX C Overview of key sensor technologies /47

5 1 Introduction According to the World Health Organisation Global status report on road safety 2009, pedestrians account for more than 19% of road fatalities in the EU-27. It is well known that most accidents with pedestrians are caused by the driver being in alert or misinterpretation of the situation. For that reason advanced forward looking integration safety systems have a high potential to improve safety for this group of road users. These systems combine reduction of impact speed by driver warning and/or autonomous braking with protective devices upon impact. Previous EU research projects resulted in systems which are gradually entering the market. However, such new systems have to be widely deployed in the marketplace to realise their potential benefits. In relation to this the objective of the ASPECSS project is to contribute towards improving the protection of vulnerable road users, in particular pedestrians and cyclists by developing harmonized test and assessment procedures for forward looking integrated pedestrian safety systems. The outcome of the project will be a suite of tests and assessment methods as input to future regulatory procedures and consumer rating protocols. Implementation of such procedures / protocols will enforce widespread introduction of such systems in the vehicle fleet, resulting in a significant reduction of fatalities and seriously injured among these road users. 1.1 ASPECSS WP2 Work package 2 Test Procedures for Preventive Pedestrian Safety Systems will develop a technologyindependent test procedure suitable for assessment of preventive pedestrian safety systems. The procedure should be capable of assessing driver-in-the-loop and automatically intervening driver assistance systems utilized for pedestrian protection. The test result should reflect a representative, comprehensive effect of a total system in real life accident scenarios as determined in WP1. For this purpose operational test scenarios will be implemented in Task 2.1 (based accident surveys from ASPECSS WP1 as well as input from related projects in the field) including the development of test targets. Using this set-up an extensive test program will be executed for a number of vehicles. Test results will be used to develop and evaluate protocols. Test data will be returned to WP1 for further analysis of the methodology and for studies into repeatability and reproducibility of the methodology. Driver behavioral aspects and pedestrian reaction models as input to the testing will be studied in task 2.2. Empirical driver and pedestrian reaction studies will be performed for effectiveness assessment of warnings and interventions. Furthermore the percentage of warned drivers who can activate different thresholds of brake assistance will be determined. In a final task 2.3 the trade-off between effectiveness (time of warning) and unjustified system responses will be investigated. For this purpose technical limitations of existing sensors and actuators used for preventive pedestrian protection systems will be investigated. It will also quantify the uncertainties inherent in pedestrian reaction variability. The resulting models will be applied to develop a method to assess the trade-off between effectiveness and unjustified system responses. This report presents specifications for the targets as defined in Task 2.1. The targets are objects that mimic humans for different sensing systems. To arrive at a technology-independent test procedure they should represent relevant physical properties for the most common sensors like radar, camera and PMD. Moreover they should be incorporated in the various test set-ups available resulting in specifications for mass, crash forgivingness, stiffness etc. 1.2 Harmonisation Platform 2: Test Equipment Because of their potential in crash avoidance and injury mitigation Euro NCAP intends to include assessment of AEBS in future protocols. Procedures will be defined by the PNCAP group using information from a number of projects: 1. Advanced Forward-Looking Safety Systems (vfss) 5/47

6 Cooperation between OEMs, research and insurance groups world-wide developing test and assessment methods for forward facing safety systems related to accidents with pedestrians and cars. vfss also develops and applies methods on system effectiveness. 2. Advanced Emergency Brake systems (AEB) Cooperation between insurance organisations Thatcham and IIHS with support from research groups, a supplier and two OEMs. Aims and goals identical to vfss. 3. Assessment methods for Integrated Pedestrian Safety Systems (ASPECSS) EU FP7 Project consortium of OEM s, suppliers, test houses, research organisations and universities. Research on test methods considering driver behavioural aspects (warning), pre-crash performance evaluation, crash performance evaluation and system effectiveness. 4. Allgemeiner Deutscher Automobil-Club (ADAC) ADAC defined an evaluation method for AEBS considering the warning and autonomous braking actions to inform consumers on the system performance. The method was applied to various systems offered to the market and reported in the media. To streamline input from the various projects so-called Harmonisation Platforms (HP s) have been established. Goal is to exchange information on key subjects, thereby generating a clear overview of similarities and differences on the approaches and results. The projects will run independently but via the HP s they are well informed of mutual developments. Three HP s have been established: HP1 Test scenarios HP2 Test equipment HP3 Effectiveness analysis The specifications in this report have been generated through HP2 integrating information from ASPECSS, vfss and AEB. A set of specifications defined by vfss was used as basis for further discussions and extensions in ASPECSS and HP2. The result will be integrated in the HP2 documentation to Euro NCAP. 1.3 Approach and Contents As stated above the specifications listed in this report have been generated in consultation with other projects in the field namely vfss and AEB. As a start a first list was drafted by vfss. The specifications were reviewed and updated by ASPECSS and AEB partners and discussed during HP2 meetings. The final result is listed in chapter 2 of this report. As the target is integrated in a full test set-up a summary of a survey into available test configurations as reported in ASPECSS D2.2 is included in chapter 3. Some specific requirements to the set-up and the track as raised during the dummy specifications are included in this chapter as well. This includes for instance requirements related to the illumination for camera sensors. As a first evaluation of the specifications a testing event was organized in which a number of targets were approached by a large number of test vehicles with different sensing systems on board. The workshop was held 26 / 27 June 2012 at BASt in Bergisch Gladbach (Germany). Because of it s relevance a full report with results and conclusions is included in chapter 4 of this report. Basically the workshop confirmed the specifications as set in chapter 2 but leaves improvements for the RCS. Following the workshop detailed measurements of the RCS of volunteers and some test targets were arranged under the MOSARIM Project [ref]. This study was facilitated by ASPECSS partner BOSCH who is also in MOSARIM. An inquiry of test parameters was made among ASPECSS and HP2 partners and based on this a test program conducted in August Results and implications for the target specifications in terms of RCS are described in chapter 4. Apart from data of the MOSARIM project other available data are included in this chapter as well. Chapter 5 discusses results collected so far and provides an overview of future activities planned for the target developments. Summary / conclusions follow in chapter 6. 6/47

7 2 Target specifications w.r.t. sensor technology 2.1 Introduction The dummy is meant as a pedestrian surrogate for testing of driver warning and autonomous braking systems. As such dummy must be able to represent the human attributes in relation to the sensors used in the vehicle. The pedestrian target has to be detectable by the most relevant automotive sensors being RADAR, Video, Laser, PMD, IR-based systems. A short description of these technologies is included in appendix C. The required sensor-relevant dummy attributes were collected from car manufacturers, system suppliers and test houses involved in vfss, AEB and the ASPECSS project. 2.2 Dimensions It was agreed by all parties to have two targets representing adults and children respectively. The adult dummy should represent an average male while the smaller one should represent a child in the age of 6 years old. Figure 1 and Figure 2 depict both sizes from different view angles in a walking (adult) and running (child) posture [1]. When collecting details for the dimensions different projects appear to use different sources. ASPECSS partner Toyota provided data from the SAE Handbook while vfss partner AUDI used information from the RAMSIS Bodybuilder. Others like AEB used off the shelf targets mannequins used in department stores. All sources provide slightly different dimensions, which at itself should not be too much of an issue for the various sensing systems as long as variations are not too large and postures as defined in the next section are achievable. Having some flexibility allows for accessibility to a wide range of base models for the targets like for instance department store mannequins. Table 1 and Table 2 provide data from the SAE Handbook and the Ramsis bodybuilder as reference. In case desired by one of the stakeholder groups a more detailed specification can be extracted from the information in these tables at a later stage. Figure 1 - Adult viewed from left (impact side), right (non-struck), front and rear side. Figure 2 - Child viewed from left (impact side), right (non-struck), front and rear side. 7/47

8 No 1 ASPECSS D2.1 Test target specification document, including sensor response requirements 2 Content requirement toleranceremarks 3 height (head) width) 1800mm 231mm 153mm ±12mmSAE ±9mmSAE 489 Table 1 Anthropometry data for adult target from SAE Handbook (ASPECSS Partners) 4 (shoulder ~ground) center (153) 56 hip knee point height height 1473mm 527mm 952mm ±20mmSAE ±15mmSAE 7 shoulder 489mm ±25mmSAE feet foot center distance(width) center~foot 280mm ±10mmSAE handbook Stationary 1800 Table 2 Anthropometry data taken from RAMSIS Body builder (vfss) Dummy Segment Aspect Unit Proposal Position Tolerance Adult Body height mm 1800 ± 20 (inclusive shoes) H-Point height mm 923 ± 20 Heel to heel distance longitudinal mm 315 ± 10 lateral mm 147 ± 10 Step width longitudinal mm 600 ± 10 Shoulder width lateral mm 500 ± 10 Torso depth longitudinal mm 235 ± 10 distance front hand longitudinal mm 530 ± 10 back side Child 6YO Body height mm 1200 ± 20 (inclusive shoes) H-Point height mm 600 ± 20 Heel to heel distance longitudinal mm ± 10 lateral mm ± 10 both Torso angle deg 85 ± 1 Upper arm angle non-impact side deg 60 ± 1 impact side deg 110 ± 1 As general guidelines for the dimensions the following relative measures should be met for the front view: Shoulders should be 2-3 times the head width Waist should be 2-3 times the head width Out-seam of the legs is 4 times the head height In seam of the legs is 3 times the head height Arm length should be 3 times the head height For side view the target has to comply with the following requirement: Legs and arms must be spread apart as in a walking motion described in the next section: this parameter can be tuned by defining the exact posture of the arms 2.3 Posture Figure 3 shows the different walking phases of a human. 8/47

9 Figure 3 - Pedestrian walking phases. In discussions with ASPECSS and HP2 partners it was decided that the dummy shall represent walking phase MSt. This posture represents the dynamics (e.g. compared to posture ISw) and is used in the Euro NCAP procedure for the testing of deployable bonnets. The leg position also refers to SAE J2782 (Proposed Draft : Performance Specifications for a Midsize Male Pedestrian Research Dummy ). The dummy shall show an inclination of about 5 as shown in Figure 1 and Figure 2 which correlates with the posture of humans when walking. The face is looking in the walking direction. Articulation of arms and legs are regarded of secondary importance for visual systems (see also feedback provided by sensor suppliers in Chapter 4). However, recent RCS measurements done by Toyota show a change in reflective characteristics when going from standing to walking. See section For this reason articulations will be included in the list of requirements but it s relevance is to be further investigated and clarified. In case of stationary dummy the position of arms and legs should be according to the dimensions in Figure 1 and Figure Clothing and surface Camera sensors The dummy must be clothed with a long-sleeved shirt and trousers which have different colors. The clothing used should ensure a minimum contrast with the scene including asphalt and air for both color and black & white (grey scales) cameras. vfss specifies that the contrast ratio of the grey pixel values of the clothing to the background must be at minimum 50% in the given lighting. Preferred colors could be based on real life situations like blue jeans in combination with a light colored shirt. Clothing has to be loosely fitted and not form any planar wrinkles. For the testing event described in chapter AUDI provided clothing in line with the above. The dummy should wear shoes or have a marking representing shoes for the camera. During the first evaluation described in chapter it became clear that the dummy should have a face with white taint. This might be realized using a skin-colored sock to cover the face PMD and IR sensors For sensors like PMD there must be no reflecting parts on the dummy or it s clothing. The IR reflectivity (around 850 nm wavelength) of the clothes must be within the range of 40 to 60%. At the selection of the clothes it has to be ensured, that the IR reflectivity measured with the 45 probe must not differ for more than 20% from the reflectivity measured with the 90 probe (see section?.?). The IR reflectivity (around 850 nm wavelength) of the visible skin surface parts has to conform to original human skin within the range of 40 to 60%. As an option the dummy can be equipped with a wig to represent the head hairs. The IR reflectivity (around 850 nm wavelength) of the wig has to conform to original human hairs within the range of 20% (dark-haired) to 50% (fair hair). The skin temperature (at locations with clothing measured below the clothing) of the dummy immediately prior to each test run must be 32 C +/- 2 C. The thermal emission must not exceed 10 W/m²K. All visible parts of the dummy mounting and guidance system must have a temperature deviation of max +/- 5 C from the ambient temperature 9/47

10 2.4.3 Radar based technologies During the detection of a pedestrian by a radar sensor the Radar Cross Section (RCS) decreases with closing proximity. This is explained by the small opening angle of the sensor which results in the fact that the pedestrian at some point is not scanned over the full height / width any longer. Figure 4 gives an example of the RCS of a pedestrian measured with a 77 GHz radar. To capture the indicated phenomenon the dummy / target is to be equipped with multiple reflecting sources distributed over its body. For the radar back scattering cross section inputs are being received while writing this report. As a first stage minimum levels were specified by vfss being 0,05 m2 for 24 GHz and 0.5 m² for 77 GHz sensors. ASPECSS partner PSA proposed to have maximum levels specified as well being 0,2 m² for 24 GHz and 0.5 m² for 77 GHz. Both minimum / maximum values relate to a fully visible pedestrian. More detailed input requirements are being generated by various HP2 partners and by the MOSARIM project (see chapter 5). These will be added once available and verified. Figure 4 - Example RCS of pedestrian at 76GHz Radar reflectivity can be introduced in different ways. The target might have some inherent reflectivity from metal parts that are included (e.g. to provide overall stiffness and joints for body part positioning) or it might be created by adding triple reflectors or reflective foil. The applicability of either option depends on the sensor algorithms and their particular focus. Acceptable means are to be further investigated over the next period. 2.5 Other requirements Apart from the specifications listed in the previous sections specific requirements may apply. These mainly related to the integration of the dummy in the test set-up used. For instance for rescue dummies used in a portal rig set-up mass and strength requirements apply enabling pulling the dummy up or rotating it away. Due to the fact that many different set-ups exist and new ones are being proposed, the listing in this section is limited to some key items. Further information can be found in D2.2. For visual (and other) systems the stability of the dummy is an issue and therefore the reproducibility of test scenarios. Any swinging due to acceleration or deceleration may cause issues in the (reliability) of the detection. As described in chapter some algorithms check on the position of the center of gravity of a person as it needs to be within the base between the feet. In general stability issues relate more to test set-ups with crashable dummies (whether platform or test rig based). In case of positioning of the dummy on a moving platform (e.g. DSD platform or DRI platform) it should be realized that the height of the target is affected by the height of the platform. Moving platforms currently available on the market have a height of around 90 mm. The standing height of the dummy should be corrected for this. This is partially overcome by the use of an outrigger with smaller ground clearance (AEB solution). For portal rigs attention should be given to the attachment of the target from the top. Systems like camera s may detect the rod or ropes and algorithms may be mislead by these items classifying the target as a non 10/47

11 human object. Also the height of the dummy above the ground should be well controlled. Any gap between the dummy and the road surface may cause issues for sensors like camera. Various groups have defined a maximum value for the gap between dummy feet and road surface. vfss specified a value of maximum 15 mm whereas some AEB partners assumes and even smaller gap of maximum 7 mm. During the workshop held 26 / 27 June at BASt tests with the Continental rig clearly showed issues when the dummy was about 5 cm above the ground. When reducing (unfortunately no detailed measurement / value available) the issue was resolved, confirming the requirement stated above. A detailed value for the gap is yet to be determined by ASPECSS. For testing of IR sensors all visible parts of the dummy mounting and guidance system must have a temperature deviation of max +/- 5 C from the ambient temperature to differentiate between the set-up and the target. In case of a test set-up in which the target might be impacted (non-rescue set-up) any damage to the vehicle under test should be avoided as this may affect the system performance (e.g. due to offset in orientation of sensors). This means that the dummy should be crash forgiving. Requirements are difficult to define but in general it can be stated that parts should have a maximum weight of 15 kg and be covered in soft foam. Due to the complexity of crash phenomena exact masses and surface stiffness are test set-up dependent and need to be explored by the test houses themselves via extensive testing. 11/47

12 3 Test set-up 3.1 Test configuration A survey of all available test facilities for forward looking pedestrian safety systems is provided in ASPECSS D2.2 [2]. It includes an overview of facilities available at ASPECSS, vfss and AEB partners as well as commercially available tools provided by third parties. Appendix B of this report gives the summary table of existing test setups. For other items including discussions on strike-able versus rescue dummy the reader is referred to ASPECSS D Test track and asphalt Regarding the asphalt that exotic colors like red should be avoided. Also it should not have any reflections for sensors like PMD or radar (e.g. manhole cover, road reflector). It is preferred to have the road surface complying with the requirements of ISO 21994: Another road surface can be used if it is ensured that it meets the requirements of ISO 21994: The surface shall be horizontal and even, maximum deviation from the standard height adjusted to the cross slope should be less than 10 mm. The approach of the test vehicle should not contain sensor targets in a lateral expansion of 6.75 m from the left and right of the desired trajectory of the vehicle with the exception of dummy or system echo. The same applies to a range of 20 meters in longitudinal direction behind the target. The background of the test system (in direction of the desired trajectory of the vehicle) has to be designed either natural or plain-colored. It must not reflect or have pedestrian similar silhouettes and ground staff should stay out of field of view of the sensors aboard the vehicle under test. 3.3 Ambient conditions Ambient conditions are specified in ISO 21994: In addition to this the ambient temperature should not be below 5 C. The road surface must be dry. There must be no restrictions on visibility, for example by fog or smog. For camera systems the light intensity during testing should be monitored and a minim value should be defined for acceptance of the test (value yet to be specified). Also the illuminance and has to be homogeneous with a tolerance of ± 20% (vfss value). The required illuminance can be achieved either by sunlight or artificial lighting. Direct back light and horizontal illuminating side or rear lights, caused by low sun or artificial sources, and unnatural shadows (e.g. by buildings with the exception of the test system itself) should be avoided. Artificial light must not be unstable or pulsating (e.g. neon lamp) and must have a color temperature between 4500 K and 7000 K. 12/47

13 4 Workshop / Testing event evaluating target specifications On July an ASPECSS Task 2.1 workshop was held at BASt in Bergisch Gladbach (Germany). The workshop was organized in consultation with HP2 on the Test Equipment. Goal was to identify promising concepts for a pedestrian target dummy for Euro NCAP testing and evaluate the correctness of specifications as defined in chapter 3. Based on a previous approach applied by the vfss project for car targets a range of test objects was subjected to a range sensing technologies integrated in various test vehicles. As a first step a high level assessment of the dummies was done by test engineers. Based on the online sensor readings and system triggering they awarded marks from 1 (very good comparable to a human) to 4 (not comparable to a human) for the respective technologies. This assessment is to be justified later on by providing detailed analysis of collected data including sensor readings and processed data. At the time of writing this report the processing is still ongoing. Results are expected by September Dummies tested Table 3 below provides an overview of the dummies available for testing. The second column gives a brief description including some references to the specifications defined in the previous chapter. Table 3 Dummies tested No. Dummy Description Picture D.1 NHTSA Child: dummy representing 6YO child developed by NHTSA. No specific measures for RCS included. For visual and PMD purposes clothing as specified in chapter 3 was used (provided by vfss). D.2 vfss Child: dummy representing 6YO child developed by vfss according to specifications listed in chapter 3 including correct walking posture. RCS was fine tuned. Also for visual and PMD purposes clothing as specified in chapter 3 was used (provided by vfss). 13/47

14 D.3 vfss adult: dummy representing adult (50 th %-ile male) developed by vfss according to specifications listed in chapter 3 including correct walking posture. RCS was not considered yet. Also for visual and PMD purposes clothing as specified in chapter 3 was used (provided by vfss). D.4 NHTSA Adult: dummy representing adult (50 th %-ile male) developed by NHTSA. No specific measures for RCS included. For visual and PMD purposes clothing as specified in chapter 3 was used (provided by vfss). D.5 Autoliv Child: dummy representing 6YO child developed by ASPECSS partner Autoliv for in-house testing related to system development. During the testing the dummy was positioned as good as possible in the posture MSt defined in chapter 3. Some radar reflection is generated by the metal frame but it was not checked whether realistic or not. D.6 Thatcham: crashable dummy representing adults as used by the AEB consortium in their testing programs done so far. Off the shelf mannequin consisting of metal frame and exterior foam. The frame was slightly adjusted for crashability and positioning purposes. During the testing the dummy was positioned in the posture MSt as defined in chapter 3 and provided with correct clothing for visual and PMD sensors. Some radar reflection is generated by the metal frame but it was not checked whether realistic or not. 14/47

15 D.7 IDIADA: crashable dummy representing adults. Under development at ASPECSS partner IDIADA. Includes options for articulations of arms and legs. This is facilitated by a metal framework which gives RCS (not checked yet if realistic RCS or to be adjusted). During this first testing event it was not possible to have the articulating arms / legs operational. This will be studied during a next event planned for September D.8 TRL adult: dummy under development at ASPECSS partner TRL. Integrated with TRL platform for articulating leg representing walking motions. Crashable concept consisting of air filled tubes for the various body segments. Because of implications with applying the cloths the shirt and jeans used on this dummy was different from the others. This might affect PMD and camera performance. No fine tuning of RCS properties. D.9, D.10 4a Dummies: developed by 4a engineering for usage in their test setups. One dummy is used in the rig portal provided by 4a while the other one is under development for the new over-ride able flat platform. MSt posture was replicated as good as possible for this dummy and provided with correct clothing for visual and PMD sensors. Some radar reflection generated by metallic tape but it was not checked whether realistic or not. 15/47

16 D.0 = D.12 Conti Adult: dummy representing 50 %-ile male developed by Continental for for in-house testing related to system development. During the testing the dummy was positioned in the posture MSt as defined in chapter 3 and provided with correct clothing for visual and PMD sensors. Some radar reflection is generated by the metal frame but it was not checked whether realistic or not. D.13 Conti Child: dummy representing 6YO child developed by Continental for system development purposes. During the testing the dummy was positioned in the posture MSt as defined in chapter 3 and provided with correct clothing for visual and PMD sensors. Some radar reflection is generated by the metal frame but it was not checked whether realistic or not. D.14 TRL Child: dummy under development at ASPECSS partner TRL. Integrated with TRL platform for articulating leg representing walking motions. Like the TRL adult dummy a crashable concept consisting of air filled tubes for the various body segments. Because of implications with applying the cloths the shirt and jeans used on this dummy was different from the others. This might affect PMD and camera performance. No fine tuning of RCS properties. D.99 No dummy, just plain propulsion system 4.2 Propulsion systems Table 4 below provides an overview of the propulsion systems available for testing. The second column gives a brief description of each system. For some systems more detailed information can be found in ASPECSS Deliverable D2.2. Since the goal of the workshop was to evaluate the dummies itself and not so much the test set-up only limited attention is given to the platforms here. Main reason to have them included is to check for possible influences on the dummy performance like RCS or elevation of the dummy above the ground. 16/47

17 A further evaluation of platforms will be done during a second workshop planned for October During that event the platforms and dummies will be checked on the capability to represent the test scenarios defined in ASPECSS WP1. Table 4 Propulsion systems used No. Platform Description Picture P.1 UFO Pedestrian system: commercially available system for AEB testing. UFO is a platform able to move different objects on a road. It is driven by electric engines and manually or dgps (under development) controlled. The test vehicle can be synchronously driven by the driving robot interface. The dgps system allows a precise loop driving of all vehicles and objects. The platform has a height of about 9 cm and is over-ride able by the test car. The concept is based on a larger version of the UFO developed for car targets. It is reduced in size for pedestrian testing (smaller size of targets) P.2 Continental rig: Portal rig used for in-house system development at Continental. Crashable dummy with rotating rod system under impact. Dummy motion synchronized with vehicle approach via laser system. P.3 4a engineering ultra flat platform: new development currently being made by 4a engineering. It consists of a 2 cm high platform that can carry a lightweight dummy. The platform is over-ride able. Propulsion via flat belt running over the road and driven from set-up placed on both sides of the track. Dummy motion in relation to vehicle approach triggered via laser system. The set-up is fully mobile and easily implemented on tracks. P.4 TRL Platform: under development at ASPECSS partner TRL. It is driven by electric engines and meant to become dgp- controlled. The test vehicle can be synchronously driven by the driving robot interface. The platform has a height of about 9 cm and is over-ride able by the test car. It is made in stealth form applying carbon to have minimal radar reflections. P.5 Solo dummy, no propulsion system P.6 UFO Large Platform: Original design of UFO capable of carrying larger targets representing full cars. See also Test vehicles Table 5 below gives an overview of the test vehicles and the sensing systems used. Table 5 Test vehicles 17/47

18 Vehicle Vehicle No. RADAR Mono Camera PMD Stereo Camera FIR Audi #1 1 x Bosch #1 2 x Audi #2 3 x MagnaElectronics via IKA 4 x Bosch #2 5 x Bosch #3 6 x Opel 7 x x Daimler 8 x Autoliv 9 x x x BMW #1 10 x BMW #2 11 x Volvo via Thatcham 12 PSA 13 x x Toyota 14 x x Continental 15 x Volvo 16 x x 4.4 Test set-up and execution Figure 5 gives an overview of the skid pad used for testing. Four stations were used to position the dummy or dummy - propulsion system combinations. These are indicated as slot s in Figure 5. Cars queued in the upper right corner, then approached slot 3 with the 4a platform, drove towards either the Conti Test-rig (slot 1) or the Solo Dummy (Slot 4) and finally towards the UFO or TRL platform at slot 2. After passing these stations the cars drove back outside of the skid pad towards the queue in the upper right corner. Blue and red lines indicated the paths travelled. At each station, the dummy and propulsion system ID was indicated to the drivers. For the majority of configurations two runs were possible. From this each car had the opportunity to measure the dummies stationary and moving (in case possible). The drivers were asked to approach the each slot at approximately 30 km/h and to stop at sufficient distance before each dummy to avoid collisions. No collision did occur during the workshop, although few impact tests were done at the end of the workshop to collect info on the crash forgivingness and durability (not reported here as this will be part of the second workshop). The second day saw some slight changes to the setup. In particularly the UFO platform was not operable due to rainfall. 18/47

19 Figure 5 - BASt Skid pad with test setup. Tests were conducted in a counter-clockwise manner. 4.5 Discussion directly following testing Immediately following the tests a first discussion round was held on-site to collect first results. Although largely subjective the response I expected to represent the feeling of the test drivers on the various dummies in a good manner. The test drivers and other audience were asked for their impression on the dummies and platforms w.r.t the different sensors. A summary for each sensor is given in the next sections followed by the complete overview of all remarks made in section Summary Optical In general, the NHTSA dummy was regarded to have problems with detection by video systems because of the posture. The vfss and other dummies that assume the MSt posture were regarded as correct ones and well detected. It was noted that the color of the face could be important. The Thatcham dummy was OK for this but others like the vfss one not. A skin-colored sock to cover the face might help. The feet of the dummy need to be close to the ground (< 5cm) as specified in chapter 3. This was an issue with the Continental rig during the first tests but adjusted properly. Mechanism holding the dummy could have an influence (e.g. wires holding the dummy which would be detected as a large hat or result in classification as another object by some camera systems). Contrast is very important for the visual systems. As such the test set-up for future events should be changed giving identical backgrounds to all slots. Weather conditions (e.g. pure sun on the first day) make the comparison between different dummies difficult because the position of the sun is important. 19/47

20 4.5.2 Summary Radar NHTSA and vfss dummies we marked to be inconsistent since they have reflective material only on the front and rear sides. This is to be updated with reflections from the sides. The platforms (both UFO, TRL) add radar reflectivity, but that added reflectivity is classified as not relevant. However, there was not too much radar reflectivity tuning on site. This needs to be carried out at a later stage Summary PMD The PMD sensor sees the UFO platform because of it s height. This is not the case for the very flat 4a rig which has a height of only 2 cm. Pylons used during the test have a negative influence on measurement with this sensor due to their reflectivity Remarks from each test drivers Daimler: NHTSA child and adult have problems with the posture. Dummy 14 (TRL child) has little contrast (however that dummy did not wear the unified clothing provided by Audi). Bosch: Saw all dummies. The continental rig was seen by the radar. MobileEye: Swinging back of the dummy is a problem. The center of gravity of a person needs to be within the base (between the feet). This is evaluated by the algorithm. Visibility of body is good. Color of the face is important, hair (e.g. at the IDIADA dummy) is nice. NHTSA and vfss dummies do not have a natural face color. The pose of the vfss dummy is a very good default pose. Toyota: The face color is quite important. A skin-colored sock is a good method to get a realistic face color. Continental portal is visible for radar. NHTSA + vfss dummies are inconsistent for the radar because the material is only on the front, not on the side. From internal measurement it has been found that there could be a significant influence of moving legs and arms for the radar. UFO and TRL platforms add reflectivity. It can be seen but it is not relevant. PMD: The PMD sensor also sees the UFO platform. It does not see the 4a rig. ADAC: A moving dummy might not represent all pedestrians, e.g. not those wearing a coat or a suitcase. Hence there is a preference from the ADAC side not to include articulations. DSD: Will check the radar reflection of their platform. Opel: Car sees all platforms. A body pose with leaning forward is good for detection (natural walking posture). The TRL child dummy for instance was leaning backwards. As a consequence it was never detected. Volvo: Clothing contrast is an issue. The Continental portal had better results because of the background on the test track. This should be improved during next test events, providing identical backgrounds to all slots. The 4a test rig has no problems while other platforms (UFO and TRL) could be too high. Audi: All dummies OK, except for the TRL dummy on the TRL platform because of the pose. Also Pylons used in marking the lanes and distances do influence the observations for the PMD sensor because of their reflectivity. Hence use should be avoided in next events. VW: Conti rig could be higher, height of dummy influences position measurement. Autoliv: Dummy 2 (vfss child) is not bad. A heated dummy could help in the night scenarios. Bosch Video: Weather conditions very different: first day, change of sun and clouds. Second day, mostly overcast, that is better for the detectability. Next time, when such a workshop takes place, all dummies should be facing in the same direction so that there are identical optical conditions. More runs should be conducted for statistical reasons. 20/47

21 IKA: Posture and distance of the feet to the ground are important. Continental portal is good when it is adjusted correctly. PSA: As stated by others the contrast is important. TRW: vfss dummies were good. Thatcham dummy also OK. Thatcham: Mechanism holding the dummy in the Continental rig is an issue. Moreover shadows of a portal can have an influence. BMW: Dummy fixation mechanism of the conti rig is detected by the sensor. No stable detection of the NHTSA CHILD, contrast to the background is an issue. 4.6 Outcome of assessment sheets Introduction During the testing drivers were requested to fill a spreadsheet with five criteria for each sensor: 1. Time for obstacle detection 2. Time for classification 3. Stability of confidence during approach 4. Robustness against angular deviations 5. Decision for system activation. See Figure 6. The spreadsheet allowed the rating for six different sensor systems: radar, camera mono, camera stereo, FIR, PMD as well as fusion between combinations of sensor. Ratings had to be given in the range from 1 (very well comparable to a human) via 2 (well comparable to a human), 3 (adequate with optimization) to 4 (not appropriate). Participants were asked to give only one rating per combination of dummy and propulsion system, even if they had the chance to perform multiple test runs. Every rating that was given by the participant s was collected in a large database, sorted by criterion, dummy, propulsion system, and type (moving or stationary). No weighting was applied between any of these criteria. From the input collected boxplots were generated to display the performance. A boxplot is a standardized statistical plot for a data set. The data set here consists of a certain subset of ratings. On each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles, the whiskers extend to the most extreme data points not considered outliers, and outliers are plotted individually. Since the data set consists only of the four ratings, the median values do not differ for the different configurations. Therefore, pentagrams mark the mean value. The mean value and the number of elements for a data set are printed below respectively above each individual boxplot. For this overview, the subsets of data are a) entire ratings given to a specific dummy for all sensors, all propulsion systems, moving and stationary ( all all ), and additionally subgrouped per sensor ( radar all, camera all etc.), or b) entire ratings given to a specific propulsion system, with all dummies tested on that propulsion system, moving and stationary, for all sensors, and as above additionally sub-grouped per sensor. These two types of plot will be discussed below. A full overview with data additionally grouped by moving and stationary targets is included in appendix I. 21/47

22 Figure 6 - Template of the assessment sheet Composition of database The composition of the database is shown in Figure 7. Note that vehicle 5 and 6 were identical and had the same drivers, therefore vehicle 5 delivered no own results. Figure 7 - Composition of database Results - Dummy The overall rating shown in Figure 8 suggests that the IDIADA, vfss adult and Continental adult dummies are the best-scoring with a mean value of 1.4, 1.5 and 1.7, respectively. See Figure 8, upper left graph. Still acceptable seems to be the Thatcham dummy with 1.9. It is noted that for the radar sensors these as well as other dummies score quite poor. Child dummies score worse than adult dummies, probably because they are smaller. The vfss child scores significantly worse than the corresponding adult (1.8 vs. 1.5). The second best child dummies in the overall rating are the Autoliv child and Conti child (1.8 and 1.9). PMD ratings are all perfect except for the Thatcham and 4a dummies (some issues, mean rating 1.3 in both cases) and the TRL adult (mean rating 3). FIR ratings are not comparable to human ( 4 ) for all dummies except for the Autoliv dummy which was heated and therefore got a 1. The radar reflectivity was not fine-tuned for except for the vfss adult. All dummies are within the range from 1.7 to 2.3 (no dummy is really bad) with the outlier being the TRL dummy. 22/47

23 Mono camera shows the same tendency as the overall rating (best dummies Conti adult then IDIADA and vfss adult, best child dummies are vfss child then Autoliv and Conti). When considering the stereo camera results, vfss adult has some more outliers than before, and the Thatcham dummy in this case is also quite good. All dummies score worse for the radar when moved and better when being stationary. Also in the camera ratings, IDIADA and Thatcham dummies are better when stationary than when moving. This might be due to the propulsion systems. Figure 8 - Overall ratings for the different dummies (moving + stationary, all propulsion systems) Results Propulsion System When looking at all ratings with respect to the individual propulsion systems (Figure 9), it can be seen that both UFOs and the TRL platform score worse than the Conti test rig and the 4a platform, however the differences here are not large. A better reflection of issues with certain propulsion systems is found in the comments (see chapter 4). Stereo camera ratings suggest that the Conti rig and 4a platform score slightly better than the UFO platforms. Mono camera ratings give an identical trend, however here the basic (large) UFO is slightly better. 23/47

24 Figure 9 - Overall ratings for the different propulsion systems (moving + stationary, all dummies) Additional comments filled on assessment sheets Apart from the ratings some additional remarks were provided on the assessment sheets. These were also provided during the discussion right after the test series (see section 4.5.4) but included here for completeness. The full set of comments (sorted per propulsion system as well as sorted per dummy) is included in annex I. The UFO-Platform (both pedestrian and basic) suffered from unsteady walking speed (probably because it was controlled manually with a remote control). A more important issue is the relatively high radar cross section of the platform itself which was indicated numerous times. Upon approaching the UFO at close range the RCS was reported to grow strongly. The TRL platform also suffered from RCS, but here also the colors of the dummy and the body pose, stationary standing, looking unnatural. Also the head was reported to be unnatural. On the Continental test rig, the swinging movement of almost all dummies when moved by the supporting beam caused detection problems. Also, since most dummies needed some improvised fixing to the rig, the body pose as well as the distance feet to ground was a bit unnatural. The NHTSA child dummy was reported to have a too low and (in some cases) unstable RCS. The NHTSA adult was reported to have a body pose which is not in line with the specified MSt step phase and therefore difficult to detect by camera systems. It also seemed to have too low RCS. The IDIADA dummy was recognized quite well and early in the approach, but for one vehicle the RCS at the end of an experiment (hence close distance) seemed to become too low. No explanation could be given for this. 24/47

25 The 4a dummy had too much RCS, even when set up on the RCS-neutral 4a platform. For visual appearance it is a bit too slim when looking from the sides and there was a remark on the missing torso. 4.7 Conclusions from the workshop Based on the results presented above the preliminary conclusion from the workshop is that the vfss (both adult and child), Continental adult and IDIADA dummies seem to be appropriate for further optimization. For the propulsion systems, the 4a platform seems to be very neutral while the UFO platform suffers from its large RCS value. The Continental test rig also seems to be a good choice if the swinging motion of the dummy can be reduced. Since the vfss and IDIADA dummies were developed using the specifications set in chapter 3 the above conclusion confirms the requirements as defined for camera and PMD sensors. RCS was not incorporated fully and needs further investigation. IR performance was not assessed hence no indication can be given for that sensor. 25/47