TOWARDS REAL-LIFE SAFETY ASSESSMENT OF ADAS AND ADS. Olaf Op den Camp TNO Integrated Vehicle Safety October 2016

Transcription

1 TOWARDS REAL-LIFE SAFETY ASSESSMENT OF ADAS AND ADS Olaf Op den Camp TNO Integrated Vehicle Safety October 2016

2 AGENDA Introduction Process to come to a test protocol for Cyclist-AEB (Autonomous Emergency Braking) systems Challenges in the assessment of Cyclist-AEB systems (example for ADAS) Challenges towards the assessment of ADS Methodology development to collect scenarios from driving on public road ' 2 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

3 SOLUTIONS TO PROTECT CYCLISTS* Collision avoidance / mitigation: Forward collision warning Autonomous Emergency Braking * IN CAR TO CYCLIST ACCIDENTS Time-to-collision (TTC) PoNR Crash ' 3 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

4 DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS) OBJECTIVE Prepare the introduction of a protocol for consumer tests of Cyclist-AEB systems on board passenger cars Propose a test setup (incl. hardware) and test protocol for Cyclist-AEB systems based on technical/scientific considerations Base the tests on analysis of most relevant cyclist accident scenarios in EU countries TIMING Start : 2014 Q1 Finished : 2016 Q2 ' 4 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

5 DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS) APPROACH Study databases for 6 European countries; Select severe car-to-cyclists accidents --> fatalities, seriously injured; Provide overview of distinguished accident scenarios; Determine the distribution of scenarios in the different countries; Prioritize scenarios & indicate coverage of fatalities and seriously injured. ' 5 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

6 DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS) FINAL CATS PROTOCOL CVBNU CVBNO CVBL ' 6 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

7 DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS) CYCLIST TARGET: SOFT BICYCLIST DUMMY ON SOFT BIKE DUMMY (version 4activeBS v5) Changeable handle bar for Dutch and European bike Materials and properties of bicyclist same as Euro NCAP Pedestrian Target Adjustable torso-angle White reflector in the front mounted on the frame Rotational joint of jip connected to bike frame Polymer frame with metal layer for radar properties Rotational joint at the knee point Plastic mud guard Rear red reflector mounted on the luggage rack Real rubber tire with reflecting ring Rotating wheels due to contact to the ground Rim with reflecting material Preparation for simulating pedaling behaviour ' 7 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

8 CYCLIST-AEB EFFECTIVENESS ASSESSMENT AEB HARDWARE & SOFTWARE IMPLEMENTED IN (TEST) VEHICLE: In physical tests on a test track, e.g. using the scenarios from the test matrix Determine performance in speed reduction by AEB for varying speed of the test vehicle AEB SOFTWARE DEVELOPMENT: Use virtual simulation tools for prospective effectiveness assessment real world/ scenarios view on surrounding sensor system sensor data vehicle response (v, a, steering angle) driver input to car car dynamics driver HMI ToC Driver overrule ADF input to car automated driving function world model vehicle state ' 8 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

9 ΔV AEB [km/h] ΔV AEB [km/h] CYCLIST-AEB EFFECTIVENESS ASSESSMENT FoV: 2 x 24 CVBNU CVBNO CVBF Current state-of-the-art FoV: 2 x 45 CVBNU CVBNO CVBF Beyond 2018 ' 9 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

10 CYCLIST-AEB EFFECTIVENESS ASSESSMENT EFFECTIVENESS: BALANCING PERFORMANCE AND ROBUSTNESS reliability TP TP Reduce Higher Probability resolution FP by of lowering correct by function smarter sensor sensitivity sensor response : reduce FN --> Performance : reduce FP --> Customer acceptance 100% FN TP: system response when needed FN: no system response when it is needed TN: no system response when not needed FP: system response when not needed FP FP robustness TN 0% 100% Not only aim at reducing FN to increase system performance in critical situations Also aim at reducing FP to increase customer acceptance and performance ' 10 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

11 ΔV AEB [km/h] CYCLIST-AEB EFFECTIVENESS ASSESSMENT POSSIBILITIES FOR CYCLIST TO AVOID COLLISION: X Test executed by Subaru to discuss false positives in the CATS-project Prototype vehicle, not massproduction True Positive Possible False Positive (braking bike up to 7 m/s 2 ) Simulated decrease in performance based on calculation of PoNR CVBNU ' 11 LINDHOLMEN SOFTWARE DAYS Possible False Positive (turning bike) 28 OCTOBER 2016

12 ΔV AEB [km/h] ΔV AEB [km/h] CYCLIST-AEB EFFECTIVENESS ASSESSMENT FoV: 2 x 24 CVBNU CVBNO CVBF Current state-of-the-art FoV: 2 x 45 CVBNU CVBNO CVBF Beyond 2018 ' 12 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

13 ADS EFFECTIVENESS ASSESSMENT FOR ADAS, TRADITIONALLY STRONG FOCUS ON ACCIDENT AND CRASH DATA To set up test protocol for rating To determine the performance of systems INCREASED NEED FOR ADDRESSING NORMAL EVERY DAY DRIVING To determine the performance on the road, not only in seldom critical/crash situations Holds for individual ADAS, but increasingly important to deal with the complexity of multiple simultaneously acting ADAS, especially in view of the transition towards AD. ' 13 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

14 CHALLENGES FOR AUTOMATED AND COOPERATIVE DRIVING FUNCTIONS Integrated and interrelated functions can no longer be tested independently. Tests are preferably conducted on public road for real life safety assessment, however safety and legal constraints apply. The number of relevant scenarios increases strongly: Multiple targets with individual manoeuvres Infrastructure related parameters: type of road, intersection, traffic rules Disturbances such as weather/lighting conditions, road works turning left bicycle car going straight turning right Most relevant conflict scenarios 6 Relevant interaction scenarios >36 Many cyclists are present in the scene Scenarios are often not reproducible, ground truth data of targets are unknown or inaccurate ' 14 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

15 SCENARIO BASED APPROACH INTERPRET TRAFFIC AS AN ENDLESS SEQUENCE OF DIFFERENT SCENARIOS DETERMINE REAL-LIFE PERFORMANCE WITHOUT DRIVING MULTIPLE MILLION KM: Data collection under real-life conditions on the road Data analysis: scenario identification & classification (big data solutions required) Use library of scenarios in virtual testing for development and implementation SCENARIO: A typical maneuver on the road with the complete set of relevant conditions and trajectories of other traffic participants that have an interaction with the host vehicle over a relevant time period (order of seconds) A ride on the road can in this way be described by a continuous sequence of scenarios; scenarios might overlap in time. ' 15 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

16 HOW TO COLLECT RELEVANT SCENARIOS? ' 16 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

17 REAL-LIFE SAFETY ASSESSMENT METHODOLOGY Real life vehicle data Big Data No ADAS response Accident databases UDRIVE Euro NCAP Vehicle data Sensor data Vehicle data Sensor data Vehicle data Sensor data # RAW DATA STORAGE - vehicles - sensors, GPS - annotations Run identification and classification algorithms SCENARIO DATABASE incl. all characteristic parameter distributions EVALUATION OF ADS: Virtual testing (extensive set) Physical testing (limited # of km) Reference measurements Calibration based on ground truth Develop classification algorithms (deep learning) Develop classification algorithms (deep learning) BEHAVIOUR MODELS - bicyclists - pedestrians - vehicles DEVELOPMENT OF ADS: Integrated behaviour prediction models Real life bicycle data # RAW DATA STORAGE - bicycles - sensors - GPS Run identification and classification algorithms 17 ' LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

18 SCENARIO IDENTIFICATION & CLASSIFICATION (EXAMPLE) TYPICAL ACC SCENARIOS: DATA COLLECTION SETUP: VIEW FROM HOST: ' 18 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

19 SCENARIO IDENTIFICATION & CLASSIFICATION (EXAMPLE) TYPICAL ANALYSIS APPROACH: ' 19 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

20 SCENARIO IDENTIFICATION & CLASSIFICATION (EXAMPLE) TYPICAL USE OF ANALYSIS RESULTS: * Prepared by: Erwin de Gelder Jan-Pieter Paardekoper ' 20 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

21 OUTLOOK REAL-LIFE SAFETY ASSESSMENT OF ADAS AND ADS: Increased use of data collected on public road during every day driving conditions for system evaluation in addition to laboratory and test track tests. This requires: (Automatic) identification and classification of scenarios, and storage into a scenario database. Big data / machine learning solutions. Interface to scenario simulation tools. Harmonization of the evaluation approach is a prerequisite to make a safe transition from ADAS to Automated Driving: Type approval by the road- and vehicle-authorities Clarity of requirements for car manufacturers and automotive suppliers Make use of triple helix approach: industry, governments, R&D organizations ' 21 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

22 THANK YOU VERY MUCH FOR YOUR ATTENTION MORE INFORMATION: TNO Integrated Vehicle Safety W: Dr.Ir. Olaf Op den Camp E: Ir. Sytze Kalisvaart E: ' 22 LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016