National Performance Evaluation Facility for LADARs

National Performance Evaluation Facility for LADARs Kamel S. Saidi (presenter) Geraldine S. Cheok William C. Stone The National Institute of Standards and Technology Construction Metrology and Automation Group SPAR 2004 Houston, TX May 19-20, 2004

Applications Courtesy of Cyra 3D Models / As-Builts World Trade Center Courtesy of J. Palmeteer, Boeing Manufacturing Inspection/Automation Courtesy of MetricVision Courtesy of J. Turner, Rapid Terrain Visualization ACTD Urban Planning Urban Search and Rescue Mapping

Applications (2) Courtesy of sbgmaps.com Volume/Area Determination Vegetation/Biomass Determination Location/Position for Autonomous Navigation Physical Dimensions Object/Target Recognition

Application: Automated Construction Testbed (1)

Application: Automated Construction Testbed (2)

Application: Real-time LADAR for Steel Frame Connection Acquisition and Docking (1) Frame Rate: 30 Hz* Range: 7.5 m (20 m)* Resolution: < 1 cm* Pixels: 124x160 FOV: ~15 h&v * Manufacturer s Specification

Application: Real-time LADAR for Steel Frame Connection Acquisition and Docking (2) Flash LADAR 25x64 pixel 10 Hz Beam Acquisition (simulated approach)

Application: Towards the Ultimate Construction Site Sensor The LADAR Holy Grail Illumination Source: Eyesafe Angular resolution: < 0.03º Field of View (FOV): 60º x 60º Frame rate: >10 Hz Range resolution: 1 mm @ < 15 m Size: Coffee cup 3 mm @ < 100 m Cost: < $1000 US Fast Angular Deflection experiment 650 khz 0-D 0 D LADAR MEMS beam deflection 1X(10)^6 Degrees/s Demonstrated 8/2003 Accuracy: 3 mm FOV: 48 deg.

The Need for a National Performance Evaluation Facility No standard methods or protocols to calibrate or evaluate LADARs NIST workshop on June 12-13, 2003 Participants from private and public sector (manufacturers and end users) from US and Canada Strong consensus on need for: Neutral performance metrics - evaluation facility Standardization Definitions of common terminology Calibration/Evaluation facility Benefit manufacturers and end-users Allow NIST to meet own internal needs

What is the intent of the facility? Is it for calibration, certification, or performance evaluation?

Calibration It is generally felt that a calibration is performed to determine the hardware characteristics of an instrument to enable setting or adjustment of instrument parameters to optimal levels.

Performance Evaluation Performance assessment / evaluation would be conducted to determine how well the instrument and the processing software would meet a user s specific requirements. This is normally a voluntary assessment - usually performed at the request of an end user.

Certification Certification has legal connotations and would involve testing of the instrument in accordance with a set of protocols and the results measured against a metric pass/fail. The testing would, in general, be conducted in a certified laboratory. Product certification is voluntary; however, lack of certification may be interpreted negatively rightly or wrongly.

LADAR Characteristics Wavelengths: 500 nm to 1500 nm Measurement technique: Time-of-flight Phase-based AM-CW FM-CW Triangulation Frame acquisition: Scanning Non-scanning ( flash, FPA etc.) Maximum range: < 10 m to > 1 km Accuracy: sub-mm to cm Beam spot size: mm to cm

Performance Evaluation: Hardware Determination of uncertainties (accuracy + repeatability/precision) Range Angular positioning (difficult) Reproducibility/sensitivity Resolution (smallest distance between 2 objects that is discernible) Beam Spread Function (BSF)/ divergence Handling of multiple returns Single measurement Mixed pixels / phantom points Multiple measurements

Multiple 1. Single measurement Returns Return Box Floor Time 2. Multiple measurements

Performance Evaluation: Software (1) Software applications Registration Bush Tree Vegetation identification Mobility go/no-go decisions Ground truth generation Surface generation

Performance Evaluation: Software (2) Software applications (cont.) Ground determination

Performance Evaluation: Software (3) Standard data sets allow: Comparison of algorithms Determination of limitation of algorithm Determination of robustness of algorithm to handle different data sets (e.g., noisy, well defined object / clean scene, cluttered scene)

NIST LADAR Performance Metrics Facility (1) Purpose of facility Calibration / performance evaluation Test bed for developing evaluation metrics and test protocols Prototype instrument design Small indoor facility Evaluate short range, very accurate instruments Size: 15 m x 15 m x 10 m Highly climate controlled Temperature Humidity Pressure Artifact based

NIST LADAR Performance Metrics Facility (2) Standard artifacts and targets Material temperature invariant Size / shape Reflectance function of wavelength Standard procedures for use Courtesy of W. Böehler, i3mainz, Inst. for Spatial Info. & Surveying Technology

NIST LADAR Performance Metrics Facility (3) Medium sized indoor facility Allows for longer range calibrations and evaluations Size: 60 m (L) x 10 m (W) x 10 m (H) Climate controlled Artifact based Rail system for positioning target

NIST LADAR Performance Metrics Facility (4) Outdoor facility Allows for Long range calibrations Evaluation in Field conditions Varying environment and seasons Encompass different types of terrain wooded semi open urban Permanent benchmarks Elevated facility Elevated facility Permanently & accurately located benchmark for sensor Test / Rail track

NIST LADAR Performance Metrics Facility (5) Software development to support facility Statistical analyses Uncertainty / standard deviation Correlation Registration Ground truth determination Object recognition / shape recovery Modeling analysis / uncertainty Physical dimensions Location of corners, planes, edges, etc. Volume, area calculation

Other Performance Evaluation Terrestrial Military Redstone Arsenal Facilities/Efforts Eglin Air Force Base National Research Council of Canada 3D-Matic, Univ. of Glasgow and Imaging Faraday Partnership, UK i3mainz, Institute for Spatial Information and Surveying Technology, Germany Gävle GIS Institute of Sweden Non-Terrestrial ASPRS Operational LIDAR Guidelines Sensor calibration / validation procedures Error types, sources, reporting Data processing NASA Stennis Space Center

Summary (1) Next steps for NIST Small indoor invariant artifact based facility Hardware/equipment Laser tracker Single axis interferometer Metrology LADAR (accuracy 50 micron at 20 m) Total station Spectrometer Linear translation stages Rotation stages Turntable Manufactured items artifacts dimension uncertainties on the order of ± 0.0254 mm (± 0.001 in) material which has low thermal expansion, e.g., Invar targets with optical quality surfaces (characterized) artifact and target mounts (for random scene generation) Courtesy of Leica Courtesy of Leica

Summary (2) Next steps (cont.) Develop set of definitions of common terminology e.g., accuracy, uncertainty, error, LADAR, LIDAR greatly help reduce or eliminate ambiguity in interpreting test results when comparing different instruments Analogous to artifacts, standard data sets for comparing algorithms determining limitations of algorithms determining robustness of algorithms to handle different types of data statistical evaluations of the algorithms possible if enough std. data sets available Software development statistical analyses uncertainty / standard deviation correlation registration ground truth determination object recognition / shape recovery modeling analysis / uncertainty physical dimensions location of corners, planes, edges, etc. volume, area

Why NIST? It s what we do best! NIST Mission: To develop and promote measurement, standards, and technology to enhance productivity, facilitate trade, and improve the quality of life. CMAG Mission: Develop standards, practices and performance metrics for advanced site metrology and automation technologies that will increase the competitiveness of the US construction industry.

Really, why NIST? Developed performance standard of the steel measuring tape Developed CMM uncertainty standard Developed Laser tracker standard

Questions? For more information: Contact: Geraldine Cheok (301) 975-6074 cheok@nist.gov JOIN US!