Comparison of Range Image Segmentation Techniques for People Detection

Comparison of Range Image Segmentation Techniques for People Detection Supervised by: Diego Tipaldi, Social Robotics Lab Project 1 1 Short Description For people tracking from mobile platforms, detection and segmentation are two key components. The task in this project is to find, study, implement and compare different 2d range image segmentation techniques for people detection and evaluate their performance against ground truth. The students should perform a literature survey, collect and label some data, implement the baseline technique provided and at least a new technique, and finally perform experiments to compare them. 2 Input The students will get labelled log-files as well as the capability of recording and labelling new ones, a C++ framework to read in log-files, a GUI to visualize laser scans, and a performance metrics for the experimental comparison. 3 Output The students will provide the implementation of various segmentation techniques (at least two), and a comparative analysis of their performance. The students will also collect new data for testing purposes and label them. The evaluation of the project will also depend on the number of techniques compared and on their complexity. 4 Type of Project 60% theory, 40% practice. It is highly recommended to use C++/Linux. 1

5 Techniques The students shuold implement the following detection and segmentation method as a baseline technique: Jump distance segmentation and shape based detection. The following segmentation and or detection alternatives should be also implemented (at least one segmentation). Single-linkage clustering (segmentation) Curvature-based segmentation (segmentation) Split and merge (segmentation) Segment-based AdaBoost [1] (detection) Consistency-based detection (detection) Sliding AdaBoost (segmentation + detection) CRF-Clustering [3] (segmentation + detection) The students are also free to choose any other technique they found during the literature review. References [1] K. O. Arras, O. Martinez Mozos, W. Burgard, Using Boosted Features for the Detection of People in 2D Range Data, in Proc. of the IEEE Int. Conf. on Robotics and Automation, Rome, Italy, 2008. [2] C. Premebida and U. Nunes, Segmentation and geometric primitives extraction from 2d laser range data for mobile robot applications, in Robotica 2005, Coimbra, Portugal, April 2005. [3] Gian Diego Tipaldi and Fabio Ramos, Motion Clustering and Estimation with Conditional Random Fields, in Proc. of the IEEE Int. Conf. on Intelligent Robots and Systems, St. Louis, USA, 2009. 2

Learning Visual Target Appearance for Laser-Based People Tracking Supervised by: Matthias Luber, Social Robotics Lab Project 2 1 Short Description Laser data contain little or no information on target appearance that can help to disambiguate target identities after lengthy occlusion events. Currently, our people tracking system is purely laser-based and we want to explore the contribution of vision as an interesting sensor modality for a more informed data association. The task in this project is to implement or reuse a people tracker, calibrate the multisensor system, develop the theory and implementation of a visual target appearance learner and study the improvement over a laser-only tracker. 2 Input The students will get several log-files with laser and image data, a tool to calibrate a system combining a laser range finder and a camera as well as a tools to record and label new data. Furthermore, a C++ framework to read in log-files with laser and vision information, a laser-based MHT people tracker, and a GUI to visualize laser scans, images and the tracking results is provided. To test the tracking results, an error metric is available. 3 Output Definition and implementation of an appearance model for people found in camera images. A theory and implementation to jointly estimate a likelihood from motion and appearance to be used for guided data association. Further, a set of experiments that quantify how tracking performance is improved over the laser-only tracker and show the limitations of the presented approach. 4 Type of Project 30% theory, 70% practice. It is highly recommended to use the provided framework (C++/Linux). 1

5 Techniques The following techniques can be used to describe the appearance of a person. Color histogram of bounding boxes around a person. Contours or silhouette of people A collection of interest points (e.g. SIFT) Image patches Other techniques can be found and implemented. References [1] D. Schulz A Probabilistic Exemplar Approach to Combine Laser and Vision for Person Tracking., Robotics: Science and Systems, 2006 [2] W. Lu and Y. Tan, A color histogram based people tracking system., IEEE International Symposium on Circuits and Systems, 2001 [3] N. A. Setiawan, S. Hong and C. Lee, Multiple People Labeling and Tracking Using Stereo for Human Computer Interaction., Human-Computer Interaction, 2007 [4] E. Seemann, B. Leibe, K. Mikolajczyk, and B. Schiele, An Evaluation of Local Shape-Based Features for Pedestrian Detection, in British Machine Vision Conference (BMVC 05), Oxford, UK, 2005. 2

Unsupervised Learning of Human Motion Primitives Supervised by: Kai Arras, Social Robotics Lab Project 3 1 Short Description People tracking is essential to any robot that interacts or cooperates with people. Once we are able to track people, we also want to know what they are actually doing. Towards this step, called human activity recognition and understanding, the robot must be able to learn and recognize human spatial motion primitives. The task in this project is to develop, implement and compare different methods to represent and cluster motion primitives based on trajectories obtained from a laserbased people tracker. Example primitives include going straight, left turn, right turn, stopping, U-turn, standing, movements on the spot etc. The experiments should result in stable and accurate recognition rates when compared to ground truth. 2 Input The students will get both, labelled and non-labeled log-files with (x y ẋ ẏ)-people trajectories. 3 Output A set of methods to represent (raw points, features, histograms) and cluster (hierarchical, k-means) motion primitives, their theory, implementation, and comparison of the classification results with respect to ground truth. 4 Type of Project 50% theory, 50% practice. Language: open. Possibly use WEKA (Java) or Orange (Python). Matlab or Octave would also be fine (but very slow). Recommended: C++ 1

5 Techniques There are several approaches to represent motion primitives and cluster them during the learning phase. Given a local neighborhood of trajectory points, options for representations are: grids or polar grids (similar to shape context descriptors), the raw points themselves or a collection of simple features (e.g. the AdaBoost features used to people detection). For clustering, either hierarchical clustering can be employed or k-means with variable k or k given by the ground truth classification. According to each representation, a distance measure must be defined as well. For grids, the Jensen-Shannon-divergence, the χ 2 -distance are appropriate choices, for raw points, variants of the Hausdorff distance can be considered, and for features, the Euclidian distance is a possible choice. Note that the problem is not just a shape-recognition problem but also involves temporal information from the velocity estimates. This should enable to robot to distinguish a straight-line motion at constant speed from accelerated movements such as stopping or slowing down. References [1] A. Panangadan, M. J. Matarić and G. S. Sukhatme, Tracking and Modeling of Human Activity using Laser Rangefinders, to appear in International Journal of Social Robotics, 2010. [2] D. Kulic and Y. Nakamura, Incremental Learning and Memory Consolidation of Whole Body Motion Patterns, International Conference on Epigenetic Robotics, 2008. 2

3D Camera Calibration Supervised by: Barbara Frank Project 4 1 Short Description The PMD[vision]-O3 camera provides depth information in addition to grey scale images based on the time-of-flight measurement principle (see [3]). In this project, we will carry out a depth calibration that accounts for different systematic errors inherent to the system. 2 Input Depth and intensity images of a planar surface are acquired from different view points. To achieve this, the camera is mounted on a manipulation robot and moved to different positions. A depth image contains the radial distance to the closest obstacle for each pixel and the intensity image contains the grey value for the corresponding pixel. Additionally, the position of the robot manipulator (x,y,z,φ,θ,ψ) is given, where φ,θ,ψ represent the roll, pitch and jaw angles, respectively. Intrinsic camera parameters (focal length, principal point, lens distortion parameters) Distance and orientation of the calibration plane relative to the robot (provided by the robot s laser range finder) 3 Output A model for the error in the depth measurements and a correction for the systematic errors Extrinsic camera parameters (transformation from manipulator to camera) 4 Techniques Model fitting[1, 2], non-linear least squares estimation (see lecture). 1

5 Suggested Work-Plan Write a library for 3D transformations (rotations and translations). Write a simple GUI that loads and displays the data. Implement the reprojection of the depth images to 3d world points using the perspective camera model and the lens distortion model presented in the lecture on camera calibration. Identify a model for the depth error camera. Optimize the parameters of the model by least squares optimization. The error between the reprojected points and the calibration plane should be minimized. Evaluate the learned error model in terms of accuracy. References [1] Stefan Fuchs and Stefan May. Calibration and Registration for Precise Surface Reconstruction. In Proceedings of the DAGM Dyn3D Workshop, Heidelberg, Germany, September 2007. [2] Marvin Lindner and Andreas Kolb. Lateral and Depth Calibration of PMD- Distance Sensors. In Advances in Visual Computing, volume 2, pages 524 533. Springer, 2006. [3] Thorsten Ringbeck and Bianca Hagebeuker. A 3D Time of Flight Camera for Object Detection. Optical 3D-Measurement Techniques, ETH Zürich, 2007. http://www.ifm-electronic.com/obj/o1d Paper-PMD.pdf. 2

Localization with Improved Proposals Supervised by: Cyrill Stachniss Project 5 1 Short Description Knowing the pose (position and orientation) a mobile with respect to a map is important for several high-level tasks. The task of estimating the robot s pose is known as localization and a commonly used techniques is localization with particle filters, called Monte-Carlo localization (MCL). The goal of this project is to realize a MCL-based localization system, that uses informed proposal distributions to generate the next generation of the sample set. This can be seen as integrating the ideas used in FastSLAM 2.0 (see references) into MCL. As a results, we can expect to have a more focused particle cloud and thus a more precise pose estimate compared to standard MCL. 2 Input Given input a map of the environment, and laser and odometry data (Carmen log format) 3 Output The localization system is expected to return at each point in time, a weighted sample set showing the current posteriors of the samples. 4 Techniques MCL [1, 5], see Introduction to Mobile Robotics FastSLAM2 [3], Grid-based RBPF mapping [2] 1

5 Suggested Work Plan Implement a standard particle filter with simplistic motion and observation models and test it on toy data Visualize your data, e.g. by dumping images to disc (so that one can see the state of the filter also backwards in time) Develop a library for loading grid maps (as images or in the Carmen log file format) Develop a library for loading Carmen log files. Develop standard MCL with the beam end-point model (see MCL chapter in Introduction to Mobile Robotics or the probabilistic robotics book [4]) Integrate the ideas of FastSLAM 2.0 in your MCL implementation Evaluate the accuracy and robustness of the approach and compare it to standard MCL. References [1] F. Dellaert, D. Fox, W. Burgard, and S. Thrun. Monte carlo localization for mobile robots. In Proc. of the IEEE Int. Conf. on Robotics & Automation (ICRA), Leuven, Belgium, 1998. [2] G. Grisetti, C. Stachniss, and W. Burgard. Improved techniques for grid mapping with rao-blackwellized particle filters. IEEE Transactions on Robotics, 23(1):34 46, 2007. [3] M. Montemerlo, S. Thrun D. Koller, and B. Wegbreit. FastSLAM 2.0: An improved particle filtering algorithm for simultaneous localization and mapping that provably converges. In Proc. of the Int. Conf. on Artificial Intelligence (IJCAI), pages 1151 1156, Acapulco, Mexico, 2003. [4] S. Thrun, W. Burgard, and D. Fox. Probabilistic Robotics. MIT Press, 2005. [5] S. Thrun, D. Fox, W. Burgard, and Dellaert. F. Robust monte carlo localization for mobile robots. Artificial Intelligence, 128(1-2), 2001. 2

Trajectory Optimization Supervised by: Cyrill Stachniss and Christoph Sprunk Project 6 1 Short Description The goal for this project is to realize a spline-based system for trajectory optimization. Trajectory optimization consists of computing a feasible path for the robot based on a set of waypoints and a corresponding, feasible velocity profile. In this project, a static world is assumed and initial waypoints are given together with a virtual corridor that is assumed to be obstacle free. For these waypoints, a smooth path has to be computed together with a velocity profile that respects vehicle limitations. 2 Input Given input a sequence of waypoints W R 2, a virtual corridor the robot is not supposed to leave (e.g., a width of 2 m), a maximum translational velocity and acceleration, a maximum steering angle, and a maximum centripetal acceleration. The vehicle for which trajectories have to be optimized is considered to have a car-like motion model, i.e., an Ackerman drive. 3 Output The trajectory optimization system is expected to return a smooth trajectory in R 2 that is not to leave the specified corridor around the straight line between the input waypoints, and a velocity profile, such that there is a mapping from time to pose and velocity of the vehicle, i.e., for each point in time the vehicle s position, orientation, and velocity are determined. 1

4 Techniques 4.1 Splines as Trajectory Representation Spline is a widely used term, for the most part it refers to the concept of stitching curves from piecewise polynomial segments. As a starting point, several spline types are introduced in [2, 3, 5]. Note that in the spline literature there are multiple variants of formulation and conflicting notational conventions exist. Spline types or families differ in their properties with respect to continuity conditions at join points. Common in this domain are the concepts of parametric and geometric continuity. Controlling these properties at segment join points is directly related to controlling continuity in position, orientation, and curvature of the represented path at these points. Two examples for spline usage in path and trajectory planning are the work of Thun et al. [4] and Connors and Elkaim [1]. 4.2 Centripetal Acceleration The centripetal acceleration is computed directly from translational and rotational velocity at a point. When strictly following a path, rotational and translational velocity are connected via the curvature of the path. References [1] J. Connors and G. Elkaim. Manipulating B-Spline based paths for obstacle avoidance in autonomous ground vehicles. In Proc. of the 2007 National Technical Meeting of the Institute of Navigation, pages 1081 1088, San Diego, California, 2007. [2] James D. Foley, Andries van Dam, Steven K. Feiner, and John F. Hughes. Computer graphics: principles and practice (2nd ed.). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1990. [3] J.D. Foley, R.L. Phillips, J.F. Hughes, A. van Dam, and S.K. Feiner. Introduction to Computer Graphics. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1994. [4] S. Thrun, M. Montemerlo, and many others. Stanley: The robot that won the DARPA Grand Challenge. Journal of Field Robotics, 23(9):661 692, June 2006. [5] A.H. Watt and M. Watt. Advanced Animation and Rendering Techniques: Theory and Practice. Addison-Wesley Publishing Company, Inc., New York, 1992. 2

3D ICP-based Scan-Matching Supervised by: Giorgio Grisetti Project 7 1 Short Description Develop a 3D ICP Scan Matcher. ICP is an algorithm for aligning two sets of 3D points. It operates by selecting pairs of corresponding points between the two scans, starting from a current alignment. From these correspondences it computes a new alignment which takes into account the found correspondences. The procedure is iterated until convergence. 2 Input Two files, each containing a description of a set of 3D points. The points in the files are ordered from left to right and from top to bottom (according to the order they are provided by the scanner). The coordinates of the points in the point cloud are relative to the reference frame of the laser. 3 Output A relative transformation between the two point clouds which minimizes their overlap. 4 Techniques ICP [1], SVD Decomposition for finding orientations [2]. (see Robotics 1). As a possible extensions consider the generalized ICP algorithm [3]. 5 Suggested Work-Plan Develop a library to manipulate 3D points and 3D transformations. Implement primitives to read point clouds and to display them on gnu plot. Implement and test the 3D singular values decomposition. Test it on a scan in case of perfect correspondences. 1

Implement a data structure for efficient nearest neighbor search (e.g. KD- Tree). To this end you can also use external libraries. Implement the iterative ICP algorithm by alternating the search for correspondences and the solution of the translation. Once this framework is running you should test different correspondence selection strategies and compare them on different input data. Evaluate the accuracy and the robustness of the different strategies. References [1] P.J. Besl and N.D. McKay. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239 256, 1992. [2] B.K.P. Horn H.M. Hilden and S. Negahdaripour. Closed-form solution of absolute orientation using orthonormal matrices. J. Optical Soc. Am., 5:1127 1135, 1988. [3] A. Segal, D. Haehnel, and S. Thrun. Generalized-icp. In Proceedings of Robotics: Science and Systems, Seattle, USA, June 2009. 2

2D Correlative Scan-Matching Supervised by: Giorgio Grisetti Project 8 1 Short Description Develop a 2D scan matcher based on scan correlation on a 2D grid (CARMEN log format [2]). 2 Input A log file obtained robot containing a set of 2D scans together with the odometry position where they were acquired. 3 Output A log file containing the corrected robot positions and the 2D scans. 4 Techniques Occupancy grid mapping [1], correlative matching [3]. 5 Suggested Work-plan Write a library to read and write log files. Develop an grid map data structure with frequency counters. For testing reasons the data structure should be exportable as image. Write a set of routines to register scans in the map. Write a function that, given a scan, a robot position and a map returns the the quality of the match of that scan in that robot position. Implement a greedy search around the current location that returns the position having the highest observation likelihood, and save the corresponding log record on the output log file. 1

Test the algorithm for different resolutions of the grid map. Investigate the construction of a hierarchical approach that operates at multiple resolutions and is able to return multiple solutions. References [1] A. Elfes. Occupancy Grids: A Probabilistic Framework for Robot Perception and Navigation. PhD thesis, Department of Electrical and Computer Engineering, Carnegie Mellon University, 1989. [2] M. Montemerlo, N. Roy, S. Thrun, D. Hähnel, C. Stachniss, and J. Glover. CARMEN the carnegie mellon robot navigation toolkit. http://carmen.sourceforge.net, 2002. [3] Edwin Olson. Real-time correlative scan matching. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 4387 4393, Kobe, Japan, June 2009. 2

6D SLAM - Least Square Optimization on Pose-Graphs Supervised by: Giorgio Grisetti Project 9 1 Short Description The aim of this project is to build a 6D graph optimization algorithm based on the least square optimization technique. 2 Input A file describing a 3D pose graph. Every node in the graph is characterized by 6 components x,y,z,φ,θ,ψ where φ,θ,ψ represent respectively the roll, the pitch and the jaw of the node s position. An edge between the node i and the node j encodes an observation made from i to j. It is labeled with a mean (6D vector) and an observation (6D vector). For further clarifications on the file format, see www.openslam.org/toro in the section file format. 3 Output A file containing the poses of the vertexes after the optimization. 4 Techniques The whole project is based on least square minimization of the lecture material and the sample code. The transformation matrices are explained in every good book of robotics like [1]-Appendix E. More advanced techniques based on a reordering of the constraint matrix are described by Dellaert et al. [3]. 5 Suggested Work-Plan The students are invited to use as a starting point the 2D graph-based SLAM implementation explained in the lecture. We recommend to proceed in the following 1

way: Develop a library for 3D transformations. Develop primitives to load/save graph files Develop primitives to save a graph in a gnuplot friendly format, to visualize the output with the splot command. Write the error function of an edge in 3D, and compute the corresponding Jacobian. Due to its complexity we recommend to compute the Jacobian numerically. Write the least square solver using the functions described above. We recommend to test the correctness of the functions immediately after their development. Once the math has been validated by using a language at your choice, you should port the programs stuff in an efficient programming language (C/C++/Java) and use a sparse matrix package to solve the linear system. To this end we recommend CSparse by Tim Davis [2] (written in C). References [1] Howie Choset, Kevin M. Lynch, Seth Hutchinson, George A. Kantor, Wolfram Burgard, Lydia E. Kavraki, and Sebastian Thrun. Principles of Robot Motion: Theory, Algorithms, and Implementations. MIT Press, Cambridge, MA, June 2005. [2] Tim Davis. Csparse: a concise sparse matrix package. http://www.cise.ufl.edu/research/sparse/csparse. [3] F. Dellaert. Square Root SAM. In Proc. of Robotics: Science and Systems (RSS), pages 177 184, Cambridge, MA, USA, 2005. 2