Human Behavior Analysis in Intelligent Retail Environments Andrea Ascani, Emanuele Frontoni, Adriano Mancini, Primo Zingaretti 1 D.I.I.G.A., Università Politecnica delle Marche, Ancona - Italy, {ascani, frontoni, mancini, zinga}@diiga.univpm.it Abstract. This paper aims to propose an innovative idea of an intelligent, multimedia and interactive shop system where human behaviours are analyzed for interactivity and statistical purposes. We discuss the need for new services into the shop, involving consumers more directly and instigating them to increase their satisfaction and, as a consequence, their purchases. To do this, technology is very important and allows making interactions between costumers and products and between customers and the environment of the shop a rich source of marketing analysis. In particular we focus on concepts of monitoring and interactivity, introducing several emerging technologies in the field of retail environments. The main novelty of the paper is the general architecture of the system together with the introduction of a series of intelligent interactive and monitoring systems, yet implemented and tested in a dataset recorded during the EuroShop 2008 trade fair. Results, even if at an early stage, are convincing and most of all the general architecture is affordable in this specific application. Keywords: intelligent retail environment, computer vision, multiple cameras, motion detection, tracking, visual surveillance. 1 Introduction The concept of shop is changing during years and today it means many different things: shop became not only the place where customers go to buy a specific product, but also the place where customers go to spend part of their time; competition between different producers is made also in the shop structure, meaning of communication, graphics, position of products and many other aspects. The aim of every retail environment is to attract customer attention: with technology and strategy, shop's owner has to attract customer attention on particular products, giving also the chance to interact physically with technology, making the experience absolutely complete and involving. The main goal of this new concept of retail environment is simple: make more business with respect to the old shop idea. First of all, in some cases technology works as a virtual shop assistant, giving information to the customer directly; in this way shop's owner save money with staff.
Second, and more important, is the fact that technology let to provide many statistic data that, opportunely elaborated, can give many information about customers, customizing the shop to their needs and adapting product expositions to their behaviours. Data extracted in this way are objective, while often this kind of decision is made looking to subjective data. The entire system presented in this paper is depicted in Fig.1 and shows the idea of intelligent shop system where a remote system management (retail is based on the concept of several shops over the world with the same format managed by a central communication and business intelligence centre) allow to manage the general vision system over the Internet and to grab important data from the vision system and statistical analysis. In particular there are three layer involved in the system: - the people tracking inside the shop to detect where people are passing by or are stopping in front of an expositor; By analyzing traffic patterns and counting patrons, retail stores can improve customer service, respond more effectively to rushes, and determine which displays and products are most effective in generating sales; - the people tracking for the analysis of people passing in front of the shop window, trying to identify how many people stop in front of the window or enter the shop; - the product analysis to detect if the user is interacting with a specific product to provide more information about the product and to track the interaction, changing the classic idea of product expositors to the concept, better described in the next section, of Pick&Play. Several aspects of these problems are currently solved using artificial intelligence and, in particular, vision. Visual surveillance in dynamic scenes, especially for humans and vehicles, is currently one of the most active research topics in computer vision. It has a wide spectrum of promising applications, including access control in special areas, human identification at a distance, crowd flux statistics and congestion analysis, detection of anomalous behaviours, interactive surveillance using multiple cameras, etc. In general, the processing framework of visual surveillance in dynamic scenes includes the following stages: modelling of environments, detection of motion, classification of moving objects, tracking, understanding and description of behaviours, human identification, and fusion of data from multiple cameras [3, 6, 7]. As a result, more and more research has been conducted in this field involving a great quantity of different vision based approaches. In this paper we will introduce the general framework of our system and we will discuss how vision and, in particular, people and object tracking, are involved in different tasks of the whole system. We will show successful applications for retail environment management and for products management, showing the visual tracking methodology and results of real scenarios with real time performances. Input data are basically constituted by image sequences analyzed, in this first stage, off-line. Software analysis, tests and experiments developed are presented to show the feasibility of the proposed approach in large scale applications, using only vision sensors. The paper is more focused on the application of vision to retail environment and on the general application of visual tracking as a main information source; main novelties are in the original and wide application of the tracking system, in the real
experimental platform described in the result section and in the combination of vision based statistical approach with marketing analysis data. Apart from showing good performances using traditional tracking methodologies, another important characteristics of the framework actually developed is to allow future improvements by an easy testing of novel tracking approaches and performance comparisons. The paper is organized as follows: Section 2 introduces the general concept of the system and describes the peculiarity of the particular project described in this paper. Section 3 describes some research work, previously done, as the reference for this research and the method used for people tracking. The experimental results are provided in Section 4. Conclusions are given in Section 5. Fig. 1. The Intelligent Retail Environment architecture. 2 The Intelligent Retail Environment Architecture The whole system is designed by different layers involving the entire concept of retail environment analysis, going from costumers to expositors; in general the system is made by the following modules:
Inside shop people tracking Shop Window analysys Intelligent vision based expositors Data exchange layer and interoperability Fig. 1 describes the particular scheme of the management of a retail environment with all the previously described modules. Analyzing the problem from the observation of real scenarios, the following characteristics for the vision capability can be defined: - in the scene there are multiple objects simultaneously; therefore it is an issue of "Multiple Object Tracking"; - the shape and the size of objects are partially known; - objects are moving in the scene slowly and regularly; it is possible to make predictions about the direction of the car or bus and is possible to guide the driver to the correct location using dynamic directional information; - it is necessary to manage the "occlusions": two or more buses can transit very near to each other or group of people can pass nearby a car in the parking area; - occlusions could last for a long time; - occlusions could be total; for example, a costumer can be entirely hidden by an expositor. All these hypotheses are verified in the testing area used for results collection and to prove the proposed approach. In particular the first two layers are better explained in the next methodology section. The system allows having a mapping of the position of customers with respect to product on the shop area map, that joined to sell out data bring to important discussions about selling in function of product positioning. At the mean time the analysis of customers positioning bring to determine the medium amount of time passed in front of a particular expositor, trying also to identify how much impressive is a form a visual merchandising. In the case of whop window analysis the system is able to detect how many people are passing by, how many of them are entering the shop and how much time they spend looking at product exposed in the shop window. The intelligent expositor presented in fig.1 is named Pick & Play and is depicted in fig. 2.
Fig. 2 The Pick & Play structure The Pick & Play system (P&P) idea is to combining interactivity between products and consumers with important information about the product that the user is grabbing. A typical case can be a focus point, with several child shoes, the consumer attract by this tool is coming to check the shoe, so he takes the item and on the screen appear a new message (video, audio etc) that gives to the consumer others interesting news about the shoe and in a very impactful way. This functionality is available thanks to a low cost webcam focused on the area where products are placed, and by a software that, using real time information by the webcam, is able to manage multimedia contents (images, audio, video and so on) on the monitor. It's important to say that this hardware has no very high cost, respect to other technology that could provide similar services (for example, RFid tags). The P&P collect also statistical analysis on the type and number of product interactions. A last contribute of this project is the standardisation of the communication modules between different layers. 3. The vision based tracking system 3.1 Related works Extensive research regarding vision based tracking has been done over the past years. Several techniques have been developed as a result of these studies. One is Blob Tracking [4]. In this approach a background model without moving objects is generated for the scene. During the sequence each frame would be compared with the background model by doing the absolute difference between them, and, consequently, obtaining a foreground blob representing the vehicles. Another method is the Active Contour Tracking [4, 5]; this method tracks the contour of the foreground blob. The 3-D Model Based Tracking [4, 9] constructs a three-dimensional model using an aerial view of the scene to eliminate all occlusions. Feature-Tracking [2, 10] is
another method that uses feature points to track the objects; this method had brought good result where partial occlusion exists. This method also requires a grouping algorithm to track multiple features. Other methods can be found in [4, 7, 11]. 3.2 The visual tracking approach For the general purpose of video processing, the background is usually considered as the scene without the presence of objects of interest, such as man-made objects or moving vehicles. Background is usually composed of non-living objects that remain passively in the scene. In a video about a general environment, the background can consist of both stationary and moving objects. The stationary background objects can be walls, doors, and furniture in an indoor scenario, as well as buildings, vegetation, and ground surfaces in an outdoor scenario. With the formulation of background and foreground classification based on Bayes decision theory, an algorithm for foreground object detection from a real-time video containing complex background is established. It consists of four parts: change detection, change classification, foreground object segmentation, and background learning and maintenance. In the first step, not-changed pixels in the image stream are filtered out by using simple background and temporal differences. The detected changes are separated as pixels belonging to stationary and moving objects according to inter-frame changes. In the second step, the pixels associated with stationary or moving objects are further classified as background or foreground based on the learned statistics of colours and colour co-occurrences, respectively, by using the Bayes decision rule. In the third step, foreground objects are segmented by combining the classification results from both stationary and moving parts. In the fourth step, background models are updated. Both off-line and incremental earning strategies are utilized to learn the statistics of feature vectors. Meanwhile, a reference background image is maintained to make the background difference accurate and adaptive to the changing background. Further details about this mix of methodologies can be found in [7, 11]. 3.3 The communication and data exchange layer With the rapid acceleration in XML standardization activities there is confusion in the industry regarding which standards should be used to implement layered solutions. A Web service is defined by the W3C as "a software system designed to support interoperable machine-to-machine interaction over a network. To ensure the interoperability between our vision software layer and the decision support system layer devoted to optimization and dynamic resources allocation we developed a web service infrastructure based on the SOAP (Simple Object Access Protocol) standard for exchanging structured information in the implementation of Web Services. This communication layer exchanges information with the whole management system such as the id of the tracked people, the time of entrance, the place visited, the global localization; the vision based tracking software gives back several statistical
data, allowing alarm in case of incorrect matching of this data with the information provided by the management layer. 4. Results and discussion The system was extensively tested in a real scenario: a wide open exposition area, depicted in Fig. 3. Results were obtained using a C++ implementation of the software and working on images of 640x480 pixels and recorded using both directional and omnidirectional cameras. Fig. 3 Grottini Stand in Dusseldorf used for simulation of the retail environment; the whole area is about 200 square maters and is monitored by 3 cameras. Here following we show the result of the tracking in the shopping area (Fig. 4); Dark area are where people pass by more often. The experiments were performed recording data and performing elaborations off line. Our implementation of the algorithm is able to perform the tracking at 10 frames per seconds. Quantitative results are obtained by comparing the visual tracking performances with a human visually inspected tracking on a time slot of 5 minutes; the system correctly track the 94% of the moving people in the area. Errors occur in cases where a group of people are moving or in the case of stopping in a dark shadowed area. In any case the percentage for people tracking is very good and also speed performances are encouraging for future improvements and further image elaborations.
Fig. 4 Results of the tracking system in 2 days; 460 people tracked. Fig. 5 The real P&P installed in Dusseldorf; it was used by 260 people during two days with a predominance of the red shoes (64% of cases). 4 Discussion and Conclusions We presented a novel application of visual tracking, using popular techniques, tested in a real environment with interesting results in the field of retail marketing analysis. The paper presented also an integrated architecture for mixing together different kind of vision based applications such as people and products tracking. Future work about
this project is, first of all, the optimization of these systems in terms of stability, performances and robustness to environmental inconveniences, considering that in a shop there is no special worker able to reset or modify these systems in case of trouble. The final step of this project is to provide the shop with a mobile robot assistant, able to move into the shop and to interact either whit the environment and with the customers, giving them information about products, indicating where they can find a specific one and so on. We will also apply novel tracking methodologies that are currently in a test phase; these approaches are based on feature group matching [1] and they are promising techniques able to make the tracking much more robust. Acknowledgments. Authors would like to thank the Grottini Communication company for providing the experience in the field and for supporting the project. References 1. Ascani A., Frontoni E., Mancini A., Zingaretti P.: Feature group matching for appearancebased localization. IEEE/RSJ 2008 International Conference on Intelligent RObots and Systems IROS 2008, Nice, (2008) 2. Cheung S. C. and Kamath C.: Robust techniques for background subtraction in urban traffic video. In: Proceedings of Electronic Imaging: Visual Communications and Image Processing, (2004) 3. Dewees, T.G.: Vehicle tracking driver assistance system, US Patent7,038,712, (2006) 4. Kanhere N. K., Birchfield S. T., Sarasua W. A.: Vehicle Segmentation and Tracking in the Presence of Occlusions. In: TRB Annual Meeting Compendium of Papers, Transportation Research Board Annual Meeting, Washington, D.C., (2006) 5. Kanhere N. K., Pundlik S. J., Birchfield S. T.: Vehicle Segmentation and Tracking from a Low-Angle Off-Axis Camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, California, (2005) 6. Krishnamurthy, J. Mohan, N. Hegde, R.: Automation of Toll Gate and Vehicle Tracking. In: International Conference on Computer Science and Information Technology - ICCSIT '08, Singapore, (2008) 7. Magee D.: Tracking Multiple Vehicles using Foreground, Background and Motion Models. Image and Vision Computing, vol 22(2), pp. 143-155, (2004) 8. Mancini A., Cesetti A., Iaulè A., Frontoni E., Zingaretti P., Longhi S.: A Framework for simulation and testing of UAVs in cooperative scenarios. Journal of Intelligent and Robotic Systems, Springer, DOI: 10.1007/s10846-008-9268-8, (2008) 9. Pundlik S. J., Birchfield S. T.: Motion Segmentation at Any Speed, In: Proceedings of the British Machine Vision Conference (BMVC), Edinburgh, Scotland, (2006) 10. Tomasi C., Kanade T.: Detection and tracking of point features. Technical Report CMU- CS-91-132, Carnegie Mellon University, (1991) 11. Williams O., Blake A., Cipolla R.: Sparse Bayesian Regression for Efficient Visual Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 27( 8), pp. 1292-1304, (2005)