Modeling KANSEI Through Real World Interaction

Modeling KANSEI through Real World Interaction with Ubiquitous Information Environment Smart Store - Kazuhide HIGUCHI*, Tomohiko SAITO, Somkiat SEA-UENG, Toshiyasu KANAMORI, Hideaki SEKIHARA, Yoichi WATANABE, Kazuya HAMATANI, Toshikazu KATO *Dept. of Industrial and Systems Engineering, Chuo University 1-13-27, Kasuga, Bunkyo-ku, Tokyo 112-8551 JAPAN kaz@indsy s.chuo-u.a c.jp Abstract: It is convenient for customers to be offered the services suitable for tastes and interests of each customer in the stores. Therefore, stores need tastes and interests of each customer on goods. We call such tastes and interests KANSEI. The purpose of this study is research and development of the goods recommendation system based on KANSEI model. Our system needs to develop following functions. (1) Function for ubiquitous interaction in the real world: We need to acquire KANSEI from the customer's action in the store. Therefore, we observe customers passively and actively using cameras on the ceiling and the shelves and touch-panel displays near the shelves. The passive observation is measuring behavior without showing any information to the consumer. The active observation is measuring his reaction to information of test samples on touch-panel displays. (2) Function for modeling each customer's KANSEI: Tastes and interests of customers in stores are shown as their tracks and gaze. Therefore, to observe customers leads to creating KANSEI model. We take much time to create the KANSEI model for each customer by the questionnaires. We can also define an average KANSEI model for a set of consumers who have common profiles such as age, sex, hobbies, and so on. By adding the individual responses to the average KANSEI model, we can create the KANSEI model for individual customer with less interaction. (3) Function for recommendation of goods based on each customer's KANSEI: We recommend the goods based on the customer s KANSEI model. We built the interactive store space to provide human-friendly interaction. Using our system, we can offer the information on the goods suitable for customer's KANSEI. Key words: interaction, smart store, augmented reality interface 1. Introduction Each person has his own tastes, interests, and hobbies. This is because each person has his own subjective and intuitive judgment standards. We model KANSEI by paying attention to this judgment standard. If we model KANSEI, we can offer the service customized to each person. We model KANSEI by paying attention to this

judgment standard. We see KANSEI in human s behavior. We observe human s behavior because we model KANSEI. A real world interaction is an interaction from which a user's natural behavior in the real world will receive the information from the influence and the information system to an information system. We call the interface for a real world interaction real world interface. We observe human s behavior through real world interface. Our research goal is to make a real world interface and to offer the service using this in not room space but office space and shop space. We call the space that human can interact with real world interface and can be offered service Smart Sphere. In first phase, as an example of Smart Sphere, we make a real world interface for shop space, and offer the service using this. We deal a goods recommendation service of a store that uses the virtual reality interface and the KANSEI model. In store, it is convenient for each customer to be served in accordance with his KANSEI. For that purpose, stores need KANSEI model of each customer on goods. Our system gets the KANSEI through interaction and customer's behavior in the store by making the whole store into an augmented reality interface. Then, our system recommends suitable goods to a customer on demand using a got KANSEI, his former behavior and the history of his purchase. The remainder of this paper is organized as follows. Section 2 describes a real world interface. Section 3 describes the measurement technique of KANSEI. Section 4 describes the actually built smart shop. Section 5 is the conclusion of this paper. 2. Augmented Reality Interface Fig. 1 an example of real world interface. 2.1. Approach by Wearable Computer By being always with a person, a wearable computer recognizes the situation that he is being put, and it intends to offer useful information. We consider that the computer is reinforcing the interaction with a real world. However, this approach needs to make a human put a computer on. Therefore, this approach is not human-friendly. For example goal of Weavy[8] is to realize the smart interface that recognizes the user s situation from sensing result and their history. This is not user-friendly because user must wear the sensor on his head and arms.

2.2. Approach by Virtual Reality Another approach is to build the virtual space by virtual reality, and use this for interface. Virtual reality makes the real world model in a computer. Modeling the real world strictly by the computer needs a lot of calculation in real time. Virtual reality has following problems: (i) detailed modeling is difficult in respect of the amount of calculation. (ii) in respect of a device, the resolution and luminosity which can be shown are not enough. (iii) in respect of a multisensory interface, smell and tactile feeling can not show enough.therefore, a natural interaction with real world is difficult in virtual reality. For example, CABIN[7] can not show smell and tactile feeling enough. 2.3. Approach by Augmented Reality The other approach is to augment the real world with the virtual reality technologies by reflecting the event in computer to real world and reflecting the event in real world to computer. This approach is user-friendlier for users than that by a wearable computer because they do not need to carry computer devices with them. Since a user interacts with computers in the real world, this approach has more inherent reality than a virtual reality. Therefore, we adopted the approach by augmented reality. Then, we are developing real world interface. A real world interface is an interface to which a user's natural action in the real world makes possible the interaction which will receive the information from the influence and the information system to an information system Fig.1..This interaction is performed through a user's behavior and sense. As an research of Smart Room, there are M.Tominaga, et al[8], E.Irfan[6], and A.Pentland[2][3]. The newest technology over four areas of (i) person identification, (ii) surveillance/monitoring, (iii) a 3D method, and (iv) a smart room/perceptual user interface is described in A.Pentland[2]. Person identification by face recognition for smart environments is described in A.Pentland[3]. They observe the indoor space that assumed the living room with the camera in M.Tominaga et al[5]. Then they estimate the position of person in the room, and recognize the hand sign. They do not offer the service using person s KANSEI. They monitor not living room but whole inside the house in E.Irfan[6]. This study assumes only the house and not monitors user s situation through sensor. Using an augmented reality interface, we want to make a KANSEI model from person s environment. An augmented reality interface presumes the tendency about user s KANSEI from his behavior pattern. An augmented reality interface needs to use the presumed tendency for (a) an implicit intention of the user, (b) choose the information, and (c) control interaction. 3. Measurement Technique of KANSEI at Smart Sphere 3.1. Macro/ Mezzo/ Micro Scope We pay attention to the situation what changes with time. By focusing on a person, we classified the real world situation into (1) mental situation and (2) physical situation, about person. As a mental situation we can give (a) user s position (example: position in the building), (b) user s behavior (example: object user looks at), (c) user s pose (example: standing).

We need to observe from a viewpoint of macro, micro, and mezzo, because we observe these physical situation and change at that time. In macro perspective, we measure positions of all the user in real world. In micro perspective, we recognize the user s face and observe the user s behavior. In mezzo perspective, we measure position of user and change of behavior using macro measurement and micro measurement. For example, in the case of a store, it is as follows. In macro perspective, we measure positions of all the user in the store. In micro perspective, we recognize the user s face and gaze in the store. In mezzo perspective, we measure user s track using macro measurement and micro measurement. 3.2. Positive and Active Observation The information system that recognized the real world situation needs interaction media to interact with a user. We measure a user through cameras, speakers, lighting, and touch-panel displays as an interaction media, and show information to a user. We observe user s KANSEI actively and passively. The passive observation is measuring behavior without showing any information to the user. If person gets interested in something, he will move to it, or he will look. Therefore by observing person s position and behavior we can estimate the object that he has interested. The active observation is measuring his reaction to information of test samples. Since a user moves or gazes to the information on things interested, an information system can measure a user's interest. We need lighting, speakers, touch-panels displays, and etc. to observe actively. 3.3. Direct/indirect Interaction In Smart Sphere, even if a user directly touches an interface and does not operates it, he can operate it indirectly by his behavior. Therefore, a user can interact with the information system through his behavior. For example, operation whose user takes goods in his hand on the inside of a shop is equivalent to the user having answered indirectly on the touch-panel display. The difference between our system and the conventional system is to get data of how a user judged through his behavior. 3.4. Modeling KANSEI As a mental situation, we can give user s behavior. An augmented reality interface can observe the mental situation by measuring his reaction to information of test samples. The method for measuring a user's judgment standard is as follows. First we show the information about object through augmented realty interface. Then, if a user has interested in it, he looks or moves to it. Next, we measure whether it led to change of a user's behavior. Change of a user's behavior shows evaluation of the user to the shown information. Thus, a user's KANSEI is acquirable by measuring a user's mental situation in the real world. Moreover, we are recording a log of change of a user's behavior and the shown information. We create the model that can estimate a user's interest by analyzing this log statistically. We call this model KANSEI model. A log is recorded on at any time by a user's behavior, and a user's KANSEI

model is updated. In the first phase of smear sphere, we get user s KANSEI through cameras and touch-panel displays that are ubiquitously in the real world space. We observe users passively and actively through cameras and touch-panel displays. The former technique needs many questionnaires and analysis to modeling KANSEI. We used the augmented reality interface in order to build KANSEI model. 4. Application to Smart Store We develop smart store as an example of smart sphere. Smart store estimates goods suitable for each customer from past and present behavior and situation. Then smart store recommends the goods. Therefore, a system always needs to know where each customer is. As smart store, we assume the store of the type of both shop and department store. This model stores up behavior of every person's past, and the log of a situation. The information system estimates based on this history (Fig.2.).

Fig. 2. Flow of Smart Store System 4.1. Interaction in the Real World Smart store has cameras, touch-panel displays, lighting, and speakers as the device to interaction with customers. At the 1st phase of smart store, we use cameras and touch-panel displays for an interaction with customers. In an interaction with human the information system needs identification of individuals. In smart store, while customer look at the goods shelf we photo a front face of customer. An information system judges person who uses the store through face recognition. If each customer can

discriminate, his name, address, etc. do not need to be known. By identifying an individual, the information system can specify the candidate of human who is interacting with the information system now. By specifying a candidate, the information system can specify whose KANSEI model it is. 4.2. Recognition Situation in the Smart Store Using recognition of real world situation, we analyze the customer s behavior from a viewpoint of (a) micro perspective, (b) macro perspective, and (c) mezzo perspective. We can presume the situation that the customer is put at that time through analyzing his behavior. In the micro perspective, the information system analyses the discrimination of the customer s face and direction of face, residence time in the front of the goods shelf, and goods with interest. By distinguishing the customer, we record on a log about his behavior and situation, even if he is in anywhere. Also we understand which goods the customer has interesting from the direction of his face. The information system decides whether the customer has interested using residence time. In the macro perspective, the information system analyses all the customers behavior in the whole store. Smart store understands which goods shelf is popularity for customers through their behavior. Also the customer can understand how the other customer behaves in the store. In the mezzo perspective, the information system analyses customer s track. By combining with the discrimination of customer s face, we can track him. Also, we understand the object in which a customer has the interest and history of behavior. The information system can observe many persons at the same time and modeling their own KANSEI in our augmented reality interface. The information system recognizes the behavior and situation as a whole because the information system can observe many persons at the same time. 4.3. Modeling KANSEI In the first phase of smart store, in order to measure a customer's judgment standard, we showed the information on goods to the customer on the touch-panel display. Then we observed change of customer's behavior. We analyzed statistically the log of the behavior and situation, and created a customer's KANSEI model. By using a real world interface, we observe customer's behavior and situation on real time, and we record on a log. Therefore, customer's KANSEI model can be updated at any time. 4.4. Advantage by Using an Augmented Reality Interface In Smart Store, we could get the following advantages using a real world interface. We guess the customer s taste from his inherent behavior through an augmented reality interface in the store. The store can automatically recommend the marvelous goods that are in accordance with customer s KANSEI. 4.5. Experiment and Evaluation We built the Smart Store with an augmented reality interface actually and experimented. We discriminate the individual as recognition of a real world situation. We use a person s face for the

individual discrimination. We obtained the following experiment results from the former research [4]. In the individual discrimination, we succeeded in the discrimination of known person at 98.64% probability. It succeeded in the discrimination of first person at 89.35% probability. Therefore, we can accumulate user's data, even if he is in anywhere. As the result, a goods information system could recommend the goods in accordance with each user KANSEI. The system was able to get many users KANSEI at the same time. As the result, we can offer the goods information suitable for customer s tastes and wide goods using our mechanism. 5. Conclusion We observed behavior of users actively and passively, and get data of his behavior. By distinguishing the individual, we can accumulate the data of the user, even if he is in anywhere. The system can to recognize the real world situation. The system can get user s KANSEI through an augmented reality interface. The system can serve in accordance with user s KANSEI in real time. References [1] J.Rekimoto: A tutorial introduction to computer augmented environments, Computer Software, Vol.13, No.3, pp4-18 (in Japanese). [2] A.Pentland: Looking at People: Sensing for Ubiquitous and Wearable Computing, IEEE PAMI, Vol.22, No.1, pp.107-119. [3] A Pentland, T.Choudhury: Face Recognition for Smart Environments, Computer, Vol. 33 Issue 2, pp.50-55. [4] H.Sekihara: Face Recognition by Local Contrast and Distinction Analysis, Japan Society of Kansei Engineering Kansei Kobo Workgroup, 2003(in Japanese). [5] M.Tominaga et al.: Extraction of Hand and Estimation of Direction from Multiple Cameras for Hand Signed Recognition, Technical Report of IEICE (PRMU2001-113, HIP2001-16, MVE2001-75), Vol.101, No.425, pp.1-8 (Nov.2001). [6] E.Irfan: Ubiquitous Sensing for Smart and Aware Environments: Technologies Towards the Building of an Aware Home, Position Paper for the DARPA/NSF/NIST Workshop on Smart Environments, July 1999. [7] M.Hirose, T.Ogi, S.Ishiwata and T.Yamada: "Development and Evaluation of Immersive Multiscreen Display "CABIN"", Systems and Computers in Japan,scripta Technica,Vol.30, No.1, pp.13-22, (Jan.1999). [8] T.Nishimura et al: A Cooperative Environment of Wearable and Ubiquitous System: Towards Barrier-free Information Support, Technical Report of IEICE(PRMU2002-179), pp.61-66(2003).