Always Listening, Always On: Advances in Sensory Processing

Transcription

1 Always Listening, Always On: Advances in Sensory Processing By Linley Gwennap Principal Analyst, The Linley Group March

2 Always Listening, Always On: Advances in Sensory Processing By Linley Gwennap, Principal Analyst for Mobile, The Linley Group Smartphones use sensors and voice input to become aware of their surroundings and to interact more naturally with their users. Smartphone makers continue to develop more advanced capabilities, adding new types of sensors and more sophisticated voice functions. If implemented improperly, these changes can greatly reduce battery life. This paper explores the latest advances in sensory processing and how new semiconductor products can implement these features using minimal power. Introduction The first smartphones tried to bring the PC experience into the palm of your hand. Apple took a different approach, creating a new user interface that relied more on touch and even on the orientation of the phone. Since then, smartphone makers have raced to add more capabilities, helping their devices to be aware of and react to their surroundings. These capabilities depend on an expanding array of sensors that react to gravity, motion, and light. Voice has become a central function of high-end smartphones. Apple again took the lead by introducing Siri, a virtual assistant, as part of ios 5 (released in 2011). Siri responds to natural-language commands, performing information searches and controlling aspects of the device. Other smartphone makers now offer similar functions, which allow unsophisticated users to more easily access advanced features of their devices. Developers are now taking these functions to the next level. By comparing sensor data to existing motion signatures, software can recognize gestures, count the number of steps the user has taken, or analyze the effectiveness of their exercise. Voice operation is being improved to operate even in difficult situations, such as noisy backgrounds, moving vehicles, or even completely hands-free. These changes require advances in smartphone hardware design. Smartphones Adopt Sensor Hubs A three-axis accelerometer is now standard in all smartphones to provide basic motion and position input to the software. A gyroscope, however, provides greater accuracy and lower latency in sensing rotational motion. (An accelerometer-and-gyro combination is sometimes called a six-axis accelerometer.) First introduced in iphone 4 (2010), gyros now appear in most mainstream smartphones. High-end smartphones continue to add sensors, including a magnetometer (compass), barometer (pressure), ALS (ambient-light sensor), UV (ultraviolet-light) sensor, and infrared proximity sensor. The barometer helps determine altitude above sea level, which improves positioning for location-based services. The ALS can extend battery life by adjusting the backlight. The proximity sensor typically turns the display off when the 2013 The Linley Group - 1 -

3 phone is held close to the face or placed in a pocket; it can also be used to detect simple gestures. In total, the sensors in the iphone 5s cost $15, according to isuppli. As their costs fall, these sensors will migrate into mainstream phones, just like the gyro did. Managing all of this sensor data is becoming complex, particularly for applications that continually measure motion, such as a fitness app. Using the main processor (for example, Cortex-A15) to monitor the sensors would be power intensive. To offload this sensor processing, the recent iphone 5s includes the Apple M7 chip, which is based on a simple Cortex-M3 microcontroller. Other high-end phones use a similar sensor hub to coordinate this data, waking the main processor only when necessary. Voice Processing Becomes Always On Voice is another form of sensory input. Users have been able to talk to their phones for several years, starting with rudimentary voice dialing and voice commands. Originally, these functions required user-specific training, but modern processors can implement speaker-independent voice recognition for a limited set of words. Sophisticated services, such as voice search and virtual assistants, need large vocabularies; these services often implement voice recognition in the cloud, sending the voice sample through the network to a high-powered server in a distant data center. These services simplify interacting with increasingly smart phones, particularly in situations where fine control of the touchscreen is difficult, such as when driving or walking, or riding in a fast-moving vehicle. One problem is that a noisy environment (e.g., a car at highway speeds, a busy street) makes voice recognition more difficult and therefore less accurate, even when performed on a server. The solution is to implement noise reduction on the phone, enhancing the voice signal while reducing the background noise. Preprocessing the voice sample in this way enables voice operation in a diverse range of situations. Another problem with voice services is that they must be enabled physically, such as by pressing a button or launching an app on the phone. These actions may be difficult or impossible in some situations. In a car, the phone should be in a true hands-free mode that does not require the driver to stop holding the steering wheel to manipulate the phone. The user may have their hands full or may be using their hands for other activities. The phone may be out of reach (perhaps on a table or nightstand) or even in a pocket. In these scenarios, using voice commands is impossible with most of today s smartphones. A device that is always on, always listening solves this problem. This technology, first implemented in the recent Moto X phone, allows the user to create a verbal phrase, such as OK Google Now, that will wake the device. The phone can then accept voice commands such as set alarm for 10AM, all without ever touching the device. Moto X also accepts gestures such as flicking the wrist to put the phone in camera mode, again without first pushing a button. These gestures reduce the time needed to open an app and, in this case, prepare to capture a fast-moving event The Linley Group, Inc

4 Always Listening, Always Power Efficient Most phones today implement voice commands on the main CPU. These phones require the user to press a button to wake up this power-hungry CPU just long enough to process the command. As with other sensors, the key to always listening is to offload this function from the main CPU. Implementing it on a smaller, power-efficient processor creates minimal impact on battery life. For example, the new es700 chip from Audience uses just 0.6mA current draw from the battery 1 when listening, meaning that turning on this feature will reduce the battery life of an average phone by an unnoticeable 15 seconds. The firmware must also be properly designed to minimize power. In its low-power listening mode, the Audience chip simply waits for any voice signal. A quiet room or one with moderate background noise will not trigger the chip. But when the chip detects a voice, it switches to a slightly higher-power mode to determine if the voice input matches the designated wake-up phrase; if not, it goes back to its low-power mode. Only if it detects the wake-up phrase will it wake the main processor and hand it the entire voice buffer. This buffer includes any words spoken after the wake-up phrase, allowing users to speak naturally and without a pause between the phrase and the command. A sensor hub creates a similar opportunity for other sensors. In the Moto X, for example, this chip monitors the accelerometer for a motion signature that matches the camera-on gesture. To improve power efficiency, this function runs on a microcontroller, not the main processor. Motorola, however, needed to add a second chip to supplement the microcontroller with DSP capability, adding cost to the system. The es700 series combines a processor and DSP capability on a single chip. A DSP uses less power than a standard CPU for voice processing, because it has been optimized for that function. Like a microcontroller, the Audience chip also integrates memory, allowing it to store the voice buffer and related data. Keeping this data on chip reduces both power and cost, compared with external memory. Furthermore, the es750 series integrates an A/D converter, audio codec, and other analog functions. The Moto X design, for example, requires a third chip for the A/D converter. These extra chips take up board area, making the phone larger than it would be with a highly integrated device such as the es750 series. Audience builds the new es700 series in 40nm LP technology. Compared to the 28nm LP process typically used for application processors (APs), 40nm LP has lower leakage current. In always-on mode, leakage current becomes a significant contributor to overall power, so this process difference gives the Audience chip an advantage over the AP for always-on processing. On the other hand, integrating always-on technology with the sensors themselves is not a good fit. Most sensors use MEMS designs. Although these micro-machines are tiny compared to real gyroscopes and compasses, they are still much larger than transistors. 1Assuming Vbat=4.2V and a DC/DC converter with 80% DC-DC efficiency 2013 The Linley Group, Inc

5 Most MEMS devices are manufactured in specialized processes that are incompatible with advanced CMOS logic. Keeping the power-sensitive circuits (DSP) on one die and the sensors on a separate die allows both to be better optimized for cost and performance. Emerging Applications Although smartphones are the initial focus for always-on features, this technology applies to other types of devices. Tablet computers are an obvious application, since they share many of the same usage models as smartphones. Bulkier and heavier than smartphones, tablets are more likely to be sitting on a table than in the user s hand, making voice activation more desirable. Moto X s wrist-flick gesture is not appropriate for a large device, but a tablet may recognize gestures using its camera or proximity sensor instead. Laptop PCs, particularly 2-in-1 devices, have similar characteristics and could benefit from voice activation and voice commands. The digital living room is becoming more complex as consumers combine smart TVs, set-top boxes, game consoles, Blu-ray players, and streaming players that deliver overthe-top Internet video. In addition to the usual cluster of remotes, some systems now offer a keyboard to enter URLs. This situation cries out for a simple voice-operated interface that can control all of these devices using natural-language processing performed in the cloud. This interface could use a remote that the consumer speaks to, using the traditional push-to-talk model. A better approach would be that the TV and other components all incorporate always listening technology, sensing when a user enters the room and directly executing speech commands. Automobiles require a hands-free environment. Many cars today have a built-in system that gives navigation instructions, plays music, controls the temperature, and connects to the driver s smartphone. Some cars use a button on the steering wheel to initiate voice commands, but an ideal system would be always listening, so the driver would not have to worry about the button. A flexible system would also have a large vocabulary to control the various functions and include noise suppression to improve operation even with a large amount of road noise. The market for wearables is just starting to emerge, but the small size of these devices makes them a strong candidate for voice control. A watch, for example, has little room for touchscreen buttons and even less for physical buttons. Fitness bands are even more limited in size. Speaking to these devices would create a much more powerful and flexible user interface. Wearables also have tiny batteries, increasing the importance of a power-efficient processor. Although this market is small today, it could grow quickly as prices come down. Conclusions As a smartphone adds more sensors and sensing, it becomes more aware of its surroundings, including its location, orientation, and motion as well as the behavior of the people around it. Combining this data with user information such as calendar and address-book entries enables powerful applications. The smartphone can quickly 2013 The Linley Group, Inc

6 respond to, and even anticipate, the user s needs. With a sophisticated voice interface that is always listening, the device becomes a life partner and not just a gizmo. Enabling this advanced functionality requires equally advanced hardware. No matter how powerful the application processor, it cannot perform these functions without quickly depleting the battery. For maximum power efficiency, sensor and voice processing must be offloaded to a separate processor. This approach, combined with intelligent power-management software, will maintain battery life even when the phone is always on. With integrated memory and DSP capability, the Audience es700 is a low-power processor that is well suited for always-listening devices. It also includes the company s industry-leading noise-suppression technology, improving voice recognition in noisy environments. Some models contain an audio codec, connecting directly to analog microphones and speakers to form a complete audio subsystem. This design reduces system cost and power. In addition to smartphones, always-listening technology is applicable to tablets, laptop PCs, smart TVs and other consumer-video products, automotive in-dash systems, and wearables. As these markets develop, consumers will discover a new, more natural way of interacting with their devices. Linley Gwennap is founder and principal analyst of The Linley Group and coauthor of A Guide to Mobile Processors. The Linley Group offers the most comprehensive analysis of the mobilesilicon industry. Our in-depth reports covers topics including application processors, cellularbaseband processors, Wi-Fi combos, and other connectivity chips. For more information, see our web site at Trademark names are used throughout this paper in an editorial fashion and are not denoted with a trademark symbol. These trademarks are the property of their respective owners. This paper is sponsored by Audience, Inc., but all opinions and analysis are those of the author The Linley Group, Inc