Debugging Android App using RxLogger Subbu Jagadeesan Sr. Engineering Manager, EMC Bangalore Solution Center
AGENDA RxLogger Overview What is it Features Configuration RxLogger Output
Overview What is RxLogger? RxLogger is a collection of Android debug/diagnostic tools Logs System Dumps Performance Monitoring ANR Capture Resource Monitoring Resource Monitoring When is it used? Diagnose problems in the field Provides remote debug capability
Features Diagnostic tool for Android TM similar to emscript on WM/CE Familiar high level CSV log format with additional drill down logs Pluggable interface Allows for 3 rd party development and seamless add-ons Flexible configuration Configurable through user interface or config xml file for ease of use in single or mass deployments Remote log retrieval Capability to push logs to a remote FTP server Remote Debugging UI available on device Ability to view logs and configure on device
USER INTERFACE Simple interface to allow user control and configuration Notification icon to indicate when the tool is running Features Backup creates a zip file of current logs View displays logs on device
Output UI and log example Application Not Responsive traces XML configuration Detailed snapshots Network traces (disabled by default) Logcat event buffer Logcat main buffer Logcat radio buffer High level resource logging Logcat system buffer Main Output Log Time Line Sub-System Information Events/Details
Plugin Menus Main System Monitor Resource.csv PushPullClient Push Log Files/Pull configuration LTS- Long Term Storage Backup RxLogger files Snapshot Detailed Process information
CONFIGURATION - main USER INTERFACE Configure settings for all available plugins Configuration stored in XML Automatic restart of service when new config file detected
PUSH PULL PLUGIN The Push Pull Plugin enables pushing of local log files to an FTP server. It can also be used to pull a file (such as a config.xml) from a server to the device. Pushed files are organized on the FTP server by device serial number. Remote folders per device
RxLogger Output
Viewing Log Output Log files are generally reviewed in the following order High Level Time line (Resourcex.csv) - Identify outliers/anomalies - Directs what details to look at next LogCat (developer trace messages) General Detailed debug traces when looking for a specific issue Application not responsive (ANR) Points to why an application locked up Catch errors users don t tell you about Various (plug in) snapshots Deep dive details on specific subsystem area
FILE STRUCTURE LOG OUTPUT Application Not Responsive traces XML configuration Detailed snapshots Network traces (disabled by default) Logcat event buffer Logcat main buffer Logcat radio buffer High level resource logging Logcat system buffer
Main CSV Log Features Date/Time Power Battery Capacity Battery Current Battery Voltage Battery Temp Screen State Screen Brightness System Resources Avg CPU% Current CPU% Free Physical Mem Free Storage Mem Process Count Foreground Package Foreground Activity Captured Parameters: Network IP Address Gateway Total TX Total RX WAN Total Mobile TX Total Mobile RX WAN State WAN Type Signal Strength Wifi Wifi Enabled BSSID RSSI AP Bluetooth BT Enabled BT Discoverable BT Connected Events Event Type HeartBeat ANR Snapshot External Power Event Details SnapShot file ANR Details
Main Log Sample Power CPU & Memory Network WLAN LAN BlueTooth Events and Details
CPU USAGE Resource0.csv Graphing the CPU usage shows a sustained elevation The foreground package change seems to correspond to the jump in CPU usage
ANR LOGS ANR or (Application Not Responsive) is an event triggered by the OS when an application s main thread blocks for a long period of time. Typically the user will be presented with a dialog to force close the app or to wait. RxLogger detects these events and saves the call stack information for later analysis.
UNRESPONSIVE APP Thread call stacks of every Dalvik thread in the system is logged in the ANR log file In this case an activity called UnresponsiveActivity called into a function called beunresponsive which then blocked in a sleep call.
LOGCAT LOG OUTPUT Logcat is the built-in Android logging mechanism. It is a collection of circular memory buffers that log information from the system as well as user applications RxLogger records all four available Logcat buffers: 1. Main Mainly used for application/framework logs Good place to start to get a general idea of what the system is doing at a given time 2. System Messages from system services (power manager, battery, package manager etc.) 3. Event Messages regarding the creating/destroying of activities and services Helpful for troubleshooting application/service crashes 4. Radio Message from the RIL and other cellular components
SNAPSHOTS LOG OUTPUT Snapshots are detailed logs that are recorded once every 30 minutes by default. They are intended to provide a second level analysis based on information obtained from the Resource.csv file. Information captured is configurable System Resources Per process CPU Per process memory usage Process/thread list Power Battery history Wakelocks CPU states System Properties Network Interfaces Routing table Data connections File System Partitions Free Space Usage Statistics Number of starts for each package per day Snapshot Configuration UI
Snapshot Sample Opening snapshot of interest shows top CPU usage by each process Badappa is consuming > 40% of each core Measurements from a TC55 - dual core processor
TCPDUMP LOG OUTPUT TCPDump collects network packet traces on the device to be viewed with a tool like Wireshark. Helpful for debugging web apps or other network issues.
M O TO RO LA an d th e S tyl ized M Lo go are re gis tere d in t he US P aten t & Tra dem ark O ff ice. A ll ot h er p rodu c t or serv ic e nam e s a re th e prop ert y o f thei r re s pec tiv e o wners. M ot orol a, I nc. 201 1 RxLogger Plug-In Development RxLogger provides an SDK to support Plug-In development. Import the sdk (rxloggerplugininterface.jar)into the Android Project and configure the Build Path ( Build Path -> Configure build path) to add the jar file. The plug-in class should have RxLoggerPluginService from the jar file as its super class. Implement the required methods as shown getpluginname() plugininit() plugindeinit() pluginstart() pluginstop() Refer to the user guide for more details: R xl og g er U ser G u id e v 1.0 Jo h n S eim e r, C a thy W e i, M att WCh itom e pan y N a m e S o lution N am e w / S olu tio n V er sio n M otorola G lo b al S olut io n Ce n te rs
THANK YOU
TROUBLESHOOTING METHODOLOGY Level 1 WHAT Identify the symptom Look for red flags that may point to an area of interest High CPU High Mem Resource.csv Short Battery Life Network App Crash Level 2 WHO Identify the offending application/process Use event column to find detailed log file around time of red flag Per Process CPU dump in Snapshot Per Process mem dump in Snapshot Wakelock dump in Snapshot TCPDump network traces Per Process CPU dump in Snapshot Level 3 WHERE Identify the part of the application causing the problem PC/Connected Debug Tools Eclipse debugger, method profiler, HPROF heap dump analyzer, Wireshark, etc.
RESOURCE CSV LOG LOG OUTPUT The Resource.csv file presents a timeline of device activity and vital statistics The default configuration will log once per minute plus asynchronous events (power notifications, etc.)
FAULT FINDING EXAMPLES
Detecting High CPU usage Resource0.csv Graphing the CPU usage shows a sustained elevation The foreground package change seems to correspond to the jump in CPU usage
CPU USAGE FAULT FINDING EXAMPLES Opening snapshot of interest shows top CPU usage by each process badappa is consuming > 40% of each core Measurements from a TC55 - dual core processor
MEMORY USAGE FAULT FINDING EXAMPLES A steady decline of Free Physical Mem over a long period of time could be indicative of a memory leak. In this case the memory keeps decreasing until the app is killed and then all the leaked memory is reclaimed.
MEMORY USAGE FAULT FINDING EXAMPLES Memory used by each process can be viewed individually in snapshot files for growth
UNRESPONSIVE APP FAULT FINDING EXAMPLES Browsing the event column of the Resource.csv file we can see an ANR situation was detected Using this message we can find the log file associated with this event (traces_com.example.unresponsiveapp.txt)
UNRESPONSIVE APP FAULT FINDING EXAMPLES Thread call stacks of every Dalvik thread in the system is logged in the ANR log file In this case an activity called UnresponsiveActivity called into a function called beunresponsive which then blocked in a sleep call.
WAKELOCKS FAULT FINDING EXAMPLES Note High current draw while the terminal is in suspend Screen State = 0 High current draw over time will impact battery life
WAKELOCKS FAULT FINDING EXAMPLES Looking at power section of snapshot shows all requested wake locks PowerManagerService was holding a wake lock for over 45 minutes in the PowerMaagerService encompasses all partial wakelocks requested by Apps so we need to look further
WAKELOCKS FAULT FINDING EXAMPLES WakeLockerBadWakeLock is the com.example.badappd
OTHER TOOLS RxLogger can coexist and complement existing Android tools. mlog is a tool built in to the OS on some of our Android products. It collects some similar information however the focus is on a user initiated export of the logs after a problem occurs.
THANK YOU