Debugging Machine Check Exceptions on Embedded IA Platforms

Transcription

1 White Paper Ashley Montgomery Platform Application Engineer Intel Corporation Tian Tian Platform Application Engineer Intel Corporation Debugging Machine Check Exceptions on Embedded IA Platforms July

2 Executive Summary Embedded systems need to be able to detect, recover from, and report errors. This is a critical feature not only during debugging but also for quality control after product manufacturing has begun. The importance of advanced error handling capabilities is often magnified for embedded systems because many are deployed in a large number of units, dispersed widely, and are running mission-critical type applications. Further, the embedded systems present a unique challenge due to their diverse form factors, vastly different feature sets, and special usage models. IA processors include Machine Check Architecture, which has built-in capability to detect, report, and attempt recovery from the system errors the CPU observes. As IA gets increasingly popular in the embedded space, the value and significance of machine check exceptions grows. Embedded products are often running critical applications non-stop for extended periods of time where unexpected system resets may present significant impact. Many times, machine check exceptions are the only available clue that the customer has during system failures and they provide a starting point for debugging. Diagnosing the cause of machine check exceptions can be challenging and time consuming. They are often difficult to reproduce in a timely manner. There are also many potential suspects involved, such as: platform design issues, the CPU operating out of specification, overloading of power supplies, software applications, and BIOS. This makes the debugging process extremely challenging. Machine check architecture has been available in IA processors since the Pentium days to report and record errors in the system as observed by 2

3 the CPU. When the CPU detects critical machine check exceptions and the errors are not correctable, the CPU will reset the system to prevent error situations from getting worse. The MCE registers capture some of the error information as seen by the CPU at the point of failure, which can be important information in order to get to the root cause of the error. This application note is intended to provide recommendations on how to debug machine check exceptions on embedded IA platforms. It also goes over the Machine Check Architecture and uses the Intel Core Duo processor and Intel Core 2 Duo processor as examples. However, the information and methodology is generic to all newer IA processors. This document presents a step-by-step approach to debugging Machine check exceptions, understanding their causes, and reaching timely resolution of the errors on embedded IA platforms. The Intel Embedded Design Center provides qualified developers with web-based access to technical resources. Access Intel Confidential design materials, step-by step guidance, application reference solutions, training, Intel s tool loaner program, and connect with an e-help desk and the embedded community. Design Fast. Design Smart. Get started today. 3

4 Contents Machine Check Architecture... 5 Machine Check Exceptions in Embedded Applications... 5 Potential Causes for Machine Check Exceptions... 6 Impact of MCEs to embedded customers... 6 Examples of MCEs in Embedded Applications... 7 Machine Check Architecture elements... 7 MCA Capability Register... 7 IA32_MCG_CTL MSR... 8 MCA Global Status Register... 8 MCE Bank Registers... 8 MCE Bank Status Registers... 8 MCE Bank Control Registers... 9 MCE Bank Address Registers... 9 MCE Bank MISC Registers MCE Coding Tables Multi-core implications Debug Process Flow Confirm platform is operating within specification Gather MCE error code displayed by OS Identify error frequency Confirm whether the MCE is reproducible on the same platform Collect as much information as possible about configuration Decode and understand MCE message Make sure latest MCU and BIOS updates are in place Research CPU Spec Update for relevant CPU errata Try to reproduce on Customer Reference Board Debug Checklist Related documents Summary

5 Machine Check Architecture IA Machine Check Architecture is an evolving technology adding new features and enhancements with each new processor generation. The common types of errors that are detectable by the CPU include: ECC errors, cache errors, system bus errors, parity errors, etc. As processors become more integrated with new additions of memory and fast I/O, the types of errors the MCA can cover become more diverse and the feature sets get even broader. By using this architecture, the CPU can be configured to generate machine check exceptions (MCEs). Some MCEs are correctable, which means that the hardware can recover from the error and correct them without being reset. Correctable MCEs do not need to generate any interrupts in earlier generations of IA processors. Beginning with the 45nm Intel 64 processor with CPUID signature 06H_1AH, the processor is able to report pertinent information related to corrected machine check errors and send a programmable interrupt to allow the software to respond to the machine check errors. This is known as corrected machine check error interrupt (CMCI). So, this option is available in case users want to take actions on correctable MCEs. Some MCEs are uncorrectable and the system will need to reset to recover itself. In this situation the CPU has concluded that the system is no longer in a safe or reliable operating mode, or the cost of trying to recover from the error (either by hardware or software) is prohibitive. The machine check architecture consists of a set of model-specific registers (MSRs) that set up the machine checking as well as additional banks of MSRs used for the recording of detected hardware errors. Machine check architecture communicates critical hardware errors to the software as well as possibly recovering from catastrophic system failures. This architecture provides error handling features, which contribute to high processor reliability, reliable error containment and recording, serviceability, and error correction without program interruption. Machine Check Exceptions in Embedded Applications The embedded IA products bring a new set of challenges to the process of analyzing and debugging machine check exceptions. These products are often deployed in large volumes in the field and it may be hard to retrieve a faulty 5

6 unit in a timely fashion in order to debug. The parts may be operating in all kinds of environments including extreme temperatures or high altitudes, which add to the complexity of trouble-shooting and the task of eliminating suspects. The embedded designs tend to have diverse characteristics compared to desktop/laptop designs. They often run non-standard applications, or have unique form factors, and have longer life cycles. Each product often comes with its own unique set of components (battery-less design, or lack of some components such as graphics or video), features, customized BIOS, OS (embedded real-time OS such as VxWorks, or some in-house solution). As a result, there may not be a standard reference platform to compare with. Each machine check exception situation may need to be treated quite differently throughout the debug process. Potential Causes for Machine Check Exceptions MCEs are difficult to debug mostly because of the large number of potential causes. Some of the potential factors are: Violations to board design guidelines. For example, routing traces over power and ground planes may cause unwanted noise and inadequate signal spacing may cause signal integrity issues. Operating the processor out of specification. Examples include overclocking of the CPU and front-side bus speeds. The behavior of the system cannot be predicted when the processor operates out of specification. Environmental factors, such as: alpha particles or cosmic ray hits, extremely hot, and cold temperatures. Improperly fitted heat sinks or fans and incorrect hardware installation. Missing proper microcode updates that could contain fixes for known processor errata. BIOS setup issue or OS issue may cause MCE handling scheme to behave differently. Faulty components, such as: add-in cards, DIMMs, etc., can also cause system errors that may eventually lead to a MCE. Impact of MCEs to embedded customers The impact of can be quite diverse but of significance especially when critical applications are running that require extreme accuracy, reliability, and are time sensitive. Some examples of embedded market segments where MCEs would be of high impact are industrial controls, financial markets, medical products, aviation, and defense to name a few. Many of these segments utilize real-time operating systems for their applications, which may not have the flexibility of recovering from critical hardware errors as compared to the 6

7 mobile and desktop markets. Several of the embedded applications are also required to operate non-stop for 7-10 years with extremely low error rates. Examples of MCEs in Embedded Applications 1. Product X is a customized, small form-factor motherboard. However, the design routes FSB data lines too close to and through ground plane voids near the board s processor heat sink mounting holes. This FSB layout causes FSB data parity errors and results in an MCE event. 2. Product Y receives an MCE event when running unique application software. The customer uses a custom PCIe NIC due to design constraints. During signal integrity analysis by this customer, the PCIe eye diagram is observed to be outside specification. The MCE event is root caused to be related to the custom PCIe NIC card. 3. Product Z is experiencing sporadic MCE events on different systems in the field during the period of a year that are not reproducible. The customer is using a customized RTOS and their own BIOS that is unable to be updated as the systems have been deployed and are currently in use by end customers. This MCE event would be very difficult to debug without a reproducible failing system. Machine Check Architecture elements This section summarizes the key elements provided by IA Machine Check Architecture. For further details on the MCA and MCE registers, refer to Ref [1]. MCA Capability Register IA32_MCG_CAP MSR is a read-only register that provides information on the MCA of the processor. Table 1. IA32_MCG_CAP register 7

8 Some of the bits may not be available in some older generation IA processors. For detailed definitions, refer to Ref [1]. These register fields can tell the OS and MCE handler what capabilities this particular processor has in terms of the MCA. IA32_MCG_CTL MSR It is important to determine if the machine check features are enabled in order for MCEs to be captured. The IA32_MCG_CTL controls the reporting of machine check exceptions. The IA32_MCG_CTL MSR is present if the capability flag, MCG_CTL_P is set in the IA32_MCG_CAP MSR register. If present, writing 1s to this register enables MCE features and writing all 0s disables MCE features. Refer to Ref [1] for more information. MCA Global Status Register This register provides information on the current status of the MCE. It will also determine if the instruction pointer is related to the MCE or if CPU operation can restart from the instruction that was pushed on the stack when the MCE was generated. Table 2 IA32_MCG_STATUS MCE Bank Registers A finer degree of the MCE is controlled and reported by the MCE Bank Registers. Each error-reporting register bank can contain IA32_MCi_CTRL, IA32_MCi_STATUS, IA32_MCi_ADDR, and IA32_MCi_MISC MSRs. Each bank usually has a special focus area in terms of the types of errors it covers. The number of banks depends on the specific processor family. MCE Bank Status Registers Each IA32_MCi_STATUS MSR contains information related to a machine check exception if its VAL (valid) flag is set (see Table 5). Software is responsible for clearing IA32_MCi_STATUS MSRs by explicitly writing 0s to them; writing 1s to them causes a general-protection exception. 8

9 Table 3 IA32_MCi_STATUS register A more detailed description of the MCE Status Registers can be found in the Machine-Check MSRs section in Ref [1]. MCE Bank Control Registers IA32_MCi_CTL MSRs control error reporting for errors produced by a particular hardware unit. Each of the 64 flags represents a potential error. If the bit is implemented on the processor, setting the bit enables the reporting. Table 4 IA32_MCi_CTL registers MCE Bank Address Registers The IA32_MCi_ADDR MSR contains the address of the code or data memory location that produced the machine check exception if the ADDRV flag in the IA32_MCi_STATUS register is set. 9

10 MCE Bank MISC Registers The IA32_MCi_MISC MSR contains additional information describing the machine check exception if the MISCV flag in the IA32_MCi_STATUS register is set. For detailed register definition information when MISCV is valid, refer to Ref [1] and [2]. MCE Coding Tables To determine the type of error being reported the machine check exception handler must read from the MCA error code field [15:0] of the IA32_MCi_STATUS register. There are two types of MCA error codes: simple error codes and compound error codes. Table 5 shows the simple error codes. These codes indicate global error information. Table 5 IA32_MCi_Status [15:0] Simple Error Code Encoding Notes: 1. BINIT# assertion will cause a machine check exception if the processor (or any processor on the same external bus) has BINIT# observation enabled during power-on configuration (hardware strapping) and if machine check exceptions are enabled (by setting CR4.MCE = 1). 2. At least one X must equal one. Internal unclassified errors have not been classified. Table 6 shows the general form of the compound error codes related to the TLBs, memory, caches, bus and interconnect logic, and internal timer. These compound errors also consist of sub-fields that describe the type of access, level in the cache, and type of request. 10

11 Table 6 IA32_MCi_Status [15:0] Compound Error Code Encoding The Interpretation column indicates the name of a compound error, which is constructed by substituting mnemonics for the sub-field names in the curly braces. Table 7 shows the 2-bit transaction type (TT) sub-field. Table 7 Encoding for TT (Transaction Type) Sub-Field Table 8 shows the 2-bit level (LL) sub-field, which indicates the level in memory hierarchy where the error occurred. Table 8 Level Encoding for LL (Memory Hierarchy Level) Sub-Field Table 9 shows the 4-bit request (RRRR) sub-field, which indicates the type of action associated with the error. 11

12 Table 9 Encoding of Request (RRRR) Sub-Field Refer to Section 15.9 of Ref [1] for the other sub-field decoding tables and for more information. Multi-core implications Most MCE registers are core-specific, that is, each core has its own set of control, status, and address registers. However, in newer processor families such as Nehalem, new banks of registers have been added to the architecture to address package-level error information. For example, in Nehalem processor families, bank 0, 1, 6, 7 are per-package and introduced to address QPI, integrated memory and graphics. Banks 2, 3, 4, 5 are more traditional MCE banks addressing per-core level information such as Data Cache, TLB, MLC, LLC etc. See Ref [2] for more information. Debug Process Flow Confirm platform is operating within specification Before in-depth debugging, it is important to make sure the MCE is not caused by something obvious. An important checkpoint will be to make sure the platform is not operating out of specification. For example, if the CPU is operating in a temperature that is outside of the specified operating range, the behavior will be unpredictable. Another example is if the voltage or frequency of the CPU is operating out of the specified range. Gather MCE error code displayed by OS OS MCE handlers typically produce some MCE code and print out the screen messages before system reset. It is important to capture as much information 12

13 as possible by carefully recording the screen message. Turning on debug message levels to get extra system messages and getting screen log messages recorded is also a good idea to help identify what is happening before the MCE event. The details to turn on debug messages are OS specific. For example, Linux has different levels of debug message displays. Identify error frequency Error frequency is an important piece of data but may be hard for a small sample size. The frequency of the MCE may shed some light on what may be causing it. For example, if the frequency of the MCE is relatively high and easily reproducible, it may indicate issues with board designs or system applications. Otherwise, if the frequency is extremely low, it may be related to environmental disturbance. Confirm whether the MCE is reproducible on the same platform It is helpful to confirm whether the issue is reproducible on the same platform. However, in some cases it is possible the error will not occur on the same platform. Collect as much information as possible about configuration Capture all the platform/os/bios information for the failing system, including: CPU stepping/sku info MCU version Chipset stepping/sku info BIOS (vendor name, whether it has the latest MCU and necessary known BIOS fixes) OS information Software applications that are running, the main transactions (I/O, memory) The kind of environment (high altitude, Extreme cold/hot) Decode and understand MCE message Correctly decoding the MCE message is a necessary and important step. An example of how to decode an MCE message follows. 13

14 For example, if the compound error code reported is: MC1_STATUS: 0xf This can be decoded by looking at the MCA error code field bits [15:0], which is 0151 of the above register. Convert this value to binary ( ) and refer to Table 6 to determine the compound error code form. In this case, the form is (000F 0001 RRRR TTLL) and is a cache hierarchy error. Next, the sub-fields can be determined: TT=00, LL=01, RRRR=0101. By using the sub-field Tables 7, 8, 9 and the corresponding interpretation form from Table 6, ({TT}CACHE{LL}_{RRRR}_ERR), the MCE is decoded as an L1 instruction fetch error. This error is an uncorrected error as can be seen by bit 61 being set in the MC1_STATUS register. The messages provided by the MCE error code can be used to understand what may potentially be causing the errors. Refer to this section for some potential common causes. Refer to Appendix E, Interpreting Machine-Check Error Codes in Ref [2] for more information on interpreting the MCA error code, model-specific error code, and other information error code fields. Make sure latest MCU and BIOS updates are in place Ensure the correct MCU and BIOS are in place. Check with BIOS vendors for relevant BIOS updates. Make sure the latest MCU code is being used as each MCU may contain bug fixes or enhancements. Research CPU Spec Update for relevant CPU errata Research the CPU Specification Update for known CPU errata that match the failure symptoms. If the errata calls for suggestions on certain software practices, then these suggestions should be reviewed, considered, and tested. Try to reproduce on Customer Reference Board When issues are highly reproducible on customer platforms, gather a reference data point by testing it on Intel CRBs. If users can re-create the same issue on Intel CRBs, this will potentially eliminate a lot of possibilities and streamline the debug process. When an MCE can be reproduced on an Intel CRB, there are usually two possibilities related to the cause. One is a potential sighting of a possible 14

15 silicon issue, or the SW/OS/BIOS that is running. If this is the case, it is recommended to alert the silicon vendor. If the MCE is reproducible on the CRB and in a common software environment, then it may be easier to engage in a productive trouble-shooting process. Debug Checklist Steps Checklist Item Outcome 1 Confirm platform is operating within specification 2 Gather MCE error code 3 Identify error frequency 4 Confirm if MCE is reproducible on the same platform 5 Collect as much information as possible about configuration (OS, BIOS version, software applications that are running) 6 Understand the MCE code 7 Make sure latest MCU & BIOS updates are in place 8 Research CPU spec update for any relevant CPU errata 9 Try to reproduce on customer reference board Related documents Ref. # Document Title Document Number/Location [1] Intel 64 and IA-32 Architectures Software Developer s Manual, Volume 3A: System Programming Guide [2] Intel 64 and IA-32 Architectures Software Developer s Manual Volume 3B: System Programming Guide Summary This application note gives an overview of machine check architecture and its purpose in detecting and reporting system errors. This architecture provides an opportunity to capture a group of error situations visible to the CPU at the point of failure. Newer additions of the MCA also make it possible to wire 15

16 interrupts to correctable MCEs, in case users are interested in checking these events as well. This document described the importance of machine check architecture to embedded products given the valuable information captured in MCE registers, which are often the only clue to what has happened. Debugging machine check exceptions on any system is a challenging task due to the number of suspects that may be involved. Embedded systems add a new dimension to the difficulty due to their vastly diverse system configuration, environments, and usage models. This document provides a quick review of machine check architecture and its key elements for debugging. It also provides recommendations on how to debug MCEs in embedded systems and provides a sample approach to help system developers debug such issues. As each failure event is rather unique, every error situation will need to be approached differently. Nevertheless, this step by step guide provides a list of items that may be helpful to this debug process. The Intel Embedded Design Center provides qualified developers with webbased access to technical resources. Access Intel Confidential design materials, step-by step guidance, application reference solutions, training, Intel s tool loaner program, and connect with an e-help desk and the embedded community. Design Fast. Design Smart. Get started today. Authors Ashley Montgomery is a Platform Application Engineer with Intel s Embedded and Communications Group. Tian Tian is a Platform Application Engineer with Intel s Embedded and Communications Group. 16

17 INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. This paper is for informational purposes only. THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. Intel disclaims all liability, including liability for infringement of any proprietary rights, relating to use of information in this specification. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted herein. BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside, i960, Intel, the Intel logo, Intel AppUp, Intel Atom, Intel Atom Inside, Intel Core, Intel Inside, the Intel Inside logo, Intel NetBurst, Intel NetMerge, Intel NetStructure, Intel SingleDriver, Intel SpeedStep, Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo, Intel StrataFlash, Intel Viiv, Intel vpro, Intel XScale, InTru, the InTru logo, InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Moblin, Pentium, Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside, vpro Inside, VTune, Xeon, and Xeon Inside are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright 2010 Intel Corporation. All rights reserved. 17