Epicor ERP Performance Tuning Guide

Transcription

1 Epicor ERP Performance Tuning Guide 10

2 Disclaimer This document is for informational purposes only and is subject to change without notice. This document and its contents, including the viewpoints, dates and functional content expressed herein are believed to be accurate as of its date of publication. However, Epicor Software Corporation makes no guarantee, representations or warranties with regard to the enclosed information and specifically disclaims any applicable implied warranties, such as fitness for a particular purpose, merchantability, satisfactory quality or reasonable skill and care. As each user of Epicor software is likely to be unique in their requirements in the use of such software and their business processes, users of this document are always advised to discuss the content of this document with their Epicor account manager. All information contained herein is subject to change without notice and changes to this document since printing and other important information about the software product are made or published in release notes, and you are urged to obtain the current release notes for the software product. We welcome user comments and reserve the right to revise this publication and/or make improvements or changes to the products or programs described in this publication at any time, without notice. The usage of any Epicor software shall be pursuant to an Epicor end user license agreement and the performance of any consulting services by Epicor personnel shall be pursuant to Epicor's standard services terms and conditions. Usage of the solution(s) described in this document with other Epicor software or third party products may require the purchase of licenses for such other products. Where any software is expressed to be compliant with local laws or requirements in this document, such compliance is not a warranty and is based solely on Epicor's current understanding of such laws and requirements. All laws and requirements are subject to varying interpretations as well as to change and accordingly Epicor cannot guarantee that the software will be compliant and up to date with such changes. All statements of platform and product compatibility in this document shall be considered individually in relation to the products referred to in the relevant statement, i.e., where any Epicor software is stated to be compatible with one product and also stated to be compatible with another product, it should not be interpreted that such Epicor software is compatible with both of the products running at the same time on the same platform or environment. Additionally platform or product compatibility may require the application of Epicor or third-party updates, patches and/or service packs and Epicor has no responsibility for compatibility issues which may be caused by updates, patches and/or service packs released by third parties after the date of publication of this document. Epicor is a registered trademark and/or trademark of Epicor Software Corporation in the United States, certain other countries and/or the EU. All other trademarks mentioned are the property of their respective owners. Copyright Epicor Software Corporation All rights reserved. No part of this publication may be reproduced in any form without the prior written consent of Epicor Software Corporation. 10 Revision: June 25, :34 a.m. Total pages: 70 sys.ditaval

3 Performance Tuning Guide Contents Contents Introduction...5 Purpose of this Guide...5 Intended Audience...5 How it is Organized...5 Performance Tuning Resources...7 Hardware Sizing Guide...7 Performance and Diagnostic Tool...7 VMWare Best Practices Guides...8 Common Patterns...9 Frequent User Complaints...9 Everything Is Slow...9 Sometimes Everything Is Slow...9 Specific Program Is Slow...10 Reboot Corrects Poor Performance...10 Unable to Reproduce by Epicor Technical Support...10 Standard Metrics...11 SAN (I/O) Test Utility...13 Run the SQLIO Utility (Standard Metric)...13 Performance and Diagnostic Tool...15 Installation...15 Client Diagnostics...16 Generate Client Logs...16 Analyze Client Logs...17 Fields and Sheets...19 Client Tracing Log Options...20 Scenarios...21 Send Client Logs to Epicor...21 Configuration Check...22 Enter Configuration Check Settings...23 Analyze Configuration (Standard Metric)...24 Network Diagnostics...25 Test Network Performance (Standard Metric)...25 Server Diagnostics...26 Write Permission...26 Server Log Size and Details...27 Generate Server Logs...29 Access Server Logs...31 Analyze Server Logs...32 GetRowsKeepIdleTime Chart (Standard Metric)...34 Fields

4 Contents Performance Tuning Guide Scenarios...38 Additional Diagnostic Resources...39 Locking and Blocking...39 Set up the sp_lock3.sql File...39 Configure the sp_lock3.sql File...40 Create a SQL Job...40 Start the Locks Log...42 Review the Locks Log...42 Locking and Blocking Article...43 Deadlock Tracing...43 Deadlock Graph Setup...43 Activate the SQL Deadlock Graph...44 Stop the SQL Deadlock Graph...44 Load Distribution...46 Application Servers...46 Task Agents...46 Create Task Agent Rules...47 Application Tuning...49 Application Troubleshooting...49 Client Cache...49 Locate the Client Cache...50 Memory Cached Programs...50 Client Customizations...50 Order Entry Performance Tuning...51 MRP Performance Tuning...52 BAQ Performance Tuning...54 BAQ Server Settings...54 BAQ Best Practices...56 SQL Syntax Issues...57 Baseline Performance Tests...58 Test Setup...58 Start the Test...58 Test Procedure...59 Activate the Trace Log...59 Open Sales Order Entry Form...59 Test Form Performance Time...59 Verify and Fix Performance Test...61 Customer Retrieval (Standard Metric)...62 Run Customer Retrieval Test...62 Sales Order Line Entry (Standard Metric)...63 Test Sales Order Detail Line Performance...63 Purchase Order Entry (Standard Metric)...65 Test Purchase Order Detail Line Performance...65 Performance Tuning Strategy

5 Performance Tuning Guide Introduction Introduction Purpose of this Guide The Performance Tuning Guide contains information on how to evaluate the performance of your Epicor ERP application. You can then determine what may be the cause(s) of poor performance and make changes as needed. Use this guide to perform diagnostic tests to compare your system performance against Epicor ERP standard metrics. You can follow these tests for your own evaluation. These tests are also used by Epicor consultants and technical support, so you can review these results with Epicor and develop a performance improvement plan. Important This version of the Performance Tuning Guide for use with Epicor ERP version 10.0 or later. Intended Audience The guide is intended for technical consultants, partners, and system administrators to help ensure the Epicor ERP application performs as expected and to provide guidance on performance areas that should be addressed before contacting Epicor consultants or Epicor Technical Support. Individuals who perform all or some of these tasks will benefit from reviewing the Performance Tuning Guide. How it is Organized This guide explains how you can test the performance of your Epicor application. The following are the main sections of this guide: Performance Tuning Resources - Details the performance tuning resources available for use with Epicor ERP. Review this section to learn more about each primary resource. Common Patterns - Describes the common patterns of poor performance. Each pattern is described along with suggestions for testing and potential solutions. Standard Metrics - Epicor has established benchmark metrics for optimal performance. This section of the guide contains a table that describes each benchmark metric and the tests you run to evaluate it. SAN (I/O) Test Utility - You can use the SQLIO Utility to evaluate various parameters of an input/output (I/O/) disk subsystem. Use this section of the guide to learn how to download and run this key Microsoft utility. Performance and Diagnostic Tool - Documents how you install and use the Performance and Diagnostic Tool, the key program you will use to evaluate the performance of client and server installations. This section documents how you activate the client logs and server logs for the Epicor application and then view them within the Performance and Diagnostic Tool. It also describes the features, sheets, and fields available within this tool. Application Tuning - This section contains some application tips and techniques you can do inside the Epicor application. You should only use the tips and techniques that match your use of the Epicor application. 5

6 Introduction Performance Tuning Guide Baseline Performance Tests - Provides detailed instructions for performing primary tests to measure how well your installed system performs. Epicor recommends you run these tests after the application is setup and configured to verify the Epicor application has optimal performance. You can also run these tests periodically later on to make sure the application performance has not degraded. 6

7 Performance Tuning Guide Performance Tuning Resources Performance Tuning Resources Epicor representatives and customers can leverage some key resources to improve the efficiency of the Epicor application and the systems on which it runs. Hardware Sizing Guide The Hardware Sizing Guide provides a practical approach to estimating the capacity you need for both the Epicor ERP application and the database server. The Hardware Sizing Guide matches the number of users against the database activity that occurs each day. It next matches these two usage variables against a list of potential hardware configurations. You can then review the various hardware and network items you could implement to improve application performance. This guide also estimates the future load that may be required later as well. So even if the current network and hardware configuration is adequate, you can still use this document to determine whether it makes sense to upgrade the system now to prevent performance issues before they occur. This document is intended as a tool to help identify potential system upgrades. Before making any significant changes, Epicor representatives, network consultants, and customers should work together to determine the best outcome. Epicor representatives and network consultants may also have further recommendations specific to a customer organization that cannot be documented in this guide. The Hardware Sizing Guide is available to download from EPICWeb: Performance and Diagnostic Tool Use the Performance and Diagnostic Tool to evaluate how the Epicor ERP application interacts with the system. This important tool can identify changes that can potentially achieve significant performance results. You run the Performance and Diagnostic Tool to analyze Epicor logs. These logs help measure performance on both the client and server installations of the Epicor application. You use this tool to evaluate the following system areas: Client Performance Server Performance Network Execution Times Server Execution Times Client Configurations The Performance and Diagnostic Tool is included with your Epicor ERP application. Review the Performance and Diagnostic Tool section later in this guide for installation and configuration information. 7

8 Performance Tuning Resources Performance Tuning Guide VMWare Best Practices Guides Virtual environments provide flexibility and can scale up or down more easily than physical environments. Through this design approach, you can clone machines and add them to the load balancing pool to accommodate changes in usage patterns. If you are implementing VMWare in your environment, be sure to review the performance tuning documents released by VMWare. These.pdf guides contain the information you need to improve the performance of your virtual environment

9 Performance Tuning Guide Common Patterns Common Patterns This section describes some common patterns for poor performance and the causes for these patterns. If you are experiencing any of these common patterns, be sure to use the Performance and Diagnostic Tool as needed to analyze the situation. This tool is discussed later in this guide. Frequent User Complaints Some users frequently complain about a performance problem. They experience poor performance regularly, but other users at this organization do not experience the same issues. You can turn on the trace log for the whole day on the end-user client machine that has the performance problem. You will most likely need to do this for several days to capture the required data. Be sure to also clear the trace log file each day so you can better evaluate the results. Note if your trace log file expands too quickly, try to save the trace log every two hours and start a new trace log. When the end user notifies you that the Epicor application had poor performance during the current day, you will have the trace data you need. Save the trace log for that day. If you needed to save multiple trace logs for the same day, review all the trace logs. If you are working with Epicor Technical Support, send the trace log (or multiple trace logs) to the consultant or technical support representative helping you with the issue. Everything Is Slow Everything is slow all of the time. The most likely cause behind this pattern is the hardware sizing for the production environment is not correct. Use the Hardware Sizing Guide to determine the recommended hardware; this guide evaluates data transaction traffic, number of users, and the system software to determine the recommended hardware. Sometimes Everything Is Slow Sometimes everything slows down, but other times the performance is fine. This performance pattern could be caused by a number of reasons. Most likely, the slow performance occurs because the Epicor ERP application was experiencing heavy data loading at that time. The other likely possibility is that either a third party ODBC connection locked a record or the Epicor application has some other locking issue. Review the Locking and Blocking and Deadlock Tracing sections later in this guide to identify and correct locking issues. 9

10 Common Patterns Performance Tuning Guide Specific Program Is Slow A specific program is slow. The causes for this pattern are described in the previous Frequent User Complaints section. You will need to turn on the client trace log and have the end user follow the interface or data entry pattern again to recreate the poor performance. You then have a trace log or multiple trace logs to review. Reboot Corrects Poor Performance Everything slows down, but a system reboot restores good performance. This common pattern can be caused by several issues: Typically several large reports or processes were generating at the same time, harming performance. These reports/processes stop when you exit and reboot the Epicor ERP application. Determine which reports and/or processes were running and evaluate whether your network has sufficient resources to handle simultaneously running these report/processes. Was a long generating process, such as MRP, running at the time your users experienced poor performance? If so, try to schedule the process at a time when less demand is being placed against the network. Was a process that requires a lot of resources, such as MRP, started on a different database (Test, Training, Pilot) being still run from the same server as your live database? If this is occurring, consider running this process on a different server. Perhaps one or multiple business activity queries (BAQs) were running simultaneously, causing the slow performance. Even one BAQ can cause a problem if both the SQL Server and the application server are run from the same machine. If this is the case, try to run these BAQs at a time when less demand is being placed against the network. Evaluate whether your network has sufficient resources to handle running these BAQs. Does your system have a memory leak? The client and server machines can then rapidly run out of memory, causing the slowdown. Are your records getting locked? Record locking can be caused either by the Epicor ERP application or by third party applications. Unable to Reproduce by Epicor Technical Support Performance is slow on your server, but the Epicor test systems cannot reproduce the issue. If Epicor Technical Support is unable to recreate the performance issue, consider the following: Evaluate the differences in hardware. Your hardware sizing needs to be evaluated by you and an Epicor consultant or Epicor Technical Support. Different data items are used for testing, and so the performance test is not comparing similar conditions. The poor performance occurred when the system was experiencing heavy data load, so the performance issues were caused by this load. 10

11 Performance Tuning Guide Standard Metrics Standard Metrics Epicor has developed a Standard Metrics table that identifies the benchmarks for optimal performance. Compare these benchmarks against the customer system to locate areas of poor performance. These performance benchmarks can identify key areas of improvement. Once these performance values are determined, customers and Epicor representatives can investigate potential ways these performance values can be tuned to achieve better results. This section first contains the Standard Metrics table and then a second example table that illustrates a specific company's metrics compared against these standard metrics. Epicor Standard Performance Metrics Performance Metrics Storage Disk I/O (SQLIO Test Utility) CPU Speed (Config Check) Epicor Component Impacted Affects the SQL Server database Data drive. Performance of VMware machines as they are stored on the shared NAS. Tests the general speed of the application server. Standard Metrics (Expected Result) Throughput of 300 MB/Sec, with less than 5ms or better latency 400 milliseconds GetRowsKeepIdleTime (GetRowsKeepIdleTime Chart) Network Test (Network Diagnostics) An overall system performance test. How fast does this common, static API method run over this infrastructure? This value can be compared across many machines. Impact of the network on the client experience. 15 milliseconds average, low performance variability throughout the day Server (Blue) < 0.5 seconds, Network (Green) < 0.4 seconds Configuration Check (Config Check) Customer Retrieval Test (Baseline Test) Reviews the performance of the Epicor application server. This test measures database retrieval time by selecting and paging through customers. The Epicor application server passes the configuration tests Observed time to move through customers < 1 Second Sales Order Test (Baseline Test) Purchase Order Test (Baseline Test) This tests the performance of Sales Order Entry, a commonly used, important business function. This test does not check customizations or BPMs. This tests the performance of Purchase Order Entry, a commonly used, important business function. This test does not check customizations or BPMs. Add 20 lines Sales Order < 72 seconds Add 20 lines Purchase Order < 51 seconds The following sections describe how you test for these standard metrics. 11

12 Standard Metrics Performance Tuning Guide Example CompanyA's Metrics Standard Test Expected Result CompanyA Production CompanyA Physical Storage Disk I/O Throughput > 300 Mb/Sec. 26 Mb/Sec N/A Storage Disk I/O Latency < 5 Milliseconds 37 Milliseconds* N/A CPU Speed < 4000 Milliseconds 3500 Milliseconds 2713 Milliseconds (In CompanyA s instance, VMs currently running on Dell R610, Intel X GHz were tested.) GetRowsKeepIdleTime < 30 Milliseconds 80 Milliseconds* 20 Milliseconds Network Test - Server < 500 Milliseconds N/A N/A Network Test - Client < 400 Milliseconds N/A N/A Configuration Check Pass Pass Pass Customer Retrieval Baseline Test <1 Second 1.2 Seconds.8 Seconds Sales Order Baseline Test < 72 Seconds 75 Seconds 64 Seconds Purchase Order Baseline Test < 51 Seconds 49 Seconds 43 Seconds * The bold values indicate a performance metric value that exceeds the value of an Epicor standard metric. 12

13 Performance Tuning Guide SAN (I/O) Test Utility SAN (I/O) Test Utility You can evaluate the efficiency of a Storage Area Network (SAN) by running a Microsoft subsystem benchmark utility called SQLIO. Through this utility, you test your system against the Storage Disk I/O standard metric. You use this utility to test various parameters of an input/output (I/O) disk subsystem like bandwidth (megabytes/second), latency (milliseconds), block size performance (64KB), file size, and I/O type (read/write and sequential against random). These parameters have a huge impact on I/O performance. You download the SQLIO utility from Microsoft. After you install SQLIO, you define which drive the utility will test. You then run a series of three tests through this utility. Run the SQLIO Utility (Standard Metric) The following steps describe how you install, set up, and test I/O disk subsystem performance. 1. Download the SQLIO utility from Microsoft. Use your internet browser to navigate to this site: 2. Install the utility. The default path is: C:\Program Files\SQLIO\ 3. Once the SQLIO utility is installed, navigate to the install path. Notice that several files installed with this utility. Important Review the readme and USINGSQLIO text files to learn how to set up and use the tool. Before you run this utility, be sure to read this documentation. 4. Right-click on the param.txt file and open it in Notepad. 5. Now define the specific drives against which you want to test the I/O performance. Example C: \testfile.dat 2 0x Save the file and close the text editor. 7. Launch the Windows Command Prompt by clicking Start > Run > CMD. The command window displays. 8. Change the directory to the SQLIO folder. Example C:\users\admin>CD C:\Program Files\SQLIO 9. Now run three tests using the parameters specified below. a. Test 1: Database (.mdf) 13

14 SAN (I/O) Test Utility Performance Tuning Guide This test evaluates 64K random writes. In the command prompt, enter the following phrase: sqlio -kw -t8 s900 -o8 -frandom -b64 -BH -LS -Fparam.txt b. b. Test 2: Transaction Log (.ldf) This test evaluates 64K sequential writes. In the command prompt, enter the following phrase: sqlio -kw -t2 s900 -dm -o8 -fsequential -b64 -BH -LS Fparam.txt c. Test 3: TempDB This test evaluates 64K random writes. In the command prompt, enter the following phrase: sqlio -kw -t8 s900 -o8 -frandom -b64 -BH -LS -Fparam.txt Use the results of each test to evaluate how your disk I/O subsystem works in comparison to similar subsystems. 14

15 Performance Tuning Guide Performance and Diagnostic Tool Performance and Diagnostic Tool Use the Performance and Diagnostic Tool to evaluate how the Epicor application performs through client, server, and network tests. You also use this tool to evaluate the system configuration and download additional diagnostic resources for use with SQL Profiler. If you are experiencing performance issues, you should first contact either your Epicor consultant or Epicor Technical Support. If the performance issue cannot be resolved through this initial contact, the technical support representative or the consultant may recommend you use the Performance and Diagnostic Tool. This tool captures performance information, and you can organize this information to receive meaningful metrics that relate to the performance of your Epicor ERP application. You can also export these results to Microsoft Excel for additional review and analysis. Through the Performance and Diagnostic Tool, you can evaluate: The performance of one client versus another client on the same system. The performance of business object methods on both the client and the server. Overall performance of the server and the network. Performance of business objects in one system against the same business objects on other systems. Performance of customizations, personalizations, Business Process Management (BPM) methods, and business activity querys (BAQs). The configuration of the Epicor ERP application. Important The Performance and Diagnostic Tool released with Epicor ERP version 10 is only compatible with the version or higher. If you need to evaluate the performance of Epicor ERP 9.05 or earlier, download the Performance and Diagnostic Tool released with the 9.05 version. Installation Follow these steps to install the Performance and Diagnostic Tool. Tool Requirements:.NET 4.5 Runtime Windows 2008 R2 or Higher Tip The Performance and Diagnostic Tool can be installed on any desktop or laptop machine where you normally install the Epicor client. When using the tool, you will be able to connect to application servers on other machines. 1. From Windows Explorer, navigate to the C:\Epicor installation folder. Expand the folder for your current version and then open the Utilities\ICE3 Performance Diagnostic Tool folder. Example C:\Epicor\ERP10\ERP \Utilities\ICE3 Performance Diagnostic Tool 2. Double-click the Performance and Diagnostics Tool icon. 3. The Welcome window appears. Click Next. 15

16 Performance and Diagnostic Tool Performance Tuning Guide 4. Select the installation folder you want and click Next. 5. Confirm the installation by clicking Next again. The Performance and Diagnostic Tool installs in the location you specified. 6. Click Close. 7. You can now launch the Performance and Diagnostic Tool. Depending on your operating system, you launch this tool in different ways: a. If you are on Windows SQL Server 2008 R2 or Windows 7, click Start > All Programs > Epicor Software > Epicor Administrative Tools > Epicor Performance and Diagnostic Tool. b. If you are on Windows SQL Server 2012 or Windows 8, press the <Windows> + F button to display the Charms bar; from the Apps screen, select Epicor Performance and Diagnostic Tool. Client Diagnostics You analyze the performance of client installations through client tracing logs. This section of the guide describes how you set up these logs and analyze their results in the Performance and Diagnostic Tool. Generate Client Logs You generate the client logs by activating the trace log in the Epicor application. To generate a client trace log: 1. Log onto a client installation of the Epicor application. 2. Click on the Settings button; from the General Options list, select Tracing Options. The Tracing Options window displays. 3. Select the Enable Trace Logging check box. 4. Click OK. 5. Now return to the Home screen and launch some programs whose performance you want to measure. Add new records, modify records, run processes, and so on. The Trace Log records gathers performance data about each action you take within the Epicor application. 6. Return to the Home window; click the Settings button and select Tracing Options. The Tracing Options window displays. 7. Click the Write button. The tracing details are copied to the log. 8. Copy the directory from the Current Log File field. 9. Launch the Performance and Diagnostic Tool. 10. Click on the Client Trace Analysis option on the Plug-Ins pane. The Client Trace Analysis interface displays. 16

17 Performance Tuning Guide Performance and Diagnostic Tool 11. For the Client Trace File Path, either paste the path you copied from the Current Log File field or click the Browse button to find and select client log file. 12. Optionally, select the Exclude System Calls check box to prevent system method calls from displaying with diagnostic results. The Performance and Diagnostic Tool will then not display values from the GetRowsKeepIdleTime business object method. This method runs in the background while a current form is active, but it does not indicate how long it takes other methods like Update, GetNew, and GetList to run. These method calls are generated by user activity. However if you are testing how the client interacts with the server when no user activity is taking place, you will want to review the calls from the GetRowsKeepIdleTime method. In this case, do not select this option so you can displays these calls in the client results. 13. Notice you can also change the Log File Date Format to select a date format used in a different locality or enter your own Custom Format. 14. Click on the Generate Diagnostics button. You can now analyze the trace log data within the Results and Summary sheets. Analyze Client Logs Use the features on the Performance and Diagnostic Tool to review the client results recorded on the trace log. This topic explores the features available for analyzing the client logs. The Results grid displays the main details gathered from each trace packet, while the Summary grid displays the performance times required for each business object to run. To analyze a client trace log: 1. Click the Results tab. The Results grid displays. 2. You can filter the log results based on values that populate each column. To do this, click on the Funnel icon that displays on each column header: 17

18 Performance and Diagnostic Tool Performance Tuning Guide 3. When you click on an icon, a drop-down list displays. Each drop-down list contains a series of default filter options followed by specific values from the trace log: All - The default filter, this option displays all the values for the column from the client tracing log. Custom - Select this option to launch the Enter filter criteria for Type window. Use this window to define both an Operator and an Operand to filter the results. You can define as many filter conditions as you need, creating And/Or relationships between each custom filter. Blank - Use this option to only display any empty rows available on the client trace log. Nonblanks - Select this option to display all the populated rows on the selected client trace log. Column Value - You can filter the results by selecting a specific value available in the column. 4. To help organize the results, use the Group By feature on the grid. Click and drag a column header (for example, Object Name) to the Drag a column header here to group by that column area. You can drag multiple column headers to further structure the results as you need. Example If you group by using the Business Object column, the grid displays all the business objects recorded on the client trace in alphabetical order. 5. You can review the following items on each business method call: Type Object Name Method Name Start Date End Date Execution Time File Name Server Execution Time Network Transport Time Appserver Thread 6. Click on the Summary tab. Each business object and method displays on this grid. 7. The Trace Date/Time and File Name fields display the selected trace log as well as the date and time it was run. 8. In the Report Monitor section, notice the GetRowsKeepIdleTime method. This business object runs in the background while a current form is active. Tip If you shut off the System Monitor, the tracing log did not record time against the GetRowsKeepIdleTime method. 9. The All grid displays the summaries of the method calls recorded on the tracing log. 10. The Calls column indicates how many times the method sent a call to the server. 18

19 Performance Tuning Guide Performance and Diagnostic Tool 11. The execution times, in milliseconds, for each call are calculated and display in the accompanying columns. Additional information displays that identifies the object information: Average Longest Least Diagnostics ID Type Object Name Total 12. Notice you can view the results in a Microsoft Excel file. To do this, click the Export to Excel button. 13. Save the.xls file to directory you can access. You can now launch the.xls spreadsheet and review it in Microsoft Excel. 14. To remove the generated results, click the Clear Results button. Fields and Sheets This topic documents the fields and sheets available for analyzing the client log files. Some fields on the interface have a context menu, which is indicated by a triangle in the upper right corner of the field. To open the context menu, right-click on the field. Client Trace Path Displays the location of your client trace logs. Browse to the file you wish to view within the Performance and Diagnostic Tool. Clear Results Click the Clear Results button to remove data from the Results and Summary sheets. You can also continue generating data from different trace logs without clearing the data. The Summary sheet then contains results from each trace log generation. Custom Format If the date format you need is not available on the Log File Date Format drop-down list, enter the Language ID for the language you use in this field. The date format linked to this language loads into the Performance and Diagnostic Tool. Exclude System Calls Select this check box to prevent system method calls from displaying with the diagnostic results. These system method calls do not relate to the calls generated by application data. Export to Excel Click the Export to Excel button to export results to Microsoft Excel for further manipulation and analysis. The exported data is based on which grid is currently visible. For example, if the Results grid is visible, the data on this grid is exported. 19

20 Performance and Diagnostic Tool Performance Tuning Guide Generate Diagnostics Click the Generate Diagnostics button to generate the log file results and display these results in the Summary and Results grids. Log File Date Format Select the date format that matches the Language identifier you use when you generate client trace log files. If your format is not available from this drop-down list, enter the format you need within the Custom Format field. Results Sheet The Results sheet contains all calls made from this client for the duration of the current trace log. This grid supports Group By functionality, so you can organize the results by Object and/or Method name to view execution times for all method calls. Summary Sheet The Summary sheet summarizes the results for each BusinessObject.Method. This sheet displays the number of calls and the Average/Least/Longest duration of these method calls. The duration is measured in milliseconds. You can also group the summary data by Diagnostic ID, Type, Object Name, and Total. Likewise you can sort the results by selecting the column that contains the most useful information to determine the type of calls made most frequently by this client -- or which method calls consume the most time. Client Tracing Log Options The Tracing Options Form has a number of Dataset Options you can select to define what data is tracked by the client tracing log. These check boxes all display in the Dataset Options section on the form. Include Server Trace Activate this option when you want to track the client's interaction with the server. This creates a <servertrace> node within trace packets (<tracepacket>) in the client tracing log. Use the database activity gathered in this section to review how the client installation may be affecting the performance of the server. Track Changes Only Select this check box to cause the tracing log to only record dataset values and records which were modified. Write Call Context Dataset If you wish to review the performance of BPM methods and customizations, select the Write Call Context Dataset check box. The Call Context Dataset initializes when a user activates a program (UIApp) that either launches a customized form or a BPM directive. As long as the program is active, method calls are sent to the Call Context Dataset. Write Full DataSet Select this check box to view the tables and columns that receive data from the called methods. Use this option to find out what specific tables and columns are used in the dataset transactions with the server. 20

21 Performance Tuning Guide Performance and Diagnostic Tool Write Response Data When a method call has a <returntype> other than void, selecting this check box causes the dataset returned from the server to display on the tracing log. Numerous method calls occur where the data is passed down, modified, not written to the database, and then returned to the client. Selecting this check box places these hidden calls on the trace log. Examples of these methods include Credit Checking, Part Verification and Pricing, and GetNewXXX (where XXX is the name of a record). Tip You typically select this option when you are developing a Service Connect workflow and need to see when a non-obvious value is set by a method call. The tracing log displays Before and After images of the dataset. Scenarios This topic contains examples of how you can measure client performance through different methods. Comparison of client performance between locations and/or client systems. You may have specific users reporting performance issues from their systems. By running client diagnostics from multiple desktops/laptops that perform similar Epicor functions, you can determine differences in client execution times for the same method calls. This may provide you with some things to consider with your network, especially when comparing LAN users against WAN users. However CPU differences will probably not be determined, because the tool is capturing execution time of the method calls, not the time taken to open forms or render the data within these forms. Identification of business object performance issues. You may suspect a business object performs poorly under certain conditions. For example, you notice that transaction entry slows down as the volume of data increases. By monitoring execution time of method calls during entry, you may be able to measure the effect. Identification of performance over time. By analyzing the execution times of similar method calls throughout the day, you should be able to identify slow periods. These slow periods may be related to network overload or server activity during those periods. Identification of Epicor usage from a client. You can analyze the log from a client to determine what activities that client engages against the Epicor application. This method also measures the frequency of calls to business object methods from that client. Identification of customization issues. Client customizations may cause unexpected or excessive calls to business object methods. For example, a poor customization could cause multiple calls to methods not made by the base application. Compare the trace logs that track the same functions with and then without client customizations; this method may identify performance issues. Send Client Logs to Epicor If you are working with an Epicor consultant or Epicor Technical Support to improve performance, follow these steps to send client trace logs and server logs to a support representative. 1. During a period of peak activity on your network, launch the Epicor application on the client workstation. 21

22 Performance and Diagnostic Tool Performance Tuning Guide 2. Click on the Settings tile; from the General Options list, select Tracing Options... The Tracing Options window displays. 3. Select the Enable Trace Logging check box. 4. Now select the Track Changes Only check box. Only update, new, delete, and other method calls that change the database are included on the trace log. 5. Select the Include Server Trace check box. This option causes calls from the client to the server to be included in the trace log. 6. Click the Clear Log button. 7. Now launch the program that has poor performance, following the interface movement pattern and data entry pattern that demonstrates the issue. Tip Typically you should have the end user who reported the issue follow the pattern that causes the poor performance. The data the user enters is also very important. The performance issue can be triggered by many factors -- the data the end user enters, the options the end user selects, the navigation path the end user follows, and so on. 8. Return to the Tracing Options Form. 9. Click the Write button. 10. Save the log file to a location on your C:\ drive. 11. Now perform a second test during a period of slow activity on your network (after hours or on a weekend). Launch the Epicor ERP application from the server machine. The reason you launch the Epicor ERP application during this time and in this environment is to eliminate load condition and network traffic between the client and the application server. Even if you know the network is not the cause of the poor performance, you must record these performance times so the Epicor consultant or Epicor Technical Support can compare the performance under these different conditions. 12. Repeat steps the files to your Epicor consultant or Epicor Technical Support. Configuration Check The Performance and Diagnostic Tool contains a utility to check the configuration of the application server. Use this Config Check option to see what issues and potential issues you may have with the application server configuration. This feature checks a number of configuration items, including the CPU Speed and Configuration Check standard metrics. After the Performance and Diagnostic Tool analyzes the configuration, this feature displays recommended actions you can follow to fix various issues. 22

23 Performance Tuning Guide Performance and Diagnostic Tool Enter Configuration Check Settings You first need to set up the tool so it connects to the application server for the Epicor instance you wish to test. 1. Launch the Performance and Diagnostic Tool. 2. Click Settings > Options. The Options window appears. 3. If you access the application server through an http link, select the Connect using Http check box. You then enter the link in the field. 4. By default you enter the net.tcp// connection. Tip Get the correct URL that points to the application server from the.sysconfig file that runs this instance of the Epicor application. You can find this value in <appsettings><appserverurl> configuration setting. In your Epicor application folder, the.sysconfig files are located in the client\config folder. 5. Use the Connection Method drop-down list to indicate how this application server checks for authentication certificates through Internet Information Services (IIS). When a user logs into the application, the selected method checks whether the user can access the Epicor application. Available options: UsernameWindowsChannel -- Authenticate using an Epicor Username and Password. Windows checks for existing Epicor user accounts to authenticate logins. When you select this option, you need to enter the User ID and Password for the account. You can select this option for both smart client and Epicor Web Access (EWA) installations. UsernameSSLChannel -- Use this option to authenticate using a Secure Sockets Layer (SSL) X509 certificate. Use this method for application servers that handle installations where users reside in different 23

24 Performance and Diagnostic Tool Performance Tuning Guide domains. By using an SSL certificate, users from these different domains can log into the Epicor application. You can select this option for both smart client and Epicor Web Access (EWA) installations. Windows Authentication -- Select this option to authenticate using a Windows Username and Password. You can select this method for application servers that handle client installations where users access the application through the same domain. Any user who has a Windows Username and Password within this domain can successfully log into the Epicor application. You can only select this option on smart client installations. Tip You can also find the Connection Method in the.sysconfig file. Locate the <EndpointBindi ng> value to see the method used the application server. 6. Enter the User Id and Password for the Epicor user account used to access the application server. 7. Next enter the Client Directory for the folder that contains the client installation for the Epicor application. You can enter this path directly or click the Browse (...) button to find and select it. 8. If you need to validate this connection through a certificate, select the Requires Certificate check box. The accompanying directory path field activates; enter the location for the certificate in this field. 9. Click OK. Analyze Configuration (Standard Metric) The Config Check sheet contains tools for analyzing your application server configuration settings. 1. Click on the Config Check option on the Plug-Ins pane. The Check Configuration interface displays. 2. Click the Check Configuration button. The Performance and Diagnostic Tool evaluates the configuration settings and the system. These processes may take some time to run. When complete, the Config Check Summary sheet displays a list of rules run against which the application server was checked. 3. The Result column displays the generated evaluation of each rule against your application server configuration. Available results include FAIL, WARNING, EXISTS, INFO, Not Available, and PASS. 24

25 Performance Tuning Guide Performance and Diagnostic Tool 4. Depending on the results, different instructional text displays in the Action Required column. INFO - Displays some key information you should review to make sure your system is set up correctly. EXISTS Notifies you that various items, like customizations and BPM directives, are active in the current system. These items should be evaluated for performance. PASS The configuration met or surpassed these rule requirements. No further action is needed. WARNING Alerts you that potential performance issues may occur. Review these items to see if further changes are needed. FAIL The application server configuration did not meet the rule requirements. The ActionRequired column displays a recommended action you can do. Not Available - Displays any item that was not available to test. You may need to correct some setup configuration items and re-run the configuration check. 5. The Config Check Details sheet displays the various rule keys run to evaluate each configuration rule. Expand one of the rules to see the specific calls. 6. Notice you can view the results in a Microsoft Excel file. To do this, click the Export to Excel button. The information from all the Configuration Check sheets display in the exported spreadsheet. 7. Save the.xls file to directory you can access. You can now launch the.xls spreadsheet and review it in Microsoft Excel. 8. To remove the generated results, click the Clear Results button. Network Diagnostics You can use the Performance and Diagnostic Tool to verify the baseline network and server performance are running at optimal levels. Use this feature to evaluate the Network Test standard metric. To do this, run multiple tests to gauge the overall performance of your network, and compare these results against the network standard metric. Just like the Configuration Check, you need to update the fields on the Settings > Options window so the tool connects to the application server. Test Network Performance (Standard Metric) Run this test to verify that the baseline network and server performance are running at optimal levels. 1. Click on the Network Diagnostics option on the Plug-Ins pane. The Network Diagnostics interface displays. 2. Click the Run Test button three times. Three bar graphs appear on this sheet. You can click on each bar graph to view the specific details of each network test. 25

26 Performance and Diagnostic Tool Performance Tuning Guide Expected Results: Server Time (blue/lower bar) < 0.5 Seconds Network Time (green/upper bar) < 0.4 Seconds over a LAN Network Time (green/upper bar) < 7 Seconds without compression or < 1.5 seconds with compression over a WAN Server Diagnostics You analyze the performance of server installations through server tracing logs. This section of the guide describes how you set up these logs and analyze the results. Write Permission Be sure the user account you use to access the server machine has write permission to the selected folder. Do the following to verify you have access. 1. Launch Internet Information Servers (IIS) Manager. Depending on your operating system, you launch this tool in different ways: a. If you are on Windows SQL Server 2008 R2 or Windows 7, click Start. In the Search field, enter Internet Information Services; when this program appears in the results, select it. b. If you are on Windows SQL Server 2012 or Windows 8, press the <Windows> + F button to display the Charms bar; from the Apps screen, select Internet Information Services (IIS) Manager. 26

27 Performance Tuning Guide Performance and Diagnostic Tool The Internet Information Services (IIS) window displays. 2. Expand the tree view to select the Application Pools pane. The Application Pools pane displays. 3. Select the EpicorERP10 application pool. 4. Now from the Actions pane, select the Advanced Settings... option. The Advanced Settings window displays. 5. Locate the Identity setting and click the Browse (...) button. 6. Select the Built-In account radio button. 7. From the drop-down list, select the Application Pool Identity option. 8. Click OK twice to close out of the Advanced Settings window. 9. Close the Internet Information Services (IIS) Manager. Server Log Size and Details As you evaluate performance issues, you will most likely create multiple application server logs. How many log files you save, the details you include, and how large you let them grow depends on your preferences and testing requirements. You should be able to save as many application server logs as you need. The number of server logs you create and the server details you wish to include depends on what aspect of the Epicor application you are evaluating. If you are evaluating system performance on a daily basis, you will most likely create at least one new application server log each day. However if you are evaluating when peak server activity occurs, you will most likely record a week or several weeks' worth of activity. By default each application server log file is limited to a size you specify. When one file reaches this size limit, the Epicor Administration Console creates a new log file. It uses the original file name as a prefix and then adds a date time stamp (the UTC date and concatenated time). 27

28 Performance and Diagnostic Tool Performance Tuning Guide To avoid running into disk space issues, you can control the number of log backups you want to maintain. Use the logcount attribute within the server log to specify how many backups you want to keep. By using the default option logcount="0", an unlimited number of backups is created. Each log also includes a series of server detail values. Depending on the purpose for creating the log, you may not need to track all of these details. The <appserver>.config file includes a setting that prevents these details from recording in the log. If you wish to change this default size, do the following: 1. Launch Windows Explorer. 2. Navigate to the web.config file; this file is located in the server folder in your Epicor installation. For example: C:\Epicor\ERP10\ERP \Server 3. Open this file in Notepad or a similar text editor. 4. Locate the <configuration> section. 5. For the <Trace> setting, locate the traceconfigurationfile attribute and review the name of the application server configuration file you use. Example <Trace loglocation="" traceconfigurationfile="appserver.config" ></trace> 6. Save the file. After you save this file, the changes you entered for the traceconfigurationfile are used immediately by the system and do not require you restart IIS. 7. Now open the <appserver>.config file in Notepad. 8. Locate the <Trace> setting. Example <Trace disabled="false" verbose="true" maxlogfilesize="10mb" lo gcount="0" includeserverdetails="true" loglocation="file://c:/programda ta/epicor/serverlog.txt"> 9. Update the maxlogfilesize value to the log size file you need. You must enter a whole number followed by KB, MB, or GB (these values are not case-sensitive). For example, you can enter "16KB", "300mb", "3GB", and so on. You can also enter "none", "unlimited", or "-1" to indicate the server log should not have a maximum file size. The default size is 10MB; the minimum log file size is 4KB. When you have an application server log run for a long period of time, the size of the log file can grow very large. Usually this will not be a problem, as the Performance and Diagnostic Tool can evaluate most logs 500 MB in size or less. However if you want to view the log in Notepad or a similar text editor, you should keep each log between 5-10 MB in size. Notepad can easily open log files this size, and they typically contain about the right amount of data for you to review. When the log file size reaches this maximum limit, the Epicor Administration Console creates a new log file with the UTC date and time values placed on the end of the filename. Because of this, the active log files all use the same file name. 10. To prevent the Machine, Process, and Thread ID server details from appearing in the log, locate the <includeserverdetails> setting. Change this setting to "False" and these items will not populate in the log results. 28

29 Performance Tuning Guide Performance and Diagnostic Tool Tip These server details may be important if you are evaluating the performance of a server farm. For example, you can use the Machine values to review the performance of a specific server. However if you are not working in a server farm environment, suppress these details. 11. Save the <appserver>.config file. 12. Exit Notepad. Note that if you have multiple application servers, you need to change all the web.config files so the traceconfigurationfile attribute in each file points to the correct <appserver>.config log for each application server. Likewise you need to adjust the settings for each <appserver>.config file so you generate the results you want. Generate Server Logs To generate an application server log, you activate logging in the Epicor Administration Console. You also define what specific information the log records. 1. You launch the Epicor Administration Console from your server machine. Depending on your operating system, you launch this tool in different ways: a. If you are on Windows SQL Server 2008 R2, click Start > All Programs > Epicor Software > Epicor Administrative Tools > Epicor Administration Console. b. If you are on Windows SQL Server 2012, press the <Windows> + F button to display the Charms bar; from the Apps screen, select Epicor Administration Console. The Epicor Administration Console displays. 2. From the tree view, expand the Server Management node and Epicor Server node. 3. Select the application server you need to monitor. 4. Now from either the Action menu or the Actions pane, select Application Server Settings. The Application Server Settings window displays. 29

30 Performance and Diagnostic Tool Performance Tuning Guide 5. Select the Trace Log Enabled check box. 6. Now use the Log File Location field to indicate where you want the application server log to generate. Either enter this path directly or click the Browse (...) button to find and select this directory path. 7. Next define what Standard Logging information you want the application server log to record. If you only are tracking a specific database activity, use one of the more specific options. Server logs are easier to review if you only capture the types of database activity you require. Available options: a. Verbose Logging - The default option, select this check box when you want the log to record all calls, triggers, and exception messages sent to the application server. b. Trigger Hits - When a record is sent to the database to be added, updated, or deleted (Write/Update/Delete), the framework creates an event that SQL Server can intercept the call to perform table specific logic. After this event is processed, the record is sent to the database. Select this check box to record these trigger events in the server log. c. BPM Logging - Select this check box to record Business Process Management (BPM) method calls. Each time user activity activates a BPM directive, the application server log records the business object method that was called and how long this call took to complete. d. BAQ Logging - Select this check box to record Business Activity Query (BAQ) database calls. Each time user activity activates a BAQ, the application server log records which query was called and how long it took this BAQ to gather the data results. e. Detailed Exceptions - Indicates you want to record the complete details of each exception message. The full stack trace of the exception is included in the server log. You then see which items in your Epicor application were affected by the exception. 30

31 Performance Tuning Guide Performance and Diagnostic Tool f. ERP DB Hits - Activate this check box to track how the Epicor application interacts with the database. You can review each database hit as well as how long it took each hit to complete. g. Data Logging - This option will be available in a future version. 8. Indicate which Advanced Logging information you want to include on the application server log. These options record calls from the overall system server, and may impact performance while active. Available options: a. System DB Hits - Select this check box to record all the hits the database receives from SQL Server. Use these values to determine the performance of SQL Server. b. System Table Methods - Activate this check box to track the method calls being placed against the system tables. 9. When you finish making your selections, click Apply and then OK. 10. The Server Manager dialog box displays, asking if you want these log settings to activate. If this is a good time to begin generating results in the application server log, click Yes. The application server is now active and recording the selected trace call options. The server log should run for a period of time, such as a single day or a couple days. When you are ready to review the results, you display the application server log in the Performance and Diagnostic Tool. Important After you gather the system information you need, be sure to return to the Epicor Administration Console and de-activate your log setting options. This reduces unnecessary calls to the server and improves performance. Access Server Logs 1. Click on the Server Diagnostics option on the Plug-Ins pane. 2. Click the Add Files button and browse to the location of the server log file. 3. Navigate to the directory location that contains the server log. Example C:\Epicor\Logs\server.log Tip You can also click the Down Arrow next to the Add Files button to select server log files from multiple server locations. You may need to do this on load balanced systems where the application is located across multiple servers. 31

32 Performance and Diagnostic Tool Performance Tuning Guide 4. If you need, enter a value in the Millisecond Threshold field. Any calls that took longer than the millisecond value are designated as Exceeds Threshold in the server log results. 5. Use the Epicor Version drop-down list to select the application version run with the selected server log(s). The default version is Select the Ignore GetRowsKeepIdelTime check box to filter any system calls sent to the server. The Performance and Diagnostic Tool will not display values from the GetRowsKeepIdleTime business object method. This method runs in the background while a current form is active, but it does not indicate how long it takes other methods like Update, GetNew, and GetList to run. These method calls are generated by user activity. 7. Now click the Generate Diagnostics button. The Performance and Diagnostic Tool analyzes the server log. You review this data on the Results, Summary Analysis, Running Processes, and Log Errors sheets. Analyze Server Logs Use the features on the Performance and Diagnostic Tool to review the results recorded on the selected server log. This topic explores the features available for analyzing the server logs. 1. Click on the Results tab to review the server calls that were larger than the Millisecond Threshold you entered. The Results grid displays. 2. You can filter the log results based on values that populate each column. To do this, click on the Funnel icon that displays on each column header: 32

33 Performance Tuning Guide Performance and Diagnostic Tool 3. When you click on an icon, a drop-down list displays. Each drop-down list contains a series of default filter options followed by specific values from the trace log: All - The default filter, this option displays all the values for the column from the client tracing log. Custom - Select this option to launch the Enter filter criteria for Type window. Use this window to define both an Operator and an Operand to filter the results. You can define as many filter conditions as you need, creating And/Or relationships between each custom filter. Blank - Use this option to only display any empty rows available on the client trace log. Nonblanks - Select this option to display all the populated rows on the selected client trace log. Column Value - You can filter the results by selecting a specific value available in the column. 4. To help organize the results, use the Group By feature on the grid. Click and drag a column header (for example, Object Name) to the Drag a column header here to group by that column area. You can drag multiple column headers to further structure the results as you need. Example If you group by using the Workstation ID column, the grid displays all the workstation entries together in alphabetical order. 5. You can review the following items on each business method call: UTC Type Object Name Method Name Execution Time ERPUser Client Workstation ID Exceeds MS Threshold - If this check box is selected, it indicates this server call took longer to execute than the Millisecond Threshold value you entered. MoreInformation TotalMilliseconds 6. The Summary Analysis sheet calculates the total performance results for each business object method. Notice you can group the results by various columns on this grid. 7. You can review several items on each business method call. Some key columns on this sheet include: Frequency - This value indicates how often the business object method sent a call to the server. % - Freq - The percentage this call was run compared to other method calls captured in this log. Total Execution Time Average Execution Time Longest Least % - Time - The percentage about how long this call was run compared to other method calls captured in this log. 8. Review the Errors and Messages sheet to analyze error messages. 33

34 Performance and Diagnostic Tool Performance Tuning Guide 9. You can review several items on each error. Columns on this sheet include: LogLine ClientID ObjectName MethodName StartDateTime ERPUser 10. Notice you can view the results in a Microsoft Excel file. To do this, click the Export to Excel button. 11. Save the.xls file to directory you can access. You can now launch the.xls spreadsheet and review it in Microsoft Excel. The information from all the Server Diagnostic sheets display in the exported spreadsheet. 12. To remove the generated results, click the Clear Results button. GetRowsKeepIdleTime Chart (Standard Metric) The GetRowsKeepIdleTime method can help you analyze how well the server interacts with Epicor client installations. You can review this standard metric through the GetRowsKeepIdleTimeChart in the Performance and Diagnostic Tool. The GetRowsKeepIdleTime method is used by the smart client. The client checks the application server to find out if the report or process has finished its run. If the run is finished, the data is sent back to the client as output for this method call. The client then uses this call to populate the results to the database or report. Because this method is a system check regularly sent to the server that typically does not return data, you can also use this method to measure network traffic that may impact performance. You use the Performance and Diagnostic Tool to display the GetRowsKeepIdleTime chart that shows you the performance time of each method call in the server log. You can then pinpoint specific times of the day when there was increased server activity that affected performance. To evaluate performance for GetRowsKeepIdleTime method calls: 1. You first need to indicate you want to generate the GetRowsKeepIdleTime Chart. Within the Performance and Diagnostic Tool, click Settings > Options. The Options window displays. 2. Click on the Server Diagnostics tab. 34

35 Performance Tuning Guide Performance and Diagnostic Tool 3. Select the Should Generate Chart check box. 4. Now indicate the Time Interval to Chart (minutes) option. This value defines the span of time you want to evaluate at each point in the GetRowsKeepIdleTime Chart. The default value is one minute. 5. Click OK. You return to the main window of the Performance and Diagnostic Tool. 6. As described previously, click the Add Files... button to find and select the server log you wish to review. 7. Click the Generate Diagnostics button. 8. Now click the GetRowsKeepIdleTime Chart tab. 35

36 Performance and Diagnostic Tool Performance Tuning Guide The chart displays using the time interval you selected. The left side of the grid indicates how long it took to run the GetRowsKeepIdleTime call, while the bottom of the grid indicates the time of the day when the call occurred. 9. Optionally, you can click the Export to Excel button to save this graph as an.xslx spreadsheet. You can then use Microsoft Excel to refine this graph as you need. You and your Epicor consultant use this spreadsheet to evaluate the overall performance of your system. Fields This topic documents the fields and sheets available for analyzing the server log files. Some fields on the interface have a context menu, which is indicated by a triangle in the upper right corner of the field. To open the context menu, right-click on the field. Add Files Click the Add Files button to select one or more server log files for analysis. Depending on what you are evaluating, multiple server.log files may have been generated. You can then analyze these files together within the Performance and Diagnostic Tool. You can also click the Down Arrow next to the Add Files button to select server log files from multiple server locations. You may need to do this on load balanced systems where the application is located across multiple servers. Clear Results Click the Clear Results button to remove the log file information from the Results, Summary, Errors and Messages, and GetRowsKeepIdleTime Chart sheets. 36

37 Performance Tuning Guide Performance and Diagnostic Tool Clear Selected Click the Clear Selected button to remove the log file paths you have currently loaded into the Performance and Diagnostic Tool. Epicor Version Use this drop-down list to select the application version run with the selected server log(s). The default version is Errors and Messages Review the Errors and Messages sheet to analyze error messages that generated in the error log. You can review several details including the client workstation that originated the error, the business object which generated it, and so on. Export to Excel Click the Export to Excel button to export results to Microsoft Excel for further manipulation and analysis. The exported data is based on which grid is currently visible. For example, if the Results grid is visible, the data on this grid is exported. Generate Diagnostics Click the Generate Diagnostics button to create the server log file data results. GetRowsKeepIdleTime Chart Use the GetRowsKeepIdleTime chart to display the performance time of each GetRowsKeepIdleTime method call in the server log. This method call is run by the System Monitor to check for server related activity like uploading reports. You can then pinpoint specific times of the day when there was increased server activity which affected performance. Ignore GetRowsKeepIdleTime Click this check box to hide all GetRowsKeepIdleTime method call information from the server analysis results. This method call is run by the System Monitor to check for server related activity like uploading reports. Because this method is called frequently by every client on your system, select this check box to prevent this active calls from appearing in the log results. Millisecond Threshold Any BusinessObject.Method calls greater than the value entered in this field have their Exceeds Threshold check boxes automatically selected. This value indicates these calls are over the threshold value (in milliseconds) you defined. Use this value to perform Sort By or Group By actions to review longer duration method calls. Results Sheet The Results sheet displays all calls received by the application server during the run of the server log. You can use Group By functionality to group the results by Object and/or Method to see the execution times for all method calls. Summary Sheet The Summary sheet summarizes the results for each BusinessObject.Method, showing the number of calls and the Total/Average/Least/Longest duration of those calls. These duration values are measured in milliseconds. You 37

38 Performance and Diagnostic Tool Performance Tuning Guide can sort the results by column to determine the types of calls made most frequently against the server, or which calls consume the most time. Scenarios This topic contains examples of how you can measure server performance through different methods. Identification of server performance compared to other servers. Use this method to identify performance differences for similar business object method calls on different servers. You do this by comparing the elapsed times for the method calls on one server against another server. These time differences may be related to CPU, database, customizations, operating systems, hardware configurations, and other differences. Analysis of activity by workstation. If you group by on the Workstation column, you can identify the method calls made by specific workstations. If a user reports slow performance, you can identify what this user was doing at the time and whether the elapsed times for method calls run using expected elapsed times. Analysis of system performance over time. If you group by using the Object Name and Method Name columns, you can observe the performance of specific method calls throughout the day. You may be able to discover slow periods by observing overall differences in execution time patterns for those methods. Analysis of most common activities and performance. If you sort by the Frequency column on the Summary sheet, you can view the most frequent method calls and their execution times. Sort by the Total Time column to see what activities consume the most time on the server. Analysis of server verses network elapsed time for methods. On the Results sheet, if you group by the WorkStation column, you can see the execution times of specific method calls on that workstation. You should also be able to identify the same calls from the Client Diagnostics for that workstation and compare the differences. For example, if you see an elapsed time of 800 milliseconds for a server call and you see 1500 milliseconds for that same call on a client, you can conclude that about 700 milliseconds was spent transferring data over your network. If you do not like this network performance, you can analyze network traffic using a network analysis tool. Identification of customization issues. You can analyze the elapsed time it takes to call specific methods compared to other server installations. This may indicate issues with customizations, business process method (BPM) method/data directives, or C# code customizations. 38

39 Performance Tuning Guide Additional Diagnostic Resources Additional Diagnostic Resources Additional resources are available that can help you locate deadlocks and locking and blocking. These tools are available through the Performance and Diagnostic Tool. Locking and Blocking occurs when two or more database connections try to access the same piece of data simultaneously. A piece of data is locked when a connection needs exclusive access to it. A Deadlock occurs when two users or sessions have locks on separate business objects, and each business object process tries to establish a lock on the business object in use by the other user/session. Use the Performance and Diagnostic Tool to download diagnostic testing resources that can check for these potential issues. These resources are SQL procedures and templates for the SQL Profiler. You use the SQL Profiler within Microsoft SQL Server Management Studio, and these templates display data specific for locking/blocking and deadlocks. To do this, click Options > Download Resources. You then download the following.zip files and set them up to work with SQL Server Management Studio. SQLProfilerTemplates.zip -- Download this file to evaluate locking and blocking on your system. DeadlockAnalysis.zip -- Download this file to evaluate deadlocks. Locking and Blocking You can use the sp_lock3.sql stored procedure to determine if a system is experiencing excessive data locking and blocking. Locking and blocking occurs when two or more database connections try to access the same piece of data simultaneously. A piece of data is locked when a connection needs exclusive access to it. SQL Server prevents, or locks, this piece of data so it cannot be updated by other connections. This data is then blocked when another connection attempts to access this data. This data becomes available when the first connection releases it. Similar to deadlocks, you need to discover what connection(s) could not access the data (the victims) and what connection blocked the data (the culprit). Excessive locking and blocking slows down database performance. By running the sp_lock3.sql stored procedure, you can determine whether excessive locks and blocks are occurring on your database. Set up the sp_lock3.sql File You download the sp_lock3.sql file through the Performance and Diagnostic Tool. 1. Launch the Performance and Diagnostic Tool. Depending on your operating system, you launch this tool in different ways: a. If you are on Windows SQL Server 2008 R2 or Windows 7, click Start > All Programs > Epicor Software > Epicor Administrative Tools > Performance and Diagnostic Tool. b. If you are on Windows SQL Server 2012 or Windows 8, press the <Windows> + F button to display the Charms bar; from the Apps screen, select Performance and Diagnostic Tool. The Performance and Diagnostic Tool displays. 2. Click Options > Download Resources. 39

40 Additional Diagnostic Resources Performance Tuning Guide The Download Resources window displays. 3. Select the check box next to the SQLProfilerTemplates.zip item. 4. Click OK. The SQLProfilerTemplates.zip file is downloaded to your computer. 5. Unzip the SQLProfilerTemplates.zip file in a file location you need. Configure the sp_lock3.sql File You use SQL Server Management Studio to configure this stored procedure file. 1. Click Start > All Programs > Microsoft SQL Server 2008 > SQL Server Management Studio. Microsoft SQL Server Management Studio displays. 2. Connect Microsoft SQL Server Management Studio to the SQL instance you wish to monitor. 3. Open the sp_lock3.sql script and run it against the database you wish to test. To do this, first click File > Open > File and navigate to the C:\_PerformTune folder; select the sp_lock3.sql file. 4. In the Tree View, expand the Databases node and select the database against which you want to test for locks and blocks. 5. Now click in the sp_lock3.sql pane; this causes the procedure to be in focus. 6. Either click the Execute button or press <Ctrl> + E. You should see a "Commands(s) completed successfully." message. This adds the sp_lock3.sql procedure to the selected database. 7. To test the procedure, click the New Query button. 8. Enter and click the Execute button. Tip Make sure you enter a space between sp_lock3 and the database name. Notice you place the name of the selected database between the brackets; for example:[mydatabase] The log will run. Depending on the locking situation, it may show some lock data. However if you receive a "Could not find stored procedure 'sp_lock3'" error message, the SQL procedure is not installed correctly. Repeat these steps to re-install the script. The sp_lock3.sql procedure is now configured to create a log against the selected database. Create a SQL Job Now within SQL Server Management Studio, you need to schedule a SQL Server Agent job that runs the sp_lock3.sql script and then appends its output to a log file. 1. Using the Object Explorer, expand the SQL Server Agent node; right-click the Jobs folder and select the New Job... option. The New Job window displays. By default, the General node is selected. 2. Enter a Name for your log file. For example, enter <XXX>_LocksLog (Where XXX are your initials or another identifying value). 40

41 Performance Tuning Guide Additional Diagnostic Resources 3. Verify the Enabled check box is selected. 4. Now on the Tree View, select the Steps node. 5. Click the New button. The New Job Step window appears. 6. Enter a Step name for this job step. Enter <XXX>_JobStep01 (Where XXX are your initials or another identifying value). 7. Click the Database drop-down list and select the database against which you want to test the locks and blocks. For example, select Demo. 8. Now in the Command field, enter the following: Tip Make sure you have a space between sp_lock3 and the database name. 9. Return to the Tree View and select the Advanced node. 10. Use the Output file field to define the directory path for the log file. You can select the directory path by clicking the browse (...) button. Example C:\_PerformTune\<XXX>LocksLog_Output.txt (Where XXX are your initials or another identifying value) 11. Select the Append output to existing file check box. 12. Close the Job Step window by clicking OK. You return to the New Job window. 13. From the Tree View, select the Schedules node. 14. Click the New button. The New Job Schedule window displays. 15. Enter the following values in this window: Name: <XXX>_LocksLogSchedule (Where XXX are your initials or another identifying value) Schedule type: From this drop-down list, verify the Recurring option is selected. Enabled: Verify this check box is selected. Occurs: From this drop-down list, select the Daily option. Occurs every: Click this radio button option and indicate this log runs every 1 minute(s). No end date: Verify this radio button option is selected. 16. To exit out of the jobs windows, click OK twice. You return to SQL Server Management Studio. 41

42 Additional Diagnostic Resources Performance Tuning Guide Start the Locks Log You now activate the log. 1. Verify the SQL Server Agent is running. To do this, go to the desktop and click Start > All Programs > Microsoft SQL Server 2008 > Configuration Tools > SQL Server Configuration Manager. The SQL Server Configuration Manager window displays. 2. In the Tree View, click the SQL Server Services icon. The right pane displays the SQL Server icons. Verify a green arrow appears next to the SQL Server instance you want running. For example: SQL Server (SQL2008) 3. Close the SQL Server Configuration Manager. You return to Microsoft SQL Server Management Studio. 4. Fom the Object Explorer, right-click the <XXX>_LocksLog (Where XXX are your initials or another identifying value). 5. From the context menu, select Start Job at Step. 6. The Start Jobs window displays with both the Success icon and statuses listed in the grid. Click the Close button. The sp_lock3 output file is appended to the database. Locking and blocking data now records to this log every minute. Track this data for a day and then return to this log to evaluate the results. Review the Locks Log This section describes how you review the locks log and resolve the locking issues. The sp_lock3 procedure only generates output when it detects locking and blocking issues. If the locks log is empty, locking and blocking is not a performance issue on the database. However if this log contains some locking and blocking entries, use the log to review what database connection is trying to access the data at the same time. In the above example, COMPANY/JoeS from the USERCOMP84 machine is accessing the Epicor ERP database. The access currently given to USERCOMP84 is not recommended (this connection is the culprit), as it can cause locking and blocking. In this case, Epicor would recommend that only the COMPANY/EPICADMIN login from the APPSERVER machine have access to this data. If the log records locking and blocking issues while users run custom or modified reports using an ODBC connection, consider creating a table view for the report. Since database tables can become locked while the report runs, the table view can prevent locking and blocking situations. To create a table view: 1. Open the report in a third party application like Microsoft SQL Server Report Builder, Excel, or another ODBC source. 2. Create a view for each table that has ODBC access. 42

43 Performance Tuning Guide Additional Diagnostic Resources 3. Enter a with(nolock) statement on the view. 4. Save the file. When users access the table view, they cannot create a share lock on the data. This improves the performance of the report. Tip If you decided not to create a table view for the report, you should instead enter the with(nolock ) option in the SELECT statement for the table. You should also use this option to optimize performance on Business Activity Queries (BAQs), Business Process Management (BPM) methods, and customizations. Locking and Blocking Article The following internet article provides some additional information on how you can detect locking and blocking. Deadlock Tracing A deadlock occurs when two users or sessions have locks on separate business objects, and each business object process tries to establish a lock on the business object in use by the other user/session. SQL Server automatically detects and resolves deadlocks. If a deadlock occurs, one process ends (the victim), while the other process runs (the culprit). The victim transaction is rolled back. While determining the victim is relatively easy, what you need to discover is the culprit process that caused the deadlock. This culprit could be another Epicor ERP application process running at nearly the same time or a third party application preventing the victim process to initialize. You can monitor when deadlocks occur using the SQL Server Deadlock Graph. While this graph runs, deadlock events are recorded in the SQL Profiler trace log. Deadlock Graph Setup Important Before you activate the SQL Deadlock Graph, you must set the Epicor server logs to the Verbose setting. When you finish deadlock tracing, be sure to deactivate the server log. You download the deadlock graph through the Performance and Diagnostic Tool. 1. Launch the Performance and Diagnostic Tool. Depending on your operating system, you launch this tool in different ways: a. If you are on Windows SQL Server 2008 R2 or Windows 7, click Start > All Programs > Epicor Software > Epicor Administrative Tools > Performance and Diagnostic Tool. b. If you are on Windows SQL Server 2012 or Windows 8, press the <Windows> + F button to display the Charms bar; from the Apps screen, select Performance and Diagnostic Tool. The Performance and Diagnostic Tool displays. 2. Click Options > Download Resources. The Download Resources window displays. 3. Select the check box next to the DeadlockAnalysis.zip item. 43

44 Additional Diagnostic Resources Performance Tuning Guide 4. Click OK. The DeadlockAnalysis.zip file is downloaded to your computer. 5. Unzip the DeadlockAnalysis.zip file in a directory location you need. Activate the SQL Deadlock Graph You access the SQL Server Profiler to activate the SQL Deadlock Graph and run a deadlock trace against a server. To do this, launch SQL Server Management Studio. 1. Click Start > All Programs > Microsoft SQL Server 2012 > SQL Server Management Studio. Microsoft SQL Server Management Studio displays. 2. When the Connect to Server window displays, click Connect. Tip Notice you can select a different server on this window. Use this feature to activate the deadlock graph for different servers. 3. Now click Tools > SQL Server Profiler. The SQL Server Profiler program displays. 4. When the Connect to Server window displays, click Connect. The Trace Properties window displays, and the General sheet is active. 5. In the Trace Name field, enter a meaning name for the tracing file. 6. From the Use the template drop-down list, select Blank. 7. Click the Events Selection tab. 8. Expand the Locks node and select the following options: Deadlock graph Lock: Deadlock Lock: Deadlock Chain 9. Click the Run button. The words "Trace Start" display in the SQL Server Profiler and the trace begins. You typically run this trace log for a full day to capture the potential deadlocks that occur on your system. Stop the SQL Deadlock Graph 1. When you are ready to stop the deadlock trace log, click File > Save As > Trace File. The Save As window displays. 2. Enter the File Name that helps you locate the file. 3. Either save the file to the default folder, or click the Browse Folders button to find and select a different folder. 4. Click Save. 44

45 Performance Tuning Guide Additional Diagnostic Resources 5. Now click File > Stop Trace. 6. Close SQL Server Profiler and Microsoft SQL Server Management Studio. You are now ready to review the SQL Deadlock Graph. 45

46 Load Distribution Performance Tuning Guide Load Distribution You can improve how the Epicor ERP application performs by assigning different tasks to specific application servers that can handle the load. This distributes the load more evenly across your system resources. If you have less than two hundred users, the Epicor ERP application can run efficiency through one server machine. However if you have more than 200 users, use VMWare, or your organization requires special processing that requires significant system resources, consider setting up task agent rules to more evenly distribute the load. For example, you could assign a process that requires significant resources to run on a more powerful server, like Material Requirements Planning, and then assign reports that generate less data to a server with fewer resources. By distributing the load between these application servers, you reduce performance bottlenecks and match a report/process with a machine best suited to run it. The Hardware Sizing Guide can help you determine whether you should assign load to multiple application servers. After you identify the hardware and usage scenario that best matches your organization, you can then decide whether distributing the load will improve performance. To set up this functionality, you first create application servers for each server machine available on your system. Within the Epicor ERP application, you set up security groups and task agent rules. Security groups determine which users have access to specific tasks. Task agent rules use security groups to determine which application server will handle the processing for a specific task or group of tasks. Tip If you do not have security groups set up yet, review the Security chapter in the Epicor Implementation User Guide. This user guide is available on EPICWeb: Application Servers An application server manages how a specific instance of the Epicor application runs. Through each application server, you can configure licenses, companies, sessions, and users for a specific database. You can set up multiple application servers to run the same database and balance the load. For example, you create two application servers for the same database, but these application servers are linked to different server machines through their endpoint bindings. One application server is set up to run Epicor Web Access (EWA) on one server machine, while another application server is set up to run a smart client through Net.TCP on a different server machine. Likewise you could set up another application server that links to a machine which only handles SSRS reporting tasks. You add the application server that will interact with your server machine through the Epicor Administration Console. This management tool is located on your server machine. For information on how to add application servers, review application help in the Epicor Administration Console. Task Agents The task agent handles all scheduled tasks for an application server. The task agent activates any program added to a recurring schedule. Users add programs to recurring schedules through the Schedule drop-down lists available on programs throughout the Epicor application. You create these schedules in the Epicor application using System Agent Maintenance, and you also use this program to create task agent rules to distribute the load. 46

47 Performance Tuning Guide Load Distribution Tip To learn how to assign tasks to automatic schedules, review the System Agent Maintenance topics in the application help or review the Automatic Data Processing chapter in the Epicor Implementation User Guide. This user guide is available on EPICWeb: You create a task agent using the Task Agent Service Configuration program. You launch this program from within the Epicor Administration Console. When you select the application server, the center pane on the Epicor Administration Console displays the settings for the application server. Click the Task Agent Configuration button to set up the task agent for the selected application server. Create Task Agent Rules After you have set up the new application server and security for your users, you are ready to create task agent rules that distribute the load between the available application servers. 1. Navigate to System Agent Maintenance. Menu Path: System Setup > System Maintenance > System Agent Important This program is not available in the Epicor Web Access. 2. Click the Agent ID button to find and select the system agent. Only one system agent is available. 3. Click on the Actions menu and select the Edit Task Agent Rules... option. The Task Agent Rules window appears. 4. Click the New button. 5. Select the Company for which this task agent rule will generate tasks. Only companies assigned to the current user account display on this drop-down list. The task agent rule will then handle processing for the selected company. 6. Optionally select the Security Group for the task agent rule. Whenever a user assigned to the selected security group runs a report or process linked to this task agent rule, the application server linked to this rule generates the system activity. 7. Next define the Rule Type option. This value indicates what tasks are handled by the task agent rule. Available options: Specific Task - Indicates this task agent rule will only run against a specific process. After you select this rule type, you next select the process from the Process Id drop-down list. All Tasks - Indicates all processes are run against this task agent rule. Any time a process is launched by users within a company or a security group, this task agent rule handles the processing. Specific Report - Indicates this task agent rule will only run against a specific report. After you select this rule type, you next select the report from the Process Id drop-down list. All Reports - Indicates all reports are run against this task agent rule. Any time a report is launched by users within a company or a security group, this task agent rule handles the report generation. 8. If you select either the Specific Task or the Specific Report rule type, you next select the Process Id for the item you want the task agent rule to run. Depending on the rule type, either reports or processes display on this drop-down list. 47

48 Load Distribution Performance Tuning Guide 9. Enter the Appserver URL for the application server that will run the activity for this task agent rule. This value links the task agent rule to the application server's location. Tip You can find this value by opening the system configuration (.sysconfig) file for the client installation. Locate the <appsettings><appserverurl> node and copy this value. 10. Use the Endpoint Binding drop-down list to indicate how this application server checks for authentication certificates through Internet Information Services (IIS). Select the Endpoint Binding defined on the application server; the same options display on this drop-down list. 11. If you select either the UsernameWindowsChannel or UsernameSSLChannel endpoint binding option, enter the User ID and Password for the account that can access this application server. 12. When you finish adding the task agent rule, click Save. Continue to add the task agent rules you need. If you need to remove a task agent rule, highlight it on the grid and click the Delete button. Now the next time the system agent activates a schedule or a user launches a process or report, the tasks are distributed to the application servers defined on the task agent rules. 48

49 Performance Tuning Guide Application Tuning Application Tuning This section contains some application tips and techniques you can do inside the Epicor application. You should only use the tips and techniques that match your use of the Epicor application. Application Troubleshooting Be sure to disable all schedules before you move from the training database to your live database. If you do not, two MRP processes run at the same time, slowing down performance. Epicor recommends that you do not run MRP processing during the work day. Schedule MRP processing during off hours. Be sure that your Business Activity Queries (BAQs) are properly written. If a BAQ pulls unnecessary data or a calculation is improperly constructed, each time this BAQ is run it slows performance. Avoid creating several Business Process Management (BPM) procedures that generate messages at the same time (synchronously). If these procedures run frequently, performance is reduced. Instead, set up your BPM procedures to run asynchronously. Avoid activating too many change logs. Only run change logs for the specific fields you need. When too many logs are running, they slow performance. Performance slows when several large reports generate at the same time. To resolve this, move your report generation processes to separate application servers. The previous Load Distribution section describes how you set up task agent rules to distribute the load as needed. Likewise, when multiple applications run at the same time, performance slows. Try to only use the programs you need when you need them. Close them if they are no longer active. Client Cache The Client Cache is the local Disk Cache used by the Epicor client machine. As the Epicor client needs various items stored on the server and in the database, these items are downloaded and stored in the local Disk Cache for better performance. The local client cache stores items like custom context menus, themes, processing calling xref files, customizations, personalizations, list of business objects access for security retrieval, and so on. Storing these items in the client cache typically improves client performance, but sometimes too many items can accumulate in this folder and cause slow performance instead. As a standard practice, users should periodically clean out their local client cache to remove rarely used items. To clear the cache: 1. Launch the Epicor ERP application. 2. From the Main Menu, click Options > Clear Client Cache. 3. When the client cache is cleared, exit the Epicor application. 49

50 Application Tuning Performance Tuning Guide Now when you restart the Epicor application, only items currently needed to launch the application are loaded into the client cache. The client machine should now have improved performance. However as time goes on, more items will download to the client cache, and the user should run the Clear Client Cache option again. Locate the Client Cache If you would like to review the items currently in a client cache, you can locate this folder in the following default directory locations: Older Windows Systems Use Windows Explorer to navigate to the C:\Documents and Settings\All Users\Application data\epicor directory. Locate the name of the Epicor ERP client installation. Newer Windows Systems Use Windows Explorer to navigate to the C:\Users\All Users\Epicor directory. Locate the name of the Epicor ERP client installation. Tip You can change where the client cache is located by changing a value in the configuration settings file. This xxx.sysconfig file (where xxx is the name of the file you use to run the client) is located in the client's Config folder. Open this file in a text editor and enter a different directory path in the AlternateCacheFolder setting. Memory Cached Programs Users can personalize programs by using memory caching. Memory caching keeps frequently used programs in memory so they launch faster during the same Epicor application session. When a program is memory cached and the user closes the program, this program is still loaded in active memory. Technically this program is really placed in hide mode. When the user launches the program again, the form is already constructed in memory and quickly displays its interface. Tip Do not confuse client caching with memory caching. Client caching refers to the various items that move between the client and the server/database. Memory caching is a performance feature available internally on most Epicor ERP application programs. Memory caching takes resources from the system, and also uses other resources including user objects. A limited number of user objects are available on each client, and when a client reaches that limit, the Epicor application can become unstable. It can vaporize or deliver strange messages. The default limit is 10,000 user objects. A program like Customer Maintenance can use 1,800 user objects, so users should only select a few programs for memory caching. If you need, administrators can increase the number of user objects available on a client to 18,000. However be aware that only 64,000 user objects are available for the Windows operating system, so be careful how many user objects you allow each client machine to use. Memory Cached forms are removed from active memory when the Epicor client session ends (the Epicor.exe file is closed). Users should only use this feature on a limited number of commonly used programs they use on a daily basis. Client Customizations Use this technique to test the performance of a client customization against the base program version. Make sure the user account currently logged into the application has Customization privileges. Then turn on customization mode from the Menu by either clicking the Settings tile and selecting Developer Mode or move your mouse pointer over the bottom of the screen and click the Developer Mode button on the toolbar. 50

51 Performance Tuning Guide Application Tuning Navigate to the customized program and launch it. When the Select Customization window appears, select the Base Only check box. Click OK and time how long it takes to launch the program. Close the program and launch it again, selecting the customization. Time how long it takes the customized version of the program to launch. Order Entry Performance Tuning Do the following to improve Sales Order Entry processing. Ready to Process - Company Configuration The Company Configuration program contains sheets that define default values for each licensed Epicor module. Unless your company typically creates small sales orders with few detail lines, you should not set up the Order Management module to cause each new order to default as ready to process. If the module is configured with this option active, each time a user adds a detail line to a sales order, taxes and book detail records automatically generate for the line or release. This causes additional processing to run each time a detail line/release is added, increasing how long it takes users to enter sales orders. Prevent this additional processing by setting up the Order Management module to not automatically run this process. To do this, launch Company Configuration: Menu Path: System Setup > Company/Site Maintenance > Company Configuration Navigate to the Modules > Sales > Order sheet. Clear the Ready to Process check box. Ready to Process - Sales Order Entry Likewise while users enter detail lines on a sales order, they should make sure the Ready to Process check box is clear (not selected) on the sales order header. If this option is not defined as the default value in Company Configuration, the Ready to Process check box is not be selected by default on each new sales order. However it can be selected or cleared on specific sales orders. As described previously, when this check box is clear, it prevents Sales Order Entry from automatically calculating taxes and creating book detail records each time users add a detail line. If this check box is selected before users finish entering detail lines, these processes run after each line is added, increasing how long it takes to enter sales orders. Users should only select the Ready to Process check box when they have finished a sales order and are ready to finalize it. Once they select this check box and click Save, the taxes and book details are calculated for all the detail lines and releases at the same time. To view this check box, navigate to Sales Order Entry. The Ready to Process check box is located on the Summary and Header sheets. Menu Path: Sales Management > Order Management > General Operations > Order Entry Tip The CRM menu path is: Customer Relationship Management > Order Management > General Operations > Order Entry Auto-Retrieving Inventory or Jobs Users should also make sure Sales Order Entry is not set up to automatically retrieve inventory quantities or released jobs. This additional processing can cause entering and updating sales orders to slow down. If users do not need this functionality, you should de-activate it. To shut off this functionality, click Actions > Auto Retrieve > Auto Retrieve Inventory Quantities and/or Actions > Auto Retrieve > Auto Retrieve Release Jobs. If a check mark appears next to either option, it indicates release job quantities and/or inventory quanties are automatically pulled into the sales order when part 51

52 Application Tuning Performance Tuning Guide records linked to inventory or release jobs are selected on a sales order detail line. To clear the check box, select the option. Now when users create a new sales order, these records will not automatically be pulled into the sales order, improving performance by eliminating this additional processing. EDI Entry Process You can customize Sales Order Entry to display an Entry Process field that improves the performance of sales order processing by linking the order directly to EDI or Service Connect. This additional field does not display on the default interface, but you can create a customization that displays this field on either the detail line or release sheets. When users enter the EDI value in this field, it indicates that the Electronic Data Interchange processes runs against the order detail lines or releases, improving how quickly detail lines and releases are generated for this sales order. To do this, create a new customization and navigate to either the Lines > Detail or Releases > Detail sheets. Launch the Toolbox to add an EpiTextBox field to the form. Depending on which sheet you are on, you bind this EpiTextBox field to either OrderDtl.EntryProcess or OrderRel.EntryProcess table columns. Save the customization and make it available to users. When users create a sales order that is populated through EDI, they enter "EDI" in this field. This improves how quickly EDI data is imported into the new sales order. Tip If this is how your organization typically creates sales orders, you could also create a Business Process Management (BPM) directive that automatically populates this field with the EDI value each time the user creates a new detail line or release. Be sure to test how long it takes to process sales orders both with or without the customization. If you notice better performance, make this customization available to all the users in the company. MRP Performance Tuning The following tips can help you improve the performance of the Material Requirements Planning (MRP) process. By following these MRP tips, you will place less demand on your network and server resources. MRP Scheduled Times Epicor recommends you do not run MRP processing during the work day. Schedule MRP processing during off peak hours. Net Change Mode versus Regenerative Mode You can run the MRP process using two calculation modes - Net Change and Regenerative. You should typically run MRP processing in Net Change mode as much as possible, as it reduces the number of calculations that must complete during the process. When MRP is run in Net Change mode, the process ignores all previously generated information and only updates suggestions for records changed since the date of the previous MRP run. This calculation mode keeps your records current, only generating items that reflect new or updated source requirements. It also reduces the resources required to run MRP on your server, generates the MRP results faster, and frees up your server for other purposes. Only use Regenerative mode through a regular, periodic schedule. Generally you should process MRP in Regenerative mode once a week or once a month. This calculation mode deletes all previously generated MRP information. Because of this, Regenerative mode actually runs through two routines - a routine that deletes all of the unfirm jobs, job suggestions, and purchase orders, and then a second routine that generates and recreates all of the suggestions. While it is important to run a full regeneration periodically to make sure you have a complete set of records that reflect the current state of your database, running MRP in Regenerative mode more frequently than once a week will negatively impact the performance of MRP and your server. 52

53 Performance Tuning Guide Application Tuning Finite Scheduling During MRP Only activate the Run Finite Scheduling During MRP Calculation check box option if you require unfirm jobs and job suggestions to be finitely scheduled in the near future. This check box is located on the Process MRP window. In order to run MRP processing in this mode, the scheduling functionality has to execute to calculate the available capacity in each resource. This additional scheduling process requires significant resources from your network to finitely schedule unfirm jobs and suggestions. Sort 0 Level MRP Jobs If you select the Sort 0 Level MRP Jobs by Requested Date check box, you may reduce MRP performance. This situation occurs because the MRP process does not send unfirm jobs for zero assembly parts to the scheduling engine until these jobs are completely processed by the MRP engine. This option improves the accuracy of the schedule, but it typically adds additional processing time for the MRP results to generate. Number of MRP Processes and Schedulers While the MRP process runs, the server can generate multiple processing threads to complete the operation. The more MRP processes you can run, the faster unfirm jobs and suggestions can execute and complete. You can also improve performance by increasing the number of schedulers that can run on your server; the more schedulers you can run, the faster the scheduling engine can schedule unfirm jobs. You modify the Number of MRP Processors and Number of Schedulers values on the Process MRP window. However as you increase the number of MRP processors and schedulers, the performance boost you receive will eventually decline. This occurs because the server will run out of capacity to handle all of the multiple threads you attempt to run concurrently. Your server has limited capacity, so there will be a point when using multiple MRP processors and schedulers can slow down MRP performance as well. To help you decide how many MRP processors and schedulers you can run at the same time, check your MRP logs to review the performance results. Continue to increase the MRP processes and schedulers until you notice the MRP performance times begin to increase again. You should then be able to determine the optimal values for your server. Most servers can handle two MRP processes and two schedulers at the same time, so you could start by entering a "2" value in both fields. As you run MRP, continue to monitor the MRP and Scheduling logs. If the schedulers are consistently waiting for the next job, you have some options. You can remove one scheduling thread to free up more CPU resources for the rest of the company. If you have CPU resources available, you can also add one more thread to the Number of MRP Processes field and keep the Number of Schedulers value the same. Likewise, if you notice times in the log where the MRP process threads are idle, the processors can also be used as scheduling threads. Tip For more information on processors and schedulers, review the MRP Logs section in the MRP Technical Reference Guide. This guide is available in the application help; navigate to the MRP > Working With > MRP Technical Reference Guide node. Moving Between Databases Be sure to disable all schedules before you move from the training database to your live database. If you do not, two MRP processes run at the same time, slowing down performance. 53

54 Application Tuning Performance Tuning Guide BAQ Performance Tuning This section details some performance tuning options for business activity queries. It also has a section which describes how to correct SQL syntax errors. BAQ Server Settings You can define some application server settings in the Epicor Administration Console to restrict how business activity queries (BAQs) generate results. By defining these options, you can limit performance issues caused when BAQs process large amounts of data. Note that when you change these application server settings, you will cause the application server to restart. Be sure to change these settings during a period of the day when few users are logged into the Epicor application. 1. You launch the Epicor Administration Console from your server machine. Depending on your operating system, you launch this tool in different ways: a. If you are on Windows SQL Server 2008 R2, click Start > All Programs > Epicor Software > Epicor Administrative Tools > Epicor Administration Console. b. If you are on Windows SQL Server 2012, press the <Windows> + F button to display the Charms bar; from the Apps screen, select Epicor Administration Console. The Epicor Administration Console displays. 2. From the tree view, expand the Server Management node and Epicor Server node. 3. Select the application server you need to monitor. 4. Now from either the Action menu or the Actions pane, select Application Server Settings. The Application Server Settings window displays. The BAQ values you modify are in the Application Settings group box: 54

55 Performance Tuning Guide Application Tuning 5. For the BAQ Query Max Result Rows field, enter the highest number of rows that can be returned by a business activity query. By entering a value in this field, you restrict how many rows can be returned by each BAQ. This prevents the query from pulling in an unlimited number of records, restricting situations where a runaway BAQ consumes too many system resources to generate query results. 6. Now in the BAQ Query Timeout field, enter how many seconds can elapse before the application server stops the query. By entering a value in this field, you define how long each BAQ is allowed to run. When a query attempts to generate results and reaches this time limit, the application server stops the query and sends the user a time out message. 7. Click Apply and then OK. 8. The Server Manager dialog box displays, asking if you want to restart the application server. If this is a good time to restart the server, click Yes. 55

56 Application Tuning Performance Tuning Guide Now when this application server processes BAQs, the queries generate using these row and timeout limits. BAQ Best Practices This section contains a series of best practice methods that will help you develop more efficient business activity queries (BAQs). If you follow these suggestions, you will have more success creating both display-only and updatable BAQs. The Data Funnel Try to create business activity queries that funnel data from the broader parent tables to the more specific child tables. These tables contain logical links between each other that you can leverage to make it easier to pull in the results you want. For example, if you want to create a BAQ that displays customer ship to locations, set up a relationship that first pulls data from the customer (Customer) table to the customer ship to (ShipTo) table. Use Established Joins You can link tables together through custom joins, but running BAQs with nonstandard joins requires additional processing to generate results. Instead, you should link tables that have standard, or proper joins that automatically appear when you select the related tables. Because these tables already contain logic that connects them, less processing is required to generate results from these queries. Fewer Tables, Better Performance Carefully consider how many tables you need to join in your business activity query. The functionality has few restrictions, so you can create incredibly complex queries. However the more tables you join on the query, the slower performance the query has during runtime. If you need to see a lot of data at once, create a series of smaller BAQs and then display them together on the same dashboard. Similarly, if you need to create a complex view of your data, consider building an executive dashboard instead. Both the dashboard and executive dashboard features use BAQs as building blocks for gathering and displaying data, but contain additional, more efficient functionality for publishing and subscribing values between tables. Typically you achieve better performance if you have a series of smaller BAQs you can manipulate through a dashboard, rather than one huge BAQ that requires a lot of processing time to pull its data. Fewer Fields, Better Performance Likewise, only select the fields you want to display in your BAQ results. The fewer fields you select, the less processing that needs to occur to populate results in the BAQ. Only select the fields you require to pull in the data you need. 56

57 Performance Tuning Guide Application Tuning Sorting Performance Sorting data by a selected column is a powerful feature, but be aware that some significant processing time may be required to display the reordered results. This situation is especially true when you sort a large amount of data. The query tool has to first return all of the records into memory before it can re-order their sequence through the selected column. All of this processing occurs on the server, so the data calls need to move across the network before they arrive at your client workstation. So if you sort on a large amount of data, be patient the reordered results are on their way. Runaway BAQs If you suspect a business activity query is causing poor performance, use the server logs and the Performance and Diagnostic Tool to determine which BAQ is causing an issue. You do this by accessing the web.config file and then setting the server log to Verbose. Run the process that launches the BAQ. When the BAQ completes its run, open the server log in the Performance and Diagnostic tool. Navigate to the Server Diagnostics > Results sheet. Group the results by Object Name and review the Execution Time values. SQL Syntax Issues SQL function syntax is stricter than Progress syntax. If you previously ran the application using a Progress database but now have moved to SQL, you may experience these syntax issues. The main issue is that you can use abbreviations within Progress; for example, ABSOLUTE can be abbreviated to ABS, ABSO, or ABSOL within Progress. BAQ formulas are directly sent to SQL. As long as these formulas do not contain any abbreviations, they work as expected. However if a formula references an abbreviation, syntax issues occur. The following table displays the functions which are not identical between Progress and SQL. The characters contained between the parentheses are optional characters in Progress. Progress ABS(OLUTE) ASC(II) INT(EGER) MAX(IMUM) MIN(IMUM) SQL (Before Translation) ABS ASC INTEGER MAXIMUM MINIMUM SQL (After Translation) abs ascii convert dbo.maximum (user-defined function) dbo.minimum (user-defined function) 57

58 Baseline Performance Tests Performance Tuning Guide Baseline Performance Tests This section outlines a series of tests you can run on your Epicor ERP application to verify its performance against established metrics. These metrics were defined using a test machine. Configuration of the test machine: Windows Server 2008R2 64Bit CPU Dual Core Intel xeon 3GHz 20 GB RAM Epicor SQL database installed on separate database server with 1G bit/sec Ethernet connection Epicor version Test Setup Be sure to run all tests on the delivered base Training database. You should not have any active Business Process Management (BPM) directives or customizations running against the business objects you will test. If you receive significant variations in the results, this may indicate that BPM directives or customizations are degrading the application performance. These tests are intended primarily to measure server and database performance. To eliminate variations caused by the network, run all of these all tests over a LAN (Local Area Network) with a ping time (32 bytes) to the Application Server < 1 ms. If you test with a higher network latency, it may result in some measurable difference in the observed results. Start the Test Do the following steps to begin the test. 1. Install the Performance and Diagnostic Tool on your server machine as described in the previous section. 2. Log into your Epicor ERP application using both the Training database and the manager account. For the User ID, enter manager; for the Password, enter manager. 3. From the Menu tree view, navigate to the Epicor USA company. 4. Now open the Main site node. 5. Close the System Monitor. If the System Monitor is running in the Task Tray, then right click on this icon and select Exit. This eliminates GetRowsKeepIdle time calls to the server. 58

59 Performance Tuning Guide Baseline Performance Tests Test Procedure The following test procedure provides you with a repeatable path that uses standard data delivered within the Training database. The results from the tests are captured in the client trace log. You can then analyze these results using the Performance and Diagnostic Tool. Activate the Trace Log Use these steps to activate the trace log. 1. Log onto a client installation of the Epicor application. 2. Click on the Settings button; from the General Options list, select Tracing Options. The Tracing Options window displays. 3. Select the Enable Trace Logging check box. This activates the trace log. 4. Notice the Clear Log button. At certain points during the following tests, you will be asked to click this button. Clicking this button removes results from the trace log. You can then run the test and only the specific calls you want display in the client trace log. 5. Notice the Write button. At certain points during the following tests, you will be asked to click this button. Clicking this button causes business object calls to be recorded in the database. Open Sales Order Entry Form This test measures the observed time it takes for you to open a typical large form. This test will use Sales Order Entry. You will clear the Client Cache to measure how long it takes for the form to load initially after a new installation or service pack/patch upgrade. You will then run the test twice to measure the time it takes the form to load using both uncached memory and then cached memory. After you complete the tests, you will write the client trace log file and use the Performance and Diagnostic Tool to analyze the results. Test Form Performance Time 1. Click the Menu tile. 2. Navigate to the Sales Order Entry: Menu Path: Sales Management > Order Management > General Operations > Order Entry Tip The CRM menu path is: Customer Relationship Management > Order Management > General Operations > Order Entry 59

60 Baseline Performance Tests Performance Tuning Guide 3. Launch Sales Order Entry. 4. Within the form, navigate to Tools > Options. 5. Verify the Memory Cache checkbox is not selected (clear). 6. Close Sales Order Entry. 7. Return to the Home screen. 8. Click on the Settings button; from the General Options list, select Tracing Options... The Tracing Options window displays. 9. Select the Enable Trace Logging check box. 10. Now from the General Options list, select the Clear Client Cache option. 11. When you are asked if you want to clear the client cache, click Yes. 12. Return to the Home screen and navigate to Sales Order Entry again. 13. Using the stopwatch on a smart phone or similar device, test how long it takes Sales Order Entry to display. Activate the stopwatch and launch Sales Order Entry. 14. Finish stopwatch recording the moment the form displays on your screen and the cursor flashes in the Sales Order field. 15. Record this elapsed time as the First time form download and open value. 16. Close Sales Order Entry. 17. Once again, use a stopwatch on a smart phone or similar device to record how long it takes for the Sales Order Entry form to display and the cursor to appear. Launch Sales Order Entry. 18. Record this elapsed time for the Second time form open value. 19. Return to the Tracing Options Form and write to the Trace Log as described in the previous Activate the Trace Log section. Note the Log File name; this value uses the TraceDataxxx.log file format. 20. Launch the Performance and Diagnostic Tool. 21. Navigate to Client Diagnostics. 22. Browse to the Client File Trace Path to locate the client trace file you wrote as described previously. 23. Click the Generate Diagnostics button to capture and review the performance results. Expected Results: First time form download and open: < 8 seconds Second time form open: < 6 seconds The following screen capture shows an example of the business objects and their performance times. You should see similar results on your Summary sheet. 60

61 Performance Tuning Guide Baseline Performance Tests If you see more object calls than above and have much slower performance, it may because the Sales Order Entry form properties are not cached. If this is the case, you may see additional Lib type calls as shown below: Verify and Fix Performance Test Use the following steps to verify and fix issues with the form performance test. 1. Locate the xxx.sysconfig configuration settings file used by the client installation. This file is typically located in the Config folder under the client direction. 2. Using a text editor or similar program, open the xxx.sysconfig file. 3. Locate the <appsettings> section 4. Review the MaxBOMRU and MaxClssAttrMRU settings. Default values: <MaxBOMRU value="100" /> <MaxClssAttrMRU value="20" /> 5. Now locate the cached Business Object and Class Attribute.xml files. Directory paths on a Windows 7 client: C:\ProgramData\Epicor\<server-port>\ \EPIC03\BOSecMRUList\BOMRUList_<username>.xml C:\ProgramData\Epicor\ <server-port> \ \EPIC03\ClsAttrMRUList\ClsAttrMRUList_<username.xml> 6. If the SalesOrder business object does not appear in either of these xml files, it implies that other business objects are used more frequently on this client and are not cached. To fix this situation, do the following: a. Log off the Epicor application. b. Delete the two xml files. 61