WW TSS-10 - Historian 2012 R2 Architecture and Troubleshooting

Slide 1

WW TSS-10 - Historian 2012 R2 Architecture and Troubleshooting Paul McGuire Jim Stella social.invensys.com @InvensysOpsMgmt / #SoftwareRevolution /InvensysVideos /Wonderware /company/wonderware 2013 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys or its subsidiaries. All third party trademarks and service marks are the proprietary marks of their respective owners.

Agenda From Historian 10 to Historian 11 HCAL Historian Project Tracking Troubleshooting Tools and Tips Slide 3

The Wonderware Historian Solution The Wonderware Historian is a high-capacity, high-throughput server that stores time-series data. The historian acquires and stores process data at full resolution or at a specified resolution and provides real-time and historical data together with configuration, event, summary, and associated production data to client applications on the desktop. The historian combines the power and flexibility of Microsoft SQL Server with high speed acquisition and efficient data compression. Slide 4

64-bit Microsoft SQL Server Support Historian Architecture OLE DB Provider runs in SQL Server process Some interprocess communication via shared memory 64-bit SQL Server 64-bit Historian (mostly, but not all) Solution in Historian 2012 R2: Native 64-bit Microsoft SQL Server Support Native 64-bit Historian OLE DB Provider 64-bit Process Address Space Ability to utilize all installed RAM Higher scalability will require 64-bit Phasing out support for case-sensitive collation Slide 5

Storage Engine Retrieval Performance Legacy Storage >2x faster Storage Engine Retrieval time for 3-months of 1-second data for 1 tag using best fit with 1-hour period* *YMMV Slide 6

Engine Data Acquisition Throughput 150,000 100,000 10.0 2012 R2 50,000 0 Sustained Burst Values Per Second Slide 7 Late

Historian 10.0 Architecture SQL Server Retrieval IDAS/ SuiteLink History Blocks Storage Engine Pull Data Acquisition Storage Push Data Acquisition Replication Acquisition Application Server Historian New historian hidden within 10.0 Slide 8

Historian 2012 R2 Architecture SQL Server Retrieval IDAS/ SuiteLink Most changes apply only here History Blocks Pull Data Acquisition Push Data Acquisition Application Server <3.6 Storage Engine Classic Storage Application Server >3.5 Push Data Acquisition Replication Acquisition Historian Slide 9

Historian 10.0 Storage Data Flow CSV Historian 10.0 named pipes outside RTW Real-time Storage Manual Storage inside RTW IDAS MDAS (DCOM) Configuration Service AI History Blocks WAS 3.1 / SDK 1.0 Microsoft TDS OLE DB Provider (SQL) ADO / SSMS MDAS2 (TCP/IP) Tier-1 Slide 10 MDAS Server Tier-2 Storage Engine (aahstorageengine.exe) S/F

Historian 11.0 Storage Data Flow CSV Historian 11.0 named pipes Classic Real-time Storage IDAS MDAS (DCOM) WAS 3.1 / SDK 1.0 Configuration Service outside RTW inside RTW Classic Manual Storage AITag=0 AI OLE DB Provider (SQL) AITag=0 ADO / SSMS HCAL (TCP/IP) Tier-1 WAS 2012 R2 SDK 2012 R2 Historian Client Access Point (HCAP) Historian Storage (aahstorage.exe) AITag=0 Slide 11 AITag=1 History Blocks AITag=1 Microsoft TDS S/F

No Real-time Window Restriction for Data in the Past Real-time window check for data in the past is removed. But time in the future is still overwritten. After reconnecting to Historian, HCAL will keep the tag in store/forward until its value s timestamp exceeds the Cut-off time to ensure that the store/forward data can cover QD10 (disconnect) injected on HCAP. Cut-off time is the latest of the following: Configuration service start time HCAP start time The tag s disconnect time (QD10 s time stamp) Cut-off time is retrieved for each tag when HCAL reconnect with HCAP. While staying in store/forward, the tag s value can still be forwarded to Historian. Slide 12

Injecting Channel Information QD 24 NULL (HCAL), real-time tag (Channel Status = 1). Condition - Application Engine is set off scan or disconnected. Value Manufactured empty value with current time stamp. QD 248 Store/forward start (HCAL), all except replication tags. Condition Storage switches to Store/Forward. Value Duplicate of last value sent to server. QD 44 Reconnect (HCAL), all except replication tags. Condition Storage switches to server. Value Duplicate of last value sent to store/forward. Adjustment Time may be adjusted by HCAP to be 1 msec after timestamp of QD10 if needed. QD 10 Disconnect (HCAP), real-time tag (Channel Status = 1). Condition HCAP lost HCAL heartbeat for more than 20 seconds and values had been sent to storage since last disconnect. Value Last value sent to storage with current time stamp. Slide 13

MDAS2 in Replication (Historian 10) Slide 14

HCAL Components Historian Client Access Layer (HCAL) consists of the following: Historian Client Access Library (HCAL) aahclient.dll + aahclientcommon.dll, static libraries, native, and managed versions (SDK) HCAL Multicast Component (HCAL MCC) aahclientmcc.dll (Dual Historian Support) Historian Client Access Point (HCAP) - aahclientaccesspoint.exe (Historian and Redundant Platform) HCAL/HCAP is based on MDAS2/MDASServer. HCAL are used by the following applications Application Engine aahclientmcc.dll + aahclient.dll + aahclientcommon.dll for Historian access - aahclientmanaged.dll (SDK) as library for scripting. Replication Service (DLL) aahclient.dll, aahclientcommon.dll (retrieval and storage) OLEDB (Static Library) aahstaticclientnative.lib, aahstaticclientcommonnative.lib SDK (Managed DLL) aahclientmanaged.dll Slide 15

Connection To Historian Remote Computer Local Computer HCAP other Historian processes / subsystems HCAP other Historian processes / subsystems Client Application Named Pipe HCAL Slide 16 TCP Net TCP Port Sharing Service (SMSvcHost.exe)

Connection Management HCAL Historian connection status is determined by communication layer. Disconnected if: A function returns network related error or timed out for storage (Typically 1 minute, 10 seconds for sending values) Communication layer generates exception Ping and report tag count/data rate to Historian periodically (every 3 seconds) Similar mechanism is used to manage connections to local Storage Engine and standby Storage Engine HCAP Client considered disconnected if no pinging from client for more than 20 seconds. Slide 17

Store/forward Store/forward values will be flushed to disk and forwarded if HCAL is connected Historian, all tags had been synchronized, and also one of the following is true: No value has been sent to store/forward engine for more than 30 seconds. More than 1 hour since last flush. Snapshot memory consumption reaches 64 MB. The S/F data gets extracted from the S/F storage engine blocks by HCAL and sent as S/F snapshots to the historian (first to HCAP and then to Storage) then the snapshot gets marked as forwarded on the client. Eventually the whole orignial.dat file on the client side gets deleted when all its snapshots have been forwarded. Slide 18

Store/forward and Forwarding A lock file HCALStoreforward.lok in store/forward directory is used by HCAL for the following purposes: Ensure that the store/forward directory is used by HCAL exclusively. Contains base key number for local key generation (redundancy support). Passing parameters (e.g. partner Historian) across redundant platform. Information for diagnostics. (e.g. computer name, process id etc.) Store/forward engine flushes both tags and values to disk. Data file (original.dat) is deleted after forwarding completed. Tag information with server tag Id will remain and will be loaded when HCAL restarts. Need to delete the store/forward directory for a Clean start. Slide 19

Redundancy Support HCAL communicates with its partner store/forward engine through HCAP. One instance of HCAP manages all store/forward engines on redundant platform. Tags are always synchronized between partner store/forward engines, even when HCAL is connected to Historian. Values sent to local store/forward are also sent to partner store/forward engine. Before forwarding values to Historian, HCAL synchronizes values between the partner store/forward engines. After values are forwarded, snapshots on both store/forward engines are deleted. Slide 20

Redundancy Support c o n tr o l v ia T C P / IP a ah C lien tm C C.dll c o n tr o l v ia T C P / IP aahclient.dll + aahc lientcom m on.dll a a hc lie nt.dll + a a hc lie ntc o m m on.d ll HCAL A p pica tion S e rver E n g in e P roce ss B a c k u p P la tfo r m HCAL a a hc lie ntm C C.d ll A p pica tion S e rver E n g in e P roce ss P r im a r y P la tfo r m s ta r t / s to p / s to r e / g e t s n a p s h o ts / m e ta d a ta s ta r t / s to p / s to r e / g e t s n a p s h o ts / m e ta d a ta S to r e /fo r w a r d T 2 E C h ild P r o c e s s HCAP s ta r t / s to p / s to r e / g e t s n a p s h o ts / m e ta d a ta S /F d a ta S /F T 2 E d a ta Slide 21 HCAP s ta r t / s to p / s to r e / g e t s n a p s h o ts / m e ta d a ta S to r e / fo r w a r d T 2 E C h ild P r o c e s s S /F d a ta S /F T 2 E d a ta

Historian Project Tracking Provide statistics on general growth and health of the deployment of the system Useful to step back and appraise overall data collection application Can provide an executive dashboard for communication with customers, peers, management Slide 22

Statistics for Monitoring/Reporting Project Deployment Activity levels tasks/time metrics Resource consumption disk storage CPU (inferred from activity) license Utilization metrics defined vs. used Factory coverage number of sensors/tools Slide 23

At-a-Glance Dashboard Slide 24

Collecting the Statistics The AAA -related metrics are gathered from data tables maintained by the AAA application DefinedTags is SELECT COUNT(*) FROM TAG ValuesPerSecond is avg(value) of SysDataAcqOverallItemsPerSec sampled during the day Disk Space Free is the value of SysSpaceMain Other metrics Tags used inferred from daily history block s internal file Daily Storage is the size of daily history block Slide 25

Project Metrics as a Tool Forces you to think big picture Useful format for reporting to customers, operations management, systems management Format is familiar to management when budgeting for new hardware, licenses, software modules Useful when planning new deployments or applications How will the new project affect daily activity/consumption/performance? Becomes a common basis for discussion across operations, systems, support, and management Slide 26

Troubleshooting Tools Historian Client SQL Management Studio/Profiler SMC log flags Regedit Process Explorer Performance Monitor PSSDiag and PAL Tool Slide 27

SMC logger Connection Related Turn on Connection log flags to see WCF messages on client computer. Check warning messages on Historian computer. Could be caused by firewall blocking, incorrect port, invalid credentials etc. Tag Related Check to see if there is warning messages in the logger. Toggle TagDump log flags to see HCAL cached tags. Value Related Turn on BufferTransmit flag to see if all tags are out of store/forward Slide 28

SMC logger S/F - StoreForward log flag in the aahclientcommon section Turn on log flags for the right component Retrieval with OLEDB aahclientcommonnative Entry-Exit the hosting process, e.g. SqlServer, AppEngine Be aware of the user account the application is running with. HCAL might fail to write to the tag trace file. Slide 29

Perfmon Performance Troubleshooting - Memory Following counters should be added to perfmon DataCollection to identify the memory issue (the star indicates all counters in the class) Memory\Available KBytes Memory\Cache Bytes Memory\Commit Limit Memory\Modified Page List Bytes Memory\Pool Nonpaged Bytes Paging File(\??\C:\pagefile.sys)\* Process(*)\Private Bytes Process(*)\Working Set SQLServer:Buffer Manager\Free pages SQLServer:Buffer Manager\Page life expectancy SQLServer:Memory Manager\Target Server Memory (KB) SQLServer:Memory Manager\Total Server Memory (KB) Slide 30

PSSDiag/PALtool Performance Troubleshooting Slide 31

PSSDiag PSSDiag is an application that provides GUI to SQL server troubleshooting tools. It can be used to define e.g. trace templates or performance counters logs. This application is used by Microsoft support and is able to log 32 bit SQL server counters as well. For details on using PSSDiag see the link: http://blogs.msdn.com/b/pamitt/archive/2011/02/25/how-to-use-thesqldiag-the-sqlnexus-and-the-pal-tools-to-analyze-performanceissues-in-sql-server.aspx Slide 32

PSSDiag The tool is started by running Pssdiag.cmd that is found in CAB file created in the UI. By default it is found in C:\Program Files (x86)\microsoft\pssdiag\customer There is a little bug in the CMD file so it has to be edited in Notepad because the path is missing a backslash and also is not customized for 64 bit OS The line should look like this: if "%toolsbin%"=="" set toolsbin=c:\program Files (x86)\microsoft SQL Server\100\Tools\Binn\ Slide 33

PAL (Performance Analysis of Logs) Tool Slide 34

PAL (Performance Analysis of Logs) Tool Easy to use tool fo analyzing performance counter logs Uses predefined trashold files for different types of analyzes (eg. SQL Server, System Overview, Exchange Server...) For download and tutorials: http://pal.codeplex.com/ Slide 35

Values are not right, or? Slide 36

Slide 37

Values are not right Slide 38

Values are not right - tracing Slide 39

Values are not right - tracing Slide 40

Tracing Storage Slide 41

Tracing Storage Slide 42

Tracing Storage Slide 43

DEMO Slide 44

Question/ feedbacks/ request Thanks!!! Slide 45

Slide 46