WW TSS 01 -Network Best Practices & Troubleshooting

Slide 1

WW TSS 01 -Network Best Practices & Troubleshooting Pierluigi Iodice social.invensys.com @InvensysOpsMgmt / #SoftwareRevolution Regional Solution Support Engineer, /InvensysVideos Global Customer Support /Wonderware Wonderware /company/wonderware Email: pierluigi.iodice@invensys.com 2013 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys or its subsidiaries. All third party trademarks and service marks are the proprietary marks of their respective owners.

Networking How we proceed Understanding Understanding - This section contains information to figure how some tools I selected could be helping out to understand the network issues with Wonderware products. Troubleshooting Network Monitoring Slide 3 Troubleshooting - This section contains notes and information on how we solved some experienced issue about network, working with Wonderware products. Monitoring -This section includes a deep dive about some Advanced tools that could help us to understand the system behavior, working with Wonderware products.

Understanding Summary 1. Understand all the Network Types, especially for Wonderware Product destination. 2. Conduct an Inventory of all your network Locations and Bandwidth. 3. Test Name Resolution from all network location to Servers Node, ensuring that both Clients and Servers resolve Hostname. 4. Test ICMP Connectivity from all network to Servers Node, ensuring that Clients can reach Servers and Server to Server. 5. Test Routes in all Nodes, understanding only how the Metric is working. 6. Test TCP/IP Reliability from all locations to Servers, ensuring that connections made can reliably be established and maintained. Slide 4

Understanding : Domain vs Workgroup 1. Understand all the Network Types, especially for Wonderware Product destinations. Windows has two modes of operation: Workgroup and Domain. Depending on the environment that your computer is in, you will be running in one of these two modes. Most home and small business environments will be Workgroup, Most mid - to large businesses will run in Domain mode. There are different features and capabilities depending on each, and each serve a purpose Workgroups can be best understood as a loosely connected group of computers. There is no centralized management and so there is a low barrier to use. Domains, on the other hand, provide centralized management and security. User access is controlled from a separate server called a domain controller and there is a trust built between systems in a domain. There are much more robust differences as well. Slide 5

Understanding : Domain vs Workgroup 1. Understand all the Network Types, especially for Wonderware Product destinations. Workgroup : A workgroup is best understood as a peer-to-peer network. That is, each computer is sustainable on its own. It has its own user list, its own access control and its own resources. In order for a user to access resources on another workgroup computer, that exact user must be setup on the other computer Slide 6

Understanding : Domain vs Workgroup 1. Understand all the Network Types, especially for Wonderware Product destinations. Domain: A domain is a trusted group of computers that share security, access control. It has data passed down from a centralized domain controller server or servers. Domain Controllers handle all aspects of granting users permission to login. They are the gatekeeper. In addition, most modern domains use Active Directory which allows an even more centralized point for software distribution, user management and computer controls. Slide 7

Understanding Draw a Structure 2. Conduct an Inventory of all your network locations and bandwidth. The best would be to have a detailed map of all related infrastructures (Servers, Workstations, Ethernet Cables, Fiber Optics, Switches, Routers, Firewalls, etc.) as well as the hosting software (Operating systems, Virus protection, Intrusion protection, etc.). Preparing a slide documentation like the above. Slide 8

Understanding How to do it 2. Conduct an Inventory of all your network locations and bandwidth. Inventory of Network Locations When planning for a Wonderware implementation, it is important to create an inventory of all locations from which users will access Server and Client The goal of this inventory is to gather information about the Intranet connectivity and capacity from each network location: Corporate, Private, Supervisory, RMC,... When conducting an inventory, include the following information about each network location: The location s name and a description of its Internet access. Example: AOS, HISTORIAN, HMI, SCADA, DAS, TSE,... Intranet bandwidth average and peak usage. I.E: 50% average usage, 70% peak usage. Number of proxy servers, and current average and peak usage. Number of firewall appliances, and current average and peak usage. Number of DNS servers, and current average and peak usage. Once you have collected this information for every network location, use the data to assess current capacity, and whether any upgrades are needed Slide 9

Understanding Bandwidth 2. Conduct an Inventory of all your network locations and bandwidth. How retrieve bandwidth average usage peak: Iperf: Iperf is a neat little tool with the simple goal of helping administrators measure the performance of their network. Worthy of mention is the fact that it can measure both TCP and UDP performance on a network. Iperf is cross platform software and open source. Slide 10

Understanding Identifying a Machine 3. Test Name Resolution from all network locations. Name Resolution: is used on networks to resolve names to IP addresses. You and I use names and words to communicate, but computers use numbers. Name resolution methods are tied to 2 types of names: Hostname: A host name can t be up to 255 characters in length and is the only type of name used on the Internet. When a hostname is combined with a domain name, it becomes a fully qualified domain name (FQDN). For example, a Windows 7 PC named Client1 in the domain corp.com has an FQDN of Client1.corp.com. Hostnames are primarily resolved by DNS servers. NetBIOS: names A NetBIOS name has 15 readable characters, with the 16th byte identifying a service running on the system. The use of NetBIOS names has been significantly reduced in networks in favor of hostnames, but they are still being used by older applications. Since NetBIOS names are not supported in IPv6, this usage will eventually disappear. NetBIOS names are primarily resolved by WINS servers. Slide 11

Understanding Host Name 3. Test Name Resolution from all network locations. Host Name Resolution Methods: While DNS is the primary method used to resolve hostnames, it's not the only method... There are 3 primary methods used to resolve hostnames: 1) Domain Name System 2) Host file 3) Host cache Slide 12

Understanding DNS 3. Test Name Resolution from all network locations. DNS Resolution : DNS servers answer queries for name resolution of hostnames. When queried with a name, the DNS server returns the IP address. DNS servers are typically configured with addresses of other DNS servers. If the queried DNS server does not know the IP address, it will forward the name resolution request to other DNS servers to determine the IP address. This forwarding occurs on internal networks and on the Internet. You can easily check which DNS server is assigned to a client with the IPConfig /All command. Slide 13

Understanding Host check 3. Test Name Resolution from all network location. DNS Test: NSLookup Although the IPConfig and Ping commands can very often be useful when troubleshooting name resolution issues, there are times when you'll want to query the DNS server directly to get detailed information. The NSLookup tool is a command-prompt utility that you can use to get specific information from a DNS server. DNS Test Advanced: DiG - is a Domain Information Groper command-line tool for querying DNS name servers for information about host addresses, mail exchanges, name servers, and related information. Dig is useful for network troubleshooting and for educational purposes. Dig can operate in interactive command line mode or in batch mode by reading requests from an operating system file. When a specific name server is not specified in the command invocation, it will use the operating systems default resolver, usually configured via the resolv.conf file. Without any arguments it queries the DNS root zone. DIG could totally replace NSLookup nslookup and the host program. Slide 14

Understanding Host file 3. Test Name Resolution from all network locations. Host files: The Hosts file is located in the %windir%\system32\drivers\etc folder. Entries in the Hosts file are automatically placed into the host cache. Malware sometimes modifies the Hosts file to prevent a client from accessing specific websites. For example, a bogus entry could be placed in the file for Microsoft's update site, and the client would no longer be able to get updates. Slide 15

Understanding Host cache 3. Test Name Resolution from all network locations. Host cache: Once a name is resolved by DNS, the result is placed in the host cache (also called the DNS cache, which is a little misleading because this cache also holds entries from the Hosts file). You can view the host cache with the IPConfig /DisplayDNS command! You can remove host cache entries from cache using the IPConfig /FlushDNS command. This will remove all entries that were cached from a DNS query, but it will not remove entries placed in cache from the Hosts file. Slide 16

Understanding Host Name 3. Test Name Resolution from all network location. NetBIOS Name Resolution Methods: NetBIOS names can be resolved using three methods. These methods are different from the primary methods used to resolve hostnames. However, if a hostname can't be resolved using the primary hostname resolution methods, you can attempt the NetBIOS name resolution methods. There are 3 primary methods used to resolve NetBIOS names: 1) WINS OLD When queried with a name, the WINS server returns the IP address. Whereas DNS servers can be configured to query other DNS servers to resolve a name, WINS servers cannot query other WINS servers. 2) LMHost file is located in the %windir%\system32\drivers\etc folder. 3) NetBIOS cache Once a name is resolved by WINS, the result is placed in the NetBIOS cache. You can view the NetBIOS cache with the NBTStat /c command. Slide 17

Understanding ICMP 4. Test ICMP Connectivity from all network to Servers Node, and vice-versa. ICMP Connectivity: Internet Control Message Protocol which is the ability to send an echo packet to a destination host and a mechanism to listen for a response from this host. Simply stated, if the requesting host receives a response from the destination host, this host is reachable. ICMP messages are typically used for diagnostic or control purposes or generated in response to errors in IP operations (as specified in RFC 1122). ICMP errors are directed to the source IP address of the originating packet.. There are 3 primary methods used to test ICPM: 1) Ping 2) Tracert 3) PathPing Slide 18

Understanding ICMP Ping 4. Test ICMP Connectivity from all network to Servers Node, and vice-versa. ICMP Test: Ping is the most commonly used network tool. This utility is used to provide a basic connectivity test between the requesting host and a destination host. You can use this PING command to provide a basic picture of where a specific networking problem may exist. For example, if an Internet connection is down at one Node. TTL Time-to-Live is really important to understand how many hops (router i.e.) the packet can cross Slide 19

Understanding ICMP Traceroute 4. Test ICMP Connectivity from all network to Servers Node, and vice-versa. ICMP Test: Tracert, once the ping utility has been used to determine basic connectivity, the tracert/traceroute utility can used to determine more specific information about the path to the destination host including the route the packet takes and the response time of these intermediate hosts. You can use this TRACERT command to provide a basic picture of how the datagrams are going to be sending over a specific networking, understanding if a problem may exist. For example, if a router is down hindering the connection at one Node. You can use also to figure the slowest network pipe. Slide 20

Understanding ICMP PathPing 4. Test ICMP Connectivity from all network to Servers Node, and vice-versa. ICMP Test: PathPing, In an effort to take advantage of the benefits of both the ping and tracert/traceroute commands. You can use this PATHPING command to take the advantages of PathPing over ping and traceroute are that each node is pinged as the result of a single command, and that the behavior of nodes is studied over an extended time period, rather than the default ping sample of four messages or default traceroute single route trace. Slide 21

Understanding Route and Metric 5. Test Route table in all Node, understanding only how the metric is working. Route Table: In computer networking a routing table is a data table stored in a router or a networked computer that lists the routes to particular network destinations, and in some cases, metrics (distances) associated with those routes. The routing table contains information about the topology of the network immediately around it. The construction of routing tables is the primary goal of routing protocols. This would be an example of Route Table: Net. Dest. Net. Mask Gateway Interface Metric 0.0.0.0 0.0.0.0 192.168.0.1 192.168.0.100 10 127.0.0.1 255.0.0.0 127.0.0.1 127.0.0.1 1 192.168.0.0 255.255.255.0 127.0.0.1 127.0.0.1 100 192.168.0.100 255.255.255.255 127.0.0.1 127.0.0.1 10 Network Destination and Netmask together describe the Network id. For example, destination 192.168.0.0 and netmask 255.255.255.0 can be written as network id 192.168.0.0/24. Gateway column contains the same information as the Next hop, i. e. it points to the gateway through which the network can be reached. Interface indicates what locally available interface is responsible for reaching the gateway. i.e. gateway 192.168.0.1(the internet router) can be reached through the local network card with address 192.168.0.100. Finally, the Metric indicates the associated cost of using the indicated route. This is useful for determining the efficiency of a certain route from two points in a network. i.e. it is more efficient to communicate with 127.0.0.1 ( localhost ) than it would be through 192.168.0.100. Slide 22

Understanding Route and Metric 5. Test Route table in all Node, understanding only how the metric is working. Route Table Test: ROUTE PRINT is a command to display the route table: Slide 23

Understanding Metric 5. Test Route table in all Node, understanding only how the metric is working. Metric: A metric is a value that is assigned to an IP route for a particular network interface that identifies the cost that is associated with using that route. Automatic Metric is a feature in Windows that automatically configures the metric for the local routes that are based on link speed. The Automatic Metric feature is enabled by default, and it can also be manually configured to assign a specific metric.: Slide 24

Understanding TCP/UDP on IP 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/IP Reliability: Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP) were the first networking protocols defined in this standard of communication thru Internet Protocol (IP). Both TCP and UDP work on a specific Port Originally, these port numbers were used by the Network Control Protocol (NCP) and two ports were needed as transmission was done at half duplex. As TCP and UDP were adopted, only one port was needed. The even numbered ports were dropped. This is why the even numbers on well known port numbers are unassigned. We could simply consider the TCP/UDP as the Postal Message, IP as Addressee, TTL how many max hops the message can thru and PORT as the Receiver Mailboxes Slide 25

Understanding TCP/UDP pipeline 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Reliability: Ensure that clients in your network can reliably establish and maintain a connection to Wonderware App. Servers. Information Client InTouch Historian Client Historian Server RMC AOS AOS Data Server AOS Data Server RMC-Redundancy Management Channel Failover w/ Load Balancing Information Server Slide 26 GR / IDE Note: TCP/UDP reliability tests are intrusive and can affect network performance. Run these tests during off-hours to gather data while causing the minimum possible the impact to your network.

Understanding TCP/UDP Port 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Port: Every Node on the network must have a unique number assigned to it called the IP address, that is used to recognize your Node among the others. When information is sent over the Network to your computer how does your computer accept that information? It accepts that information by using TCP or UDP Ports. You can have a total of 65,535 TCP Ports and another 65,535 UDP ports. Another easy way to understand ports is to imagine your IP address is a cable TV box and the ports are the different channels on that cable box. The cable company knows how to send cable to your cable box based upon a unique serial number associated with that box (IP Address), and then you receive the individual shows on different channels (Ports). Each single Application can handle one or more ports to communicate with others Slide 27

Understanding TCP/UDP and Ports 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Port: NETSTAT is a Command Prompt command that you can use to show up TPC/UDP all the estabilished connection over some specific port, and all the Open Port that are waiting for a connection. Slide 28

Understanding TCP/UDP and Ports 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Port: NETSTAT with NETSTAT -o can show al which PID still handling the port: Slide 29

Understanding TCP/UDP and Port 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Port: NETSTAT with NETSTAT -b can show directly the Applications are still handling the port: Slide 30

Understanding TCP/UDP Protector 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Protector: Firewall Even if a Port is open, the connection can be denied! Firewall can either be software-based or hardware-based and is used to help keep a network secure. 1. Many computer operating systems include software-based firewalls to protect against threats from the public Internet, included Ms Window our base OS. 2. Could be there also other 3rd part software installation like Norton, McAfee and son on 3. In addition IT or Network Administrator could implement some router/firewall hardware All Wonderware Product needs to be excluded from Firewall protection. Slide 31

Understanding TCP/UDP Protector 6. Test TCP/IP Reliability from all locations to Servers, and vice-versa. TCP/UDP Protector: Firewall Some Port that need to be excluded from: Function 1st Connection 2nd Connection Remarks CIFs TCP 445 Out File serving, deploying. IDE WAS NETBIOS TCP 139 Out Server Message Block (SMB). WAS Domain Controller(s) NMXSVC TCP 5026 Out A2 Channel: Peer-to-Peer, bi-directional between all ArchestrA-enabled nodes. RPC DCE TCP 135 Out DCOM: Peer-to-Peer, bidirectional between all ArchestrA-enabled nodes. RPC Dyn Range TCP 6000-6050 Out *Custom range. Peer-to-Peer, bi-directional between all ArchestrA-enabled nodes. SQL Server TCP 1433 In/Out Inbound: SQL Server Client, and outbound: Client SQL Server SUITELINK TCP 5413 Out LDAP TCP 389 Active Directory Domain, Cient Domain Controller(s) KERBEROS TCP 88 OS Authentication NETBIOS UDP 137 Tx/Rx SQL Browser UDP 1434 Tx/Rx Only if implementing SQL Server instances NTP UDP 123 Tx/Rx Time Synchronization Client Domain controller(s) or Time Master DNS UDP 53 Slide 32 TCP 1024-65000 UDP 138 Tx TCP 53 SuiteLink: InTouch, IO Server communication. NetBios Name Service WAS WINS/Browser/Domain Master Domain Name Service, Client DNS Server

Understanding End! And now? Slide 33

Troubleshooting Summary 1. Possible issues about Network Types. 2. Possible issues about Location and Bandwidth. 3. Possible issues about Name Resolution. 4. Possible issues ICMP Connectivity. 5. Possible issue about Routing table 6. Possible issues about TCP/IP Reliability. How to solve the issue using the Understanding we got!... And now? Slide 34

Troubleshooting Network Type 1. Possible issues about Network Types. Network Type: DOMAIN Major issue we were experienced about the Domain Networks could be addressed to a Local IT that handle the Machine Local Security Policy ISSUE: Failed to deploy Platform SR 103127740 / CR L00125270 Slide 35

Troubleshooting Network Type 1. Possible issues about Network Types. Network Type: DOMAIN Major issue we were experienced about the Domain Networks could be addressed to a Local IT that handle the Machine Local Security Policy App. Server as well as communicate with Client node, need to register the components using DCOM access, so it need to logon on the machine to do that, Local IT to prevent unauthorized access tend to restrict the following Local Policy: Allow/Deny access from network Allow/Deny log on locally Local It has changed User with Power User Slide 36

Troubleshooting Network Type 1. Possible issues about Network Types. Network Type: DOMAIN Often IT doesn t accept changes, so far we need to find out a workaround: In this case we put the Authenticated User into Power User group Slide 37

Troubleshooting Estimate Bandwidth 2. Possible issues about Location and Bandwidth. Bandwidth: Major issue we were experienced about the Orange Icon was addressed to a excessive band usage, ISSUE: Orange Icon SR xxx / CR xxx Slide 38

Troubleshooting Estimate Bandwidth 2. Possible issues about Location and Bandwidth. Bandwidth: NetLimiter - To replicate the issue, I was using this Software, and I have put the Bandwidth like a GSM, 32Kbit/s, further detail about this SW later in Monitoring section. Slide 39

Troubleshooting Estimate Bandwidth 2. Possible issues about Location and Bandwidth. Bandwidth: IPERF - Iperf command was being used to state the Bandwidth boundaries: Slide 40

Troubleshooting Estimate Bandwidth 2. Possible issues about Location and Bandwidth. Bandwidth: SMC - In addition, enabling DeployStatus flag we have recognized that system was busy yet, copying the InTouch file: Slide 41

Troubleshooting Estimate Bandwidth 2. Possible issues about Location and Bandwidth. Bandwidth: Unlock Bandwidth - In this case I can, often we cannot!: Slide 42

Troubleshooting Host resolving 3. Possible issues about Name Resolution. Name Resolution: HOST Name Major issue we were experienced about the Host resolution could be addressed to a Multiple Network interface present on the system, in this case The Supervisory Network is unable to work properly due a corporate Network DNS ISSUE: Unable to resolve Host SR common issue Slide 43 / CR no CR related

Troubleshooting Host resolving 3. Possible issues about Name Resolution. Name Resolution: So far Wonderware Information Server can not be reached by Client using the Hostname, work only with IP Address: Slide 44

Troubleshooting Host resolving 3. Possible issues about Name Resolution. Name Resolution: HOSTS The issue has been addressed to a missing NetBIOS option enabled Supervisory Network of WIS Node, so far one way to solve the issue has been edit Hosts file: Slide 45

Troubleshooting ICMP 4. Possible issues ICMP Connectivity. ICMP : Sometime we were experiencing some LMX warning message, but apparently no issue has occurred, but a continuously spam message would be better to analyze. ISSUE: Platform x exceed max heartbeat timeout Slide 46 SR 103126814

Troubleshooting ICMP 4. Possible issues ICMP Connectivity. ICMP : This would be the scenario I have found i.e. on Historian Node: Slide 47

Troubleshooting ICMP 4. Possible issues ICMP Connectivity. ICMP : Ping Running ping command we figured that in round an ICMP request will be sent on both Control and Supervisory, in round robin,: Slide 48

Troubleshooting ICMP 4. Possible issues ICMP Connectivity. ICMP : WireShark/NetMonitor These tools would be really helpful to understand the ICMP walkthrough, this topic will treated later in the monitor section: Slide 49

Troubleshooting ICMP 4. Possible issues ICMP Connectivity. ICMP : Double gateway issue So far in Windows having a double gateway means that you would have a Network Load Balance, in which and packet will be send round robin on both interface, crossing through the two different gateway: Gateway need to be set only when the Supervisory Network can have other clients in al different network, sometime would be best to enlarge the subnet mask to keep together instead to hop to another network! Slide 50

Troubleshooting Route and Metric 5. Possible issue about Routing table. Routing table: Metric Major issue we were experienced about the Multiple Network Adapter, so far there might be a wrong routing addressing for the Supervisory Network. ISSUE: While starting up, view.exe hangs 5-15 minutes while loading WWSQL SR 103127614 When InTouch window is starting up, a custom script try to connect some tags on SQL Sever on Rachet node. In the Log we will see the connection fail. Slide 51

Troubleshooting Route and Metric 5. Possible issue about Routing table. Routing table: Metric Despite we have the Binding List in a proper manner, the request come thru a Corporate Network instead of Supervisory Corporate network can access on Internet, so can have a FIREWALL! Slide 52

Troubleshooting Route and Metric 5. Possible issue about Routing table. Routing table: Metric Getting a look on route table we ll see that Corporate Network have a low Metric: So we could increase the Corporate Metric, and decrease Supervisory to choose a fast track! Slide 53

Troubleshooting TCP/UDP 6. Possible issues about TCP/IP Reliability. TCP/UDP issue: Firewall Several issues related to TCP/UDP reliability could be addressed at closed firewall port. ISSUE: Unable to connect Logger SR 34113469 Slide 54

Troubleshooting TCP/UDP 6. Possible issues about TCP/IP Reliability. TCP/UDP issue: NetStat -a aalogger Service To share the logger information it will be used the TCP 49156, So netstat was used to understand the port status on Historian Node. Slide 55

Troubleshooting TCP/UDP 6. Possible issues about TCP/IP Reliability. TCP/UDP issue: ProcessMonitor Assuming that the port is listening without any active connection estabilished, we have to test the all pipeline, understanding if the communication was blocked. We could use Process Monitor tool from Microsoft Sysinternals, to figure what SMC (mmc.exe) was trying to do with TCP protocol: Filter: Process Name = mmc.exe Operation >= TCP Slide 56

Troubleshooting TCP/UDP 6. Possible issues about TCP/IP Reliability. TCP/UDP issue: ProcessMonitor Below the results, trying to open historian node with SMC: This meant that Rachet (Historian Node) can be reachable at epmap port (135 DHCP server, or DNS server, or WINS server) but not at 49156 used by SMC Then the client (Ironide) was still trying to reconnect until a timeout will be expired. Slide 57

Troubleshooting TCP/UDP 6. Possible issues about TCP/IP Reliability. TCP/UDP issue: Firewall For sure a firewall in somewhere was denying the access at 49156 port, so far we have enabled to log some additional information about the firewall behavior: Slide 58

Troubleshooting TCP/UDP 6. Possible issues about TCP/IP Reliability. TCP/UDP issue: Firewall That is the root cause, the firewall was dropping all the connection for the logger! Slide 59

Troubleshooting End! And now? Slide 60

Monitoring Summary Advanced Monitoring, only overview: A. Using: Performance Monitor B. Using: TCPView C. Using: Process Monitor D. Using: NetMon/Wireshark Slide 61

Monitoring Performance Monitor A. Using Performance Monitor Object Counter Description Performance Monitor expose several pen to draw a network trend, here below the most helpful:connections Established TCPv4 The number of simultaneous connections supported by TCP. This counter displays the number of connections last observed to be in the ESTABLISHED or CLOSE-WAIT state. Network Interface Bytes Received/Sent The cumulative total number of bytes received/sent. Web Service Max. Connection Cur. Connection The maximum/current number of simultaneous connections to the Web service Active Server Pages Requests/sec The number of ASP page requests executed per second. Use this number to provide an indication of how heavy the stress on the Web server is. SQLServer User Connections Shows the number of active SQL users. Each connection uses 37 KB of memory. Compare this number to the Active Server Pages :Requests/sec counter to get an idea of how much the scripts are really working the SQL server. A large difference may indicate that the test script is not a valid stress of SQL server. SQLServer NetNetwork Reads/sec Shows the number of data packets read from the network. An extremely high value for an extended time indicates that either the network card has a problem, or more likely, the application is not using enough stored procedures and is written improperly. Slide 62

Monitoring Performance Monitor A. Using Performance Monitor Slide 63

Monitoring TCPView for TCP/UDP B. Using TCPView TcpView - TCPView is a Windows program that will show you detailed listings of all TCP and UDP endpoints on your system, including the local and remote addresses and state of TCP connections. Slide 64

Monitoring TCPView for TCP/UDP B. Using TCPView TcpView - TCPView report SQL Server was consuming lot of network send/receive How could be possible that we had two instance of SQL Server that were flooding the communication, looked like a nested loop, on port 49398??? Slide 65

Monitoring ProcessMonitor on TCP C. Using Process Monitor ProcessMonitor- Process Monitor is an advanced monitoring tool for Windows that shows realtime file system, Registry and process/thread activity. It capture several information including Network communication. So we started it with the following filter: Slide 66

Monitoring ProcessMonitor on TCP C. Using Process Monitor ProcessMonitor- Analyzing the results, we can figure out several request and response coming across Sql Server and on the same node: Slide 67 49398 MS-SQL MS-SQL 49398

Monitoring ProcessMonitor on TCP C. Using Process Monitor ProcessMonitor- We could get a look at properties of call, and having a deep about the all the processes involved on, we will see that also Historian was involved: Slide 68

Monitoring ProcessMonitor on TCP C. Using Process Monitor ProcessMonitor- We could get a look at Network Summary to discover all the client that was querying at this Server: Slide 69

Monitoring ProcessMonitor on TCP C. Using Process Monitor ProcessMonitor- With netstat we knew that Client node (ironide) was calling at MSSQLServer Port(1433) trhought his own port (52408 and 52409): Slide 70

Monitoring ProcessMonitor on TCP C. Using Process Monitor? ProcessMonitor- Using process monitor on ClientIronide node (ironide) we wil discover who was calling at MSSQLServer Port(1433) trhought his own port (52408 and 52409): Trend: 52408/9 Rachet: 49398 Rachet: MS-SQL Rachet:MSSQL Slide 71 Rachet: 49398

Monitoring ProcessMonitor on TCP C. Using Process Monitor ProcessMonitor- With netstat we knew that Server node (rachet) was responding at Ironide Client node (ironide) over 1433 Port (MSSQLServer): Rachet:1433 Trend: SqlServer 52408/9 Rachet: 49398 Rachet: MS-SQL Rachet: MS-SQL Slide 72 Rachet: 49398

Monitoring Wireshark D. Using NetMon/Wireshark Whireshark - Wireshark is a network protocol analyze, It allows the live capture of network traffic and includes many powerful tools to examine, analyze, filterwith and otherwise manipulate this blank we willthe let data. WS capture all the traffic, in order to refine it later. Slide 73

Monitoring Wireshark D. Using NetMon/Wireshark Wireshark We let WS running for several seconds in order to gather the following: Slide 74

Monitoring Wireshark D. Using NetMon/Wireshark Wireshark Now we have to figure something about the content of the call, filtering all the call to SQLServer coming through a port 52408/9: Slide 75

Monitoring Wireshark D. Using NetMonitoring/Wireshark Wireshark Then we followed all the stream to recap the information exchanged: Historian Query We could also filter out this Stream Slide 76

Monitoring Why? The Answer Thanks to Monitoring: Just FYI Historian AnalogHistory view use the OLEDB linked Server, and this meant a double call on Server Node to provide the answer at client: Slide 77

Monitoring End! And now? Slide 78

Question/ feedbacks/ request Thanks!!! Slide 79

Slide 80