The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri March 2015

Transcription

1 The Aruba Tech Support Top 10 Tips Tarun George & Gowri Amujuri March 2015

3 Aruba OS Segmental Troubleshooting AP Stability and System profile Optimization Optimize load on processes Datapath Debugging Deployment Tips 3 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

5 Segmental Troubleshooting Segmental Troubleshooting gains. # Faster root cause analysis. # One time Data Collection Transition Content # Bring focus on the smallest segment in the network within our control. 5 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

7 Segmental Troubleshooting User show tech-support user mac <Mac Address> Transition Content tar logs user mac <User Mac > tech-support User Debugging Logging Level debugging user-debug <Mac Address> 7 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

8 Segmental Troubleshooting AP Transition Content show ap tech-support ap-name <Name of AP> show ap debug counters show ap bss-table ap-name show ap debug system-status ap-name 8 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

9 Segmental Troubleshooting Controller show tech-support tar log tech-support Transition Content Outside world Debugging for Specific process/sub-cat.. (Explained) Pcap show interface gigabitethernet <slot/module/port> Network Diagram Note: Show tech-support <filename> Store output in file. 9 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

10 Segmental Troubleshooting.. Processes show process monitor statistics Transition Content Process Monitor Statistics Name State Restarts Allowed Restarts Timeout Value Timeout Chances Time Started /mswitch/bin/dbstart PROCESS_RUNNING Sat Feb 28 21:31: /mswitch/bin/packet_filter PROCESS_RUNNING Sat Feb 28 21:31: Mdns, httpd_wrap, Authmgr,STM, WMS, cfgm, dhcp 10 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

12 AP Stability and System Profile Optimization AP System Status. Campus and Remote APs have similar challenges to stay connected to the controller. Health Check of the AP is vital, since it can trigger client and controller anomalies. Show AP debug system-status ap-name <Name of AP> 12 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

13 AP Stability and System Profile Optimization Reboot Information DHCP/Controller/Keep Alive miss Rebootstrap Information Date Time Reason (Latest 10) LMS Change/Heartbeat Miss HA Failover Information Date Time Reason (Latest 10) 13 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

14 AP Stability and System Profile Optimization Recent Control Messages from AP to Controller Date Time Message Description Sun Mar 1 12:29: (164 secs ago): SENT REQ type=keepalive len=45 peer= seq_num=4567 num_attempts=1 rtt=0 secs Rebootstrap LMS Crash Information 14 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

15 AP Stability and System Profile Optimization CPU and Memory Usage Timestamp CPU Util(%) Memory Util(%) :32: Peak CPU Util in the last one hour Timestamp CPU Util(%) Memory Util(%) :19: CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

16 AP Stability and System Profile Optimization Heartbeat Stats of Serving Controller Heartbeats Sent Sent Seqnum Heartbeats Received Rcvd Seqnum MTUs sent Misc sent Measurement Duration since last rebootstrap n/a n/a total since bootup Interface counters Interface Rx_pkts Rx_errors Rx drops Tx_pkts Tx_errors Tx_drops Resets wifi wifi CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

17 AP Stability and System Profile Optimization MTU Discovery Probes Responses Last Sent Last Rcvd Switch MTU, 1500 Ethernet bonding SlaveId Name Link State #LinkFails Ethernet Duplex/Speed Settings Autoneg Speed (Mbps) Duplex Iface 0 eth0 UP ACTIVE 0 on 1000 Full eth0 eth1 DOWN STANDBY 0 on 10 Half eth1 17 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

18 AP Stability and System Profile Optimization Controller Information Item Value Primary LMS Backup LMS AP to Active Controller Message Information Item Value AP state REGISTERED Power Status Operational State : Unknown Current HW State POE-AT: No restrictions 18 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

19 AP Stability and System Profile Optimization MTU Discovery Probes Responses Last Sent Last Rcvd Switch MTU, 1500 Ethernet bonding SlaveId Name Link State #LinkFails Ethernet Duplex/Speed Settings Autoneg Speed (Mbps) Duplex Iface 0 eth0 UP ACTIVE 0 on 1000 Full eth0 eth1 DOWN STANDBY 0 on 10 Half eth1 19 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

21 AP Stability Optimizations Heartbeat DSCP: Assign a DSCP value to AP heartbeats to prioritize heartbeats traveling over low-speed links. The supported range is 0-63, and the default value is 0. Bootstrap threshold: Number of consecutive missed heartbeats on a GRE tunnel (heartbeats are sent once per second on each tunnel) before an AP rebootstraps. On the controller, the GRE tunnel timeout is 1.5 x bootstrap-threshold; the tunnel is torn down after this number of seconds of inactivity on the tunnel. SAP MTU: Maximum Transmission Unit, in bytes, on the wired link for the AP. Spanning Tree: Select this checkbox to enable the Spanning Tree protocol. 21 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

23 HTTPD Stress on the webserver can be because of number of sessions in the initial role for guest access. This is either because of large certificate (Key Length 2048 or 4096) that are used by the server or a large number of devices (phones/tablets with APPs) that generate HTTP/HTTPS sessions and get re-directed to the web-server. show web-server profile Web Server Configuration Parameter SSL/TLS Protocol Config Captive Portal Certificate Value tlsv1 GUEST-AUTH User session timeout < > (seconds) 3600 Maximum supported concurrent clients <25-320> 75 Enable WebUI access on HTTPS port (443) Enable bypass captive portal landing page true false 23 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

24 HTTPD show web-server statistics Web Server Statistics: Current Request Rate: 1 Req/Sec Current Traffic Rate: 1 KB/Sec Busy Connection Slots: 7 Available Connection Slots: 68 Total Requests Since Up Time: 284 Total Traffic Since Up Time: 1122 KB Avg. Request Rate Since Up Time: 1 Req/Sec Avg. Traffic Rate Since Up Time: 6144 Bytes/Sec Server Scoreboard: KKKKKK_W Scoreboard Key: _ - Waiting for Connection, s - Starting up R - Reading Request, W - Sending Reply K - Keepalive, D - DNS Lookup C - Closing connection, L - Logging G - Gracefully finishing, I - Idle cleanup of worker. - Open slot with no current process 24 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

25 STM and WMS STM Station Management is responsible for all AP information and Station information. This process can be over run if there is, # Aggressive polling from Airwave/SNMP servers for Wlan tables. # Network wide AP reboot and bootstraps # AP debug scripts run from the controller. WMS IDS/IPS events and frequent AP bootstraps could lead to WMS being busy. WMS is actively looking for RF information of WiFi devices(rogue/valid/interfering). 25 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

26 Mitigation Use AMON WMS Offload to Airwave. Reduce SNMP polling or increase the polling period Disable WMS functionality if you do not require IDS/IPS functionality. 26 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

28 Datapath Monitoring Show datapath utilization show datapath utilization Datapath Network Processor Utilization Cpu utilization during past Cpu 1 Sec 4 Secs 64 Secs 10 99% 99% 99% 11 0% 0% 0% 12 0% 0% 0% 13 0% 0% 0% 14 0% 0% 0% 15 0% 0% 0% 16 0% 0% 0% show datapath frame 10 SUM/ CPU Addr Description Value [00] Allocated Frames [01] Max Allocated Frames [03] Unknown Unicast [34] Flood Frames [00] Rx Frames [01] Rx Bytes [02] Tx Frames CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

29 Datapath Bandwidth Management show datapath bwm Datapath Bandwidth Management Table Entries Type Id Bits/sec Policed Bytes Bytes Flags CPU Status /0 9 ALLOCATED /0 9 ALLOCATED /0 9 ALLOCATED /0 9 ALLOCATED /0 9 ALLOCATED /0 9 ALLOCATED Firewall: Rate limit CP untrusted ucast traffic Rate limit CP untrusted mcast traffic Rate limit CP trusted ucast traffic Rate limit CP trusted mcast traffic Enabled 20 Mbps Enabled 4 Mbps Enabled 160 Mbps Enabled 4 Mbps /0 9 ALLOCATED Rate limit CP route traffic Enabled 2 Mbps Rate limit CP session mirror traffic Enabled 2 Mbps Rate limit CP auth process traffic Enabled 2 Mbps 29 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

30 CP and DP Packet Capture Wifi -Client packet-capture datapath wifi-client aa:aa:aa:aa:aa:aa all VIA client/rap packet-capture datapath ipsec <peer-ip> Generic traffic to controller packet-capture controlpath tcp/udp CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

32 Deployment Tips Honey Comb Pattern Wireless Local Probe Threshold = 25 Transmit Power of AP 5Ghz Min Tx 12 Max Tx 15 Avoid Asymmetric RF 2.4 Ghz Min Tx 6 Max Tx 9 The difference between minimum and maximum Tx power on the same radio should not be more than 6dbm DMO Enable Basic and Beacon rate a 5Ghz g 2.4Ghz Mhz Channel bonding - DFS Channels 32 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

33 Deployment Tips.. Contd GRE Stripping IP - VRRP for LMS and Stripping IP Jumbo Frames - Enabled 802.3at Airgroup Dot1x OKC Validate PMK ID r/k/v EAPOL Rate Optimization 33 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

36 Network Services AirWave ClearPass 36

38 ClearPass Platform: System Cleanup Options Free disk space threshold is a config in Cluster Wide Service Parameter. Default 30% A system cron job runs every hour and checks the disk utilization. If the free space falls below the configured threshold, an alert is logged into the system. NOW in addition, the following aggressive cron cleans up anything more than 1 day old in version 6.5 of CPPM Log database records Core files System load monitor files Application and system log files Auto and manual backup files Stored reports Expired guest accounts Audit records #ATM15 38 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

40 ClearPass Platform: System Cleanup Options We also introduced some new CLI commands Check on disk-space and memory usage - show sysinfo system cleanup [# of days to retain] **This is an on-demand task #ATM15 40 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

43 ClearPass Platform : Graphite Graphite is a new reporting tool to compliment Insight in CPPM from 6.3 version. Graphite runs on every node irrespective of standalone or cluster and statistics can be viewed from any node. Performance monitoring Display is disabled by default and should be enabled manually and set access permission levels accordingly. To access Graphite data, use the URL IP Address>/graphite #ATM15 43 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

46 ClearPass Platform : Graphite We can allow or deny any networks to access Graphite for a node or cluster. Make sure stats collection is set true True under Service parameters. #ATM15 46 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

50 ClearPass : Upgrade Utility Tool The Cluster Upgrade Tool is a simple user interface that automates the upgrade procedure for a ClearPass cluster. When the upgrade is initiated, no manual actions are required until all selected nodes have been upgraded. The Upgrade Tool is not available while the publisher is rebooted and migrating the Configuration Database. The Upgrade Tool will not detect nodes that were upgraded manually without the tool. If a configured standby publisher node was manually upgraded without the tool, the Upgrade Tool will not restore the state of the standby publisher configuration. #ATM15 50 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

51 ClearPass : Upgrade Utility Tool The Cluster Upgrade Tool is released as a patch update. It can be downloaded and installed either through Policy Manager s Software Updates portal, or from the Aruba Support portal. #ATM15 51 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

52 ClearPass : Upgrade Utility Tool Log in to Policy Manager on the publisher node and go to Administration > Agents and Software >Updates > Software Updates. When the installation is complete, the Admin service will be restarted. You do not need to reboot. #ATM15 52 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

53 ClearPass : Upgrade Utility Tool Before you begin the upgrade, the upgrade image must be present on the publisher node of the cluster. Download the upgrade image to the publisher under Software updates #ATM15 53 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

54 ClearPass : Upgrade Utility Tool To monitor the progress of the other nodes in the cluster, wait until the database migration is complete and then log in to the tool again. Change the url to IP Address/upgrade We should see all the subscribers status that are in sync. The list of subscribers will be present and subscriber upgrades will go in parallel. Start Upgrade to start the upgrade on the servers. #ATM15 54 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

57 ClearPass : Upgrade Utility Tool Check the logs for each node by View Logs next to each node and we can see the progress of patches and upgrades from publisher. #ATM15 57 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

60 AirWave VisualRF Performance Tips Sometimes we see VisualRF takes long time to show up new AP s to deploy on the floor plans. We can manually force VisualRF to poll the AP s to get new AP or existing AP s updated details. NO need to restart VisualRF to show up new AP s. Change the url in AMP to IP Address>/visualrf/poll_aps_now.xml #ATM15 60 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

61 AirWave VisualRF Performance Tips In 8.x moved to HTML5 for VisualRF UI for faster UI interaction with backend. Whilst we are improving the features on the new UI, there are some features which were present in flash and not in HTML5. In VisualRF > Setup page, we can switch between HTML5 and flash so that we can take advantage of options present in both version Switching between HTML5 and Flash version is easy with below URL without refreshing VisualRF. Change the URL to IP Address/site?campus_id=6c56c239-bfba-4d19-aeca-8ec5af68b725 from IP Address/vrf?campus_id=6c56c239-bfba-4d19-aeca-8ec5af68b725 #ATM15 61 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

64 AirWave VisualRF Performance Tips We can change vrf to site on any page for VisualRF URL s to switch to flash mode from HTML5 mode. At times we see Heat maps not showing/updating properly in VisualRF We can resize the floor plan to same size so that the grid calculation happens and heat maps will be re-drawn. No need to restart VisualRF for heat maps to update. #ATM15 64 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

66 Airwave VisualRF Performance Tips Select the distance, click OK and Save without changing the distance, this will trigger floor plan to recalculate the heatmpas. #ATM15 66 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

68 Airwave Tips for Data Retention Settings Data in AirWave is primarily stored in 2 formats: Postgres - an open source, relational SQL database. Usually, when you see data in tables, that data is stored in Postgres. RRD Files - used for storing data that's displayed in time-sequence graphs (i.e, client count over the last year, bandwidth used over the last month). There can be many thousands of RRD files on a single AirWave server. One benefit of RRD is that its files have a fixed size. As data is inserted to an RRD file (like by an AirWave monitoring process), it does not grow. A downside of this is that the file starts using storage space as soon as it is created. #ATM15 68 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

69 Airwave Tips for Data Retention Settings We can set data retention settings under AMP Setup > General page under the section Historical Data Retention. Client Association and VPN Session History. This setting has a bearing on how much history we can show in the association history on the client historical table and how much data can be included in the user session data. Its recommend to keep high because the data is useful #ATM15 69 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

70 Airwave Tips for Data Retention settings Inactive Client and VPN User Data. This setting determines how long we keep the information on every client that has ever connected to the network. This impacts how long we keep RRD files. Keeping it low can save disk space. #ATM15 70 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

71 Airwave Tips for Data Retention settings Client data retention Interval : This influences how much historical data you can see for each client in the graphs, for example the signal quality, usage graphs on the client detail/diagnostic page. It's very important to keep this low, like in the days range. This is especially important in public wi-fi deployments that will have lots of unique users. This setting controls what size RRD files are created to store per-user historical signal, usage, goodput, health and other metrics. Keeping it low doesn't impact device, group and folder-level monitoring, and it doesn't have any negative impact on reports. It only impacts the graphs on the client Detail and Diagnostic pages. #ATM15 71 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

72 Airwave Tips for Data Retention settings By default Rogues are kept forever which will impact the overall system performance and for RAPIDS page load. It also impacts VisualRF for Rogue calculation if it has thousands of Rogue devices. This is especially important in public wi-fi deployments that have open SSID and lot of nearby devices are detected as Rogues Setting the value to low as 14 days will greatly help. #ATM15 72 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

73 Airwave Tips for Data Retention settings Airwave by default has 20+ Reports which runs daily. Keep Reports that are needed for the environment and delete/disable the default reports. Report retention setting can be costly in high dense environments especially for disk space. This increase the nightly backup file size, nightly maintenance time and report generation time. Exporting the reports via.csv or.pdf or ing them is a good option. Keeping the retention value will have the pickled client tables not to grow huge in size and makes report generation faster. #ATM15 73 CONFIDENTIAL Copyright Aruba Networks, Inc. All rights reserved

75 THANK YOU 75