SNMPv3 in Practice Workshop

Transcription

1 SNMPv3 in Practice Workshop 1

2 Objectives Configure Agents to use SNMPv3 Cisco IOS Juniper JunOS Net-SNMP Configure Managers for SNMPv3 Net-SNMP CLI Tools Net-SNMP Trap Daemon OpenNMS 2

3 More Objectives Configure USM (Authoritative) Engine ID Security Name Authentication Parameters Privacy Parameters Configure VACM Groups Views Access Rights 3

4 Even More Objectives Some minor and some not so minor quirks Troubleshooting tools Getting your hands dirty Programming SNMPv3 with Perl 4

5 JunOS: Configure SNMP Polling Steps to perform: Set up general parameters Set up an SNMPv3 user Set up a group for the user Define a MIB view Permit access 5

6 JunOS: Configure SNMPv3 Polling Define General SNMP Options [edit snmp] description "Juniper SRX100"; location "OUCE Lab"; contact ", engine-id { use-mac-address; 6 The first options should be defined for every SNMP agent. They does not serve any technical purpose but help a lot when one tries to figure out what the hell the box that OpenNMS happened to discover is doing The engine ID is a very central item in SNMPv3. It must be unique within a management realm, and it doesn t hurt to make it globally unique, as several company mergers should have taught everyone working in networking by now. There are several possible mechanisms to create the engine ID: It can be derived from an IPv4 or IPv6 address, from a MAC address or by manually setting it to a fixed string. By RFC 3411, the IPv4 or IPv6 address that has to be used in case of multiple network adapters is required to be the lowest non-special address on the system which obviously may change. You don t want that to happen. A better choice is to use the MAC address, as it tends to be more stable than any IP address. Again, the lowest MAC address on the system is to be used. Using a locally-defined string can be a good choice if one makes sure it is unique. Net-SNMP, for example, uses a combination of the time and a pseudo-random number by default. The format of the engine ID is a source of problems in practice... stay tuned.

7 JunOS: Configure SNMPv3 Polling Set up a Local Engine User [edit snmp v3 usm] local-engine { user OpenNMS-Monitor { authentication-sha { authentication-password "ouce2013-auth" privacy-aes128 { privacy-password "ouce2013-priv" 7 The local-engine user is used by remote management entities to authenticate against the local agent. There are some different options for the authentication digest and encryption algorithms, but currently using SHA for authentication and AES-128 for encryption is the best bet with respect to compatibility and security, as MD5 and (3)DES have already been proven insecure.

8 JunOS: Configure SNMPv3 Polling Add MIB Views (numeric) [edit snmp] view All-View { oid.1 include; view Restricted-View { oid include; oid exclude; view OpenNMS-View { oid include; oid include; oid include; oid include; oid include; oid include; oid include; oid include; oid include; oid include; 8 Views are used to limit the scope of variables within the OID tree that a manager can read or write to. There are two ways to specify views: Numerically (as shown here) or symbolically. Although at first glance it looks like the numeric specification is much harder to read, in practice it has its advantages: One can see the structure of data referred to by a view much easier in the numeric representation. JunOS permits both ways of entering OIDs, and keeps them in the configuration store as they have been entered, as opposed to Cisco IOS, which always converts everything to the symbolic representation.

9 JunOS: Configure SNMPv3 Polling Add MIB Views (symbolic) [edit snmp] view All-View { oid iso include; view Restricted-View { oid internet include; oid ipforward exclude; view OpenNMS-View { oid iso8802 include; oid system include; oid interfaces include; oid ip include; oid icmp include; oid tcp include; oid ospf include; oid dot1dbridge include; oid ifmib include; oid jnxoperatingentry include; 9 The same config in symbolic representation.

10 JunOS: Configure SNMPv3 Polling Create a Security Group [edit snmp v3 vacm] security-to-group { security-model usm { security-name OpenNMS-Monitor { group OpenNMS-Monitor-Group; 10 The next step is to assign a group name to the user (specifically for each security model, in case there is something else in use beside USM). Strictly speaking, the security-name does not have to be a user name as specified in USM, but can also be mapped to an SNMPv1/v2c community name. As we are talking about v3 only in this workshop, we ll resort to the shorter term user.

11 JunOS: Configure SNMPv3 Polling Permit Access [edit snmp v3 vacm] access { group OpenNMS-Monitor-Group { default-context-prefix { security-model usm { security-level privacy { read-view OpenNMS-View; 11 The last step is to define what the members of the group have the right to access in which way. In most cases, the default context prefix will be the only one in use (some exceptions will be mentioned later on). The security model should be USM if security is of concern, and in most cases the security level will be the highest one possible, which is privacy (that level enforces authentication as well). The read-view defines the objects the group has the right to access.

12 Use Net-SNMP to Query SNMPv3 Agents Net-SNMP is a free collection of SNMP tools on Unix/Linux Install the command line tools Set up a default configuration Perform queries against SNMP agents 12

13 Net-SNMP Example: CentOS Installation and Basic Setup ~]# yum -y install net-snmp net-snmp-tools... ~]$ cat ~/.snmp/snmp.conf defversion v3 defsecurityname OpenNMS-Monitor defsecuritylevel authpriv defauthpassphrase ouce2013-auth defprivpassphrase ouce2013-priv defauthtype SHA defprivtype AES defsecuritymodel usm mibdirs +/usr/share/snmp/mibs mibs +ALL 13

14 Net-SNMP Example: CentOS Sample snmpwalk on the SRX100 ~]$ snmpwalk SNMPv2-MIB::sysDescr.0 = STRING: Juniper SRX100 SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: ( ) 9:13:04.27 SNMPv2-MIB::sysContact.0 = STRING:, pe-ouce2013@hindenburgring.com SNMPv2-MIB::sysName.0 = STRING: ouce2013-juniper-srx100.example.com SNMPv2-MIB::sysLocation.0 = STRING: OUCE Lab SNMPv2-MIB::sysServices.0 = INTEGER: 4 IF-MIB::ifNumber.0 = INTEGER: 41 IF-MIB::ifIndex.4 = INTEGER: 4 IF-MIB::ifIndex.6 = INTEGER: 6... many, many OIDs... SNMPv2-SMI::enterprises = Gauge32: 0 SNMPv2-SMI::enterprises = Gauge32: 3 SNMPv2-SMI::enterprises = Gauge32: 0 SNMPv2-SMI::enterprises = No more variables left in this MIB View (It is past the end of the MIB tree) 14

15 Use OpenNMS to Monitor SNMPv3 Agents OpenNMS must be configured to support SNMPv3 by editing XML files GUI only knows SNMPv1/v2c communities, not USM Configuration is done by editing $OPENNMS_HOME/ etc/snmp-config.xml Restart OpenNMS after editing the configuration file Add node using any method (provisioning requisition, discovery, manual ) 15

16 Sample snmp-config.xml file <?xml version="1.0"?> <snmp-config retry="2" timeout="2000" version="v3" security-name="opennms-monitor" security-level="3" auth-passphrase="ouce2013-auth" auth-protocol="sha" privacy-passphrase="ouce2013-priv" privacy-protocol="aes" /> 16

17 Cisco IOS: Configure SNMPv3 Polling Steps to perform: Define general SNMP options Set up an SNMPv3 user and group Define a MIB view Permit access 17

18 Cisco IOS: Configure SNMPv3 Polling Define General SNMP Options snmp-server location OUCE Lab snmp-server contact, snmp-server ifindex persist 18

19 Cisco IOS: Configure SNMPv3 Polling Setup an SNMPv3 User Name snmp-server user OpenNMS-Monitor OpenNMS-Group v3 \ auth sha ouce2013-auth \ priv aes 128 ouce2013-priv 19 This command defines an SNMPv3 USM username, assigns a group name to it, and specifies authentication and encryption algorithms and passwords. The user name will not be visible in the IOS running-config after generation, but can be displayed using the EXEC command show snmp user.

20 Cisco IOS: Configure SNMPv3 Polling Display SNMPv3 Users in IOS ouce2013-cisco876#show snmp user User name: OpenNMS-Monitor Engine ID: B2BCD751F storage-type: nonvolatile! active Authentication Protocol: SHA Privacy Protocol: AES128 Group-name: OpenNMS-Group 20

21 Cisco IOS: Configure SNMPv3 Polling Add MIB Views (numeric) snmp-server view All-View 1 included snmp-server view Restricted-View included snmp-server view Restricted-View excluded snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included snmp-server view OpenNMS-View included 21 Note that in IOS the leading dot of an OID may not be specified! Otherwise, there are no surprises here.

22 Cisco IOS: Configure SNMPv3 Polling Add MIB Views (symbolic) snmp-server view All-View iso included snmp-server view Restricted-View internet included snmp-server view Restricted-View ipforward excluded snmp-server view OpenNMS-View iso included snmp-server view OpenNMS-View system included snmp-server view OpenNMS-View interfaces included snmp-server view OpenNMS-View ip included snmp-server view OpenNMS-View icmp included snmp-server view OpenNMS-View tcp included snmp-server view OpenNMS-View ospf included snmp-server view OpenNMS-View dot1dbridge included snmp-server view OpenNMS-View ifmib included snmp-server view OpenNMS-View cisco included 22 Nor here, except for the first OID, which is a mixture of symbolic and numeric representation, a result of a just partially known OID tree (due to a missing MIB in the router firmware).

23 Cisco IOS: Configure SNMPv3 Polling Permit Access snmp-server group OpenNMS-Group v3 priv match exact read OpenNMS-View 23

24 Net-SNMP: Configure SNMP Polling Steps to perform: Set up general parameters Set up an SNMPv3 user Define a group for the user Define a MIB view Permit access 24

25 Net-SNMP: Configure SNMP Polling Set up an SNMPv3 User ~]# service snmpd stop ~]# vi /var/lib/net-snmp/snmpd.conf... createuser OpenNMS-Monitor SHA ouce2013-auth AES ouce2013-priv... ~]# service snmpd start ~]# cat /var/lib/net-snmp/snmpd.conf... usmuser 1 3 0x80001f8880c4ce6b73e6c1cc x4f70656e4e4d532d4d6f6e69746f7200 0x4f70656e4e4d532d4d6f6e69746f7200 NULL x2dccf9a08560ec6a0c4c97b645e31ed3eb2d0d x9182b534b98f1446c db04dee0 ""... oldengineid 0x80001f8880c4ce6b73e6c1cc perl -e 'print pack "H*", "4f70656e4e4d532d4d6f6e69746f72";'

26 Configure SNMP Polling Alternate Way of Setting up USM Users ~]# cat /etc/snmp/snmpd.conf [...] group Security-Admin-Group usm Security-Admin [...] view Admin-View included view Admin-View included [...] access Security-Admin-Group "" usm authpriv exact Admin-View Admin-View none ~]# snmpusm create New-User Initial-User ~]# snmpusm -Ca -a SHA \ passwd initial-auth-password new-auth-password \ New-User [root@ouce2013-linux1 ~]# snmpusm -Cx -x AES \ passwd initial-priv-password new-priv-password \ New-User [root@ouce2013-linux1 ~]# snmpvacm \ createsec2group 3 New-User OpenNMS-Monitor-Group 26 The user executing the command must have write access to the usmusertable and the vacmseuritytogrouptable in order to create and modify users and add them to existing VACM groups. When a new user is created, it should be cloned from an existing initial user. Then the authentication and privacy passwords of the user must be changed using snmpusm -Ca -Cx. If the original password is entered wrong, the command will succeed, but the new password will not be usable. The snmpvacm command cannot be used to override any settings in the permanent configuration of, e.g. Net-SNMP. The whole mechanism only works over SNMPv3. Functionality of this way of configuring SNMPv3 USM/VACM is very dependent on the particular implementation on the SNMP agent s end. With a Net-SNMP agent it works without a flaw, with JunOS there are some timing issues but if suitable precautions are taken it is working as well, and Cisco IOS couldn t be persuaded to create users at all.

27 Net-SNMP: Configure SNMP Polling Sample Configuration File sysdescr CentOS 6.3 Virtual Machine for OUCE2013 Workshop syslocation OUCE Lab syscontact, dontlogtcpwrappersconnects true group OpenNMS-Monitor-Group usm OpenNMS-Monitor view All-View included.1 view Restricted-View included view Restricted-View excluded view OpenNMS-View included view OpenNMS-View included view OpenNMS-View included view OpenNMS-View included view OpenNMS-View included view OpenNMS-View included view OpenNMS-View included access OpenNMS-Monitor-Group "" usm authpriv exact OpenNMS-View none none 27

28 JunOS: Configure SNMPv3 Notifications Slightly different logic for traps and informs: Local engine for traps vs. remote engine for informs For Informs, the remote engine ID must be known or engine ID discovery must be implemented by the receiver 28 OpenNMS does not seem to implement RFC 5343 at this time. This is, however, a minor problem compared to the fact that the whole implementation of SNMPv3 informs in OpenNMS is broken. See JIRA issue NMS-2995 for more information. In short: OpenNMS can receive SNMPv3 informs properly, but sends corrupt confirmation packets. Since informs are all about getting confirmations for notifications, that makes it pretty useless.

29 JunOS: Configure SNMPv3 Traps Set up a Local Engine User [edit snmp v3 usm] local-engine { user OpenNMS-Trap { authentication-sha { authentication-password "ouce2013-trap-auth"; privacy-aes128 { privacy-password "ouce2013-trap-priv" 29 This is exactly the same setup we did for the SNMPv3 polling configuration. Since for SNMPv3 traps, the engine ID of the local node is used for encrypting the authentication and privacy passwords, a local engine user has to be created for this purpose.

30 JunOS: Configure SNMPv3 Traps Set up a Notification Filter [edit snmp v3] notify-filter OpenNMS-Notify-Filter { oid iso include; 30 Notify filters work in exactly the same way as views: A numeric or symbolic OID determines the subtree of trap OIDs that will be matched by the filter.

31 JunOS: Configure SNMPv3 Traps Set up Parameters for the Notification Target [edit snmp v3] target-parameters OpenNMS-Trap-Parameters { parameters { message-processing-model v3; security-model usm; security-level privacy; security-name OpenNMS-Trap; notify-filter OpenNMS-Notify-Filter; 31 The target parameters define the way the local engine communicates with the target system. Here we define the SNMPv3 message processing model with USM authentication, set the security level to authentication and privacy and define that authentication has to be done using the OpenNMS-Trap user defined earlier. The notify filter defined in the last step is also applied as part of the target-parameters.

32 JunOS: Configure SNMPv3 Traps Configure a Notification Object [edit snmp v3] notify Traps { type trap; tag Trap-Tag; 32 No, I really don t have an idea what serious use that could put into... type only accepts the values trap and inform, and basically this only inserts an additional level of naming.

33 JunOS: Configure SNMPv3 Traps Set up the Notification Target [edit snmp v3] target-address OpenNMS-Trap-Destination { address ; tag-list Trap-Tag; target-parameters OpenNMS-Trap-Parameters; 33 A target address specifies a trap (or inform) target. Actually the tag-list parameter seems to be fairly useless, since it is only possible to set a tag for traps and another for informs.

34 There is a JTAC PR open on this issue (PR844853). JunOS: Configure SNMPv3 Informs Configure the Remote Engine User [edit snmp v3 usm] remote-engine 80001F C29115B66 { user OpenNMS-Inform { authentication-sha { authentication-password "ouce2013-inform-auth"; privacy-aes128 { privacy-password "ouce2013-inform-priv" 34 The remote engine user must be configured specifically for each target system. For the authentication process to work, the engine ID of the target system must be known, either by using engine ID discovery, or by configuring it by hand. Either way the engine ID has a very annoying limitation: JunOS limits the length of the ID as input on the command line to 32 characters. Of these 16 bytes, the Engine ID format needs the first five for the longword specifying the enterprise ID of the engine (with the first bit set, e.g E20 for enterprise 15904) and the Engine ID format flag (e.g. 01 for IPv4 Address or 03 for MAC-Address). That leaves 11 bytes for the actual Engine ID. This is wrong in several ways with respect to RFC 3411, which specifies a length of octets for the Engine ID (textual-convention SnmpEngineID). It works fairly well for textual (type 4) Engine IDs, except that JunOS permits longer IDs than are allowed by the RFC, and it works fine for type 1 (IPv4 address) and type 3 (MAC address) IDs as well. But since an IPv6 address has 16 octets, a type 2 Engine ID is 21 bytes in length, and so it cannot be configured in JunOS. Additionally, there are some tools such as Net-SNMP that use a pseudo-random Engine ID by default, which in the case of Net-SNMP is octets long and cannot be configured in JunOS as well. OpenNMS uses the IPv4 address to generate the Engine ID by default.

35 JunOS: Configure SNMPv3 Informs Set up a Notification Filter [edit snmp v3] notify-filter OpenNMS-Notify-Filter { oid iso include; 35 This is exactly the same as for traps

36 JunOS: Configure SNMPv3 Informs Set up Parameters for the Notification Target [edit snmp v3] target-parameters OpenNMS-Inform-Parameters { parameters { message-processing-model v3; security-model usm; security-level privacy; security-name OpenNMS-Inform; notify-filter OpenNMS-Notify-Filter; 36 The same as well, just with the OpenNMS-Inform security-name instead of OpenNMS-Trap. Note that while OpenNMS-Trap and OpenNMS-Inform are not the same type of user (one is local, the other remote), they are referenced in exactly the same way within the targetparameters.

37 JunOS: Configure SNMPv3 Informs Configure a Notification Object [edit snmp v3] notify Informs { type inform; tag Inform-Tag; 37 Same as for traps, except the type is inform instead of trap

38 JunOS: Configure SNMPv3 Traps Set up the Notification Target [edit snmp v3] target-address OpenNMS-Inform-Destination { address ; tag-list Inform-Tag; target-parameters OpenNMS-Inform-Parameters; 38 And this is exactly the same as for traps as well. Note that when you set up SNMPv3 informs, JunOS will try to do engine ID discovery (by sending a get-request) to the remote agent exactly once. If that fails, it will not attempt to send any informs to that target-address for a long time (not sure how long, but too long when you re experimenting). This problem can be resolved by deactivating and reactivating the target-address in JunOS, committing the change after each step. And there s another little problem with SNMPv3 informs: The mechanism is broken in OpenNMS (OpenNMS Issue NMS-2995). You can, however, experiment with Net-SNMP s snmptrapd trap (and inform) daemon.

39 Cisco IOS: Configure SNMPv3 Notifications Again, informs and traps are treated in slightly different ways: Local engine for traps vs. remote engine for informs (it s part of the standard, after all) For Informs, the remote engine ID must be known IOS image restrictions apply not every feature set supports SNMP informs 39 OpenNMS does not seem to implement RFC 5343 at this time. This is, however, a minor problem compared to the fact that the whole implementation of SNMPv3 informs in OpenNMS is broken. See JIRA issue NMS-2995 for more information. In short: OpenNMS can receive SNMPv3 informs properly, but sends corrupt confirmation packets. Since informs are all about getting confirmations for notifications, that makes it pretty useless.

40 Cisco IOS: Configure SNMPv3 Traps snmp-server enable traps snmp snmp-server enable traps config... snmp-server user OpenNMS-Trap OpenNMS-Trap-Group \ v3 auth sha ouce2013-trap-auth \ priv aes 128 ouce2013-trap-priv snmp-server group OpenNMS-Trap-Group v3 priv snmp-server host traps version 3 priv OpenNMS-Trap config snmp 40 The snmp-server enable trap command specifies which traps are generated by IOS. There are some traps that will be generated if nothing is defined here, but the documentation is dim about which ones exactly that are linkup/linkdown traps are mentioned explicitly, as they can also be configured on interface level. The user will not be visible in the IOS running-config, but only via the show snmp user exec command. The group specifies which security model applies, and what level of security (none, auth, or priv) is required. Additionally it is possible to define a notify view for a group, but Cisco does recommend against it in normal cases. Since IOS controls which notifications are sent using different mechanisms, it would probably not be a good idea anyway. The last command specifies the host (and, optionally, the port) that notifications will be sent to, which security model and security level apply, and which user (or, with v1 and v2c, which community string) is used for authentication. At the end of the line the types of traps that are sent to this particular host must be listed. There can be multiple instances of this configuration directive. When the command is viewed in the configuration database, the trap keyword is omitted.

41 Cisco IOS: Configure SNMPv3 Informs snmp-server enable traps snmp snmp-server enable traps config... snmp-server engineid remote F C29115B66 snmp-server user OpenNMS-Inform OpenNMS-Inform-Group \ remote v3 \ auth sha ouce2013-inform-auth \ priv aes 128 ouce2013-inform-priv snmp-server group OpenNMS-Trap-Group v3 priv snmp-server host informs version 3 priv OpenNMS-Inform config snmp snmp-server manager 41 Again, the remote engine ID must be configured in order to send SNMP informs from a Cisco IOS box to a manager. The remote user is specific to a remote engine (specified by the IP address and optionally the UDP port of the manager). If there is no engine ID for a host/port combination in the configuration database, this command is discarded by IOS. The group setup is the same as for SNMPv3 traps. The snmp-server host command differs only in the use of the keyword informs in place of traps. The last command is snmp-server manager. If the image does not support this command, which enables an SNMP manager used to work as a proxy, receive traps and receive response PDUs for informs, all the commands above can be entered, but do not do anything useful - IOS does not send informs (neither SNMPv3 nor v2c), and it doesn t even tell you about it. Bad luck. E.g. IOS 12.4(24)T2 for the 876 in the lab has the feature in the Advanced IP Services feature set, but lacks it in the Advanced Enterprise set. Took me a while to figure that out, especially since at least half the documentation does not mention the need of explicitly starting the SNMP manager at all

42 Set up OpenNMS to Receive SNMPv3 Traps Again, editing XML files is necessary to make OpenNMS understand SNMPv3 traps There is no GUI for the trap receiver, as SNMPv1/v2c don t need much configuration anyway Configuration is done by editing $OPENNMS_HOME/ etc/trapd-configuration.xml Restart OpenNMS after editing the configuration file 42

43 Sample trapd-configuration.xml file (for traps and informs) <?xml version="1.0"?> <trapd-configuration snmp-trap-port="162" new-suspect-on-trap="false"> <snmpv3-user security-name="opennms-trap" security-level="3" auth-protocol="sha" auth-passphrase="opennms-trap-auth" privacy-protocol="aes" privacy-passphrase="opennms-trap-priv" /> </trapd-configuration> 43 No surprises here By the way, there is no way to configure SNMPv1/v2c communities in this file, because the trap receiver ignores the community string if incoming messages anyway. Try it out, this is a very good reason to use v3 only (just that you can t disable v1/v2c in OpenNMS). The only surprise is that SNMPv3 informs don t currently work as of OpenNMS version At all.

44 Net-SNMP: Configure the Trap Daemon Net-SNMP can be used as an SNMPv3 trap/inform receiver Useful for testing Can also be used to execute actions when traps are received Extremely flexible and (mostly) RFC compliant 44

45 Net-SNMP: Configure the Trap Daemon Set up SNMPv3 Users ~]# service snmptrapd stop ~]# vi /var/lib/net-snmp/snmptrapd.conf... createuser -e 80001f8880c4d035673a48d OpenNMS-Trap SHA ouce2013-trapauth AES ouce2013-trap-priv createuser OpenNMS-Inform SHA ouce2013-inform-auth AES ouce2013-inform-priv... ~]# service snmptrapd start ~]# service snmptrapd stop ~]# service snmptrapd start ~]# cat /var/lib/net-snmp/snmptrapd.conf... usmuser 1 3 0x80001f c29115b66 0x4f70656e4e4d532d496e666f726d00 0x4f70656e4e4d532d496e666f726d00 NULL xfb46b6b0fe0b329ca1cb8eb8912aab140122b xfb61068da944afbe8dbc3749c25f9e6d 0x usmuser 1 3 0x80001f8880c4d035673a48d x4f70656e4e4d532d x4f70656e4e4d532d NULL x0c2152dcd026ef2b5ed021d725312efab5d6aeae x3fb6e8797fbdfadf6baca2c8d724659f "" Very similar to snmpd, but with a different persistent config file. There is a fundamental difference between configuring a user for SNMPv3 traps and informs on the target system: In order to receive a trap, the engine ID of the sending system must be used to encrypt the passwords, because with traps, the sending engine is authoritative. Since Net-SNMP encrypts the user names when it stores them, the engine ID to use for encryption has to be known at configuration time. Eventually this makes SNMPv3 informs easier to configure than SNMPv3 traps Strangely enough, snmptrapd needs two restarts as opposed to snmpd to write its encrypted configuration users into the configuration file. It should be done, however, to avoid to have the clear text passwords on disk until the next reboot occurs.

46 Net-SNMP: Configure the Trap Daemon Sample Configuration File engineidtype 3 group OpenNMS-Trap-Group usm OpenNMS-Trap group OpenNMS-Inform-Group usm OpenNMS-Inform view All-View included.1 setaccess OpenNMS-Trap-Group "" usm authpriv exact log All-View setaccess OpenNMS-Inform-Group "" usm authpriv exact log All-View 46 The engineidtype 3 statement makes the trap demon derive the engine ID from the MAC address instead of the default Net-SNMP mechanism of using a pseudo-random octet string as the engine ID. This is important e.g. when JunOS clients will try to send informs, which because of a bug in JunOS does not work with the 17 byte random engine IDs Net-SNMP uses Use setaccess instead of access and don t rely on the Net-SNMP documentation on the format of that statement, because it has the last two parameters swapped. After changing the configuration file, restart the SNMP trap daemon. All traps received will be logged into the file the daemon uses for its messages, in RHEL/CentOS 6.3 that is /var/log/ messages.

47 Net-SNMP: Configure the Trap Daemon Sample Output Dec 17 03:09:07 ouce2013-linux1 snmptrapd[32708]: :09:07 <UNKNOWN> [UDP: [ ]:64391->[ ]]:#012DISMAN-EVENT- MIB::sysUpTimeInstance = Timeticks: ( ) 3 days, 4:21:52.45#011SNMPv2- MIB::snmpTrapOID.0 = OID: IF-MIB::linkUp#011IF-MIB::ifIndex.511 = INTEGER: 511#011IF- MIB::ifAdminStatus.511 = INTEGER: up(1)#011if-mib::ifoperstatus.511 = INTEGER: up(1)#011if-mib::ifname.511 = STRING: fe-0/0/2 Dec 17 03:09:07 ouce2013-linux1 snmptrapd[32708]: :09:07 <UNKNOWN> [UDP: [ ]:64391->[ ]]:#012DISMAN-EVENT- MIB::sysUpTimeInstance = Timeticks: ( ) 3 days, 4:21:52.54#011SNMPv2- MIB::snmpTrapOID.0 = OID: IF-MIB::linkUp#011IF-MIB::ifIndex.515 = INTEGER: 515#011IF- MIB::ifAdminStatus.515 = INTEGER: up(1)#011if-mib::ifoperstatus.515 = INTEGER: up(1)#011if-mib::ifname.515 = STRING: fe-0/0/ The engineidtype 3 statement makes the trap demon derive the engine ID from the MAC address instead of the default Net-SNMP mechanism of using a pseudo-random octet string as the engine ID. This is important e.g. when JunOS clients will try to send informs, which because of a bug in JunOS does not work with the 17 byte random engine IDs Net-SNMP uses Use setaccess instead of access and don t rely on the Net-SNMP documentation on the format of that statement, because it has the last two parameters swapped. After changing the configuration file, restart the SNMP trap daemon. All traps received will be logged into the file the daemon uses for its messages, in RHEL/CentOS 6.3 that is /var/log/ messages. The log format output can be customised using the -F command option or the format1 and format2 configuration file options.

48 Net-SNMP: Sending Traps and Informs The Net-SNMP CLI tools snmptrap and snmpinform can be used to send traps and informs (who d have thought?) Useful for testing Very handy for job monitoring with OpenNMS 48

49 Net-SNMP: Sending Traps and Informs Sending an SNMPv3 trap snmptrap -v3 \ -u OpenNMS-Trap \ -l authpriv \ -a SHA -A ouce2013-trap-auth \ -x AES -X ouce2013-trap-priv \ Sending an SNMP trap is very straightforward, it s receiving it where the problems start OpenNMS stores SNMPv3 authentication and privacy passwords in clear text, so it doesn t have to know the engine ID of an agent sending SNMPv3 traps to it. RFC 3414, Section 11.2: Note however, that if user's password is disclosed, then key localization will not help and network security may be compromised in this case. Therefore a user's password or non-localized key MUST NOT be stored on a managed device/node. Instead the localized key SHALL be stored (if at all), so that, in case a device does get compromised, no other managed or managing devices get compromised. There is some room for interpretation, though the RFC explicitly speaks of a managed device in the part where storing non-localised keys or passwords, and of managed or managing devices later on. So in terms of RFC conformance, OpenNMS ought to be on the safe side. Net-SNMP on the other hand applies the RFC statement much more stringently and doesn t store non-localised keys on the managing device (snmptrapd is a management application, not an agent) as well.

50 Net-SNMP: Sending Traps and Informs Sending an SNMPv3 inform (the easy way) snmpinform -v3 \ -u OpenNMS-Inform \ -l authpriv \ -a SHA -A ouce2013-inform-auth \ -x AES -X ouce2013-inform-priv \ Yes, this works in many cases without knowing the remote Engine ID! The Net-SNMP CLI tools implement USM engine ID discovery. Unfortunately, using the process described in RFC 3414 requires that the non-localised authentication and privacy passwords for the account on the managing node (the node receiving the informs) are stored on the managed node (the node sending the informs), which, when applied for informs, violates RFC 3414 as seen earlier on, so strictly speaking the protocol should not be used for informs. But it is more comfortable, I ll grant that.

51 Net-SNMP: Sending Traps and Informs Sending an SNMPv3 inform (the RFC way) snmpinform -v3 \ -u OpenNMS-Inform \ -l authpriv \ -e 80001f c29115b66 \ -a SHA -A ouce2013-inform-auth \ -x AES -X ouce2013-inform-priv \ If engine ID discovery is not implemented on the receiving end, the engine ID of the inform receiver must be known when sending informs to it. The good thing is that with informs, all configuration is on the sending side, and the configuration is the same for all senders. With traps, the receiver has to have the localised key for every sending instance, even if they all use the same SNMPv3 user name, hash/encryption algorithms and passwords. So having to store the engine ID of the receiver as well is a minor effort, and the whole system is much easier to maintain.

52 Programming SNMPv3 A Perl Example It s easy to program SNMPv3 in Perl Modules are freely available The methods are very simple and logical Useful for notifications in own applications or management scripts 52

53 Programming SNMPv3 in Perl Prerequisite Modules Net::SNMP Crypt::DES Digest::MD5 Digest::SHA1 Digest::HMAC Crypt::Rijndael (for AES) Socket6 (for IPv6) 53 The modules can be installed using CPAN (if you have the time and resources), or as binaries from the repository of your choice (e.g. EPEL for RedHat/CentOS). Despite the name, the Net::SNMP module does not have a connection to Net-SNMP that would be the SNMP module. Obvious, isn t it?

54 Programming SNMPv3 in Perl Setting up the Session #!/usr/bin/perl use strict; use warnings; use Net::SNMP; my ($session, $error) = Net::SNMP->session( -hostname => ' ', -port => 162, -nonblocking => 0, -version => 3, -domain => 'udp/ipv4', -timeout => 10, -retries => 5, -username => 'OpenNMS-Inform', -authprotocol => 'sha1', -authpassword => 'ouce2013-inform-auth', -privprotocol => 'aes', -privpassword => 'ouce2013-inform-priv' ); 54 This very simplistic example shows how to set up an SNMPv3 session. The authentication and encryption parameters should be well known by now. It is possible to use Net::SNMP in a much more sophisticated way, e.g. using the calls in nonblocking mode, running SNMP over TCP or over IPv6 and so forth. But this example is meant to be a simple starting point, so I left all those niceties aside. Note that although we want to send an SNMPv3 inform, we did not specify the engine ID. The simple reason for this is that with Net::SNMP you currently can t - if the receiving side doesn t support RFC 5343, replace the -authpassword and -privpassword parameters with -authkey and -privkey and use the localised keys instead of the passwords. There is a utility snmpkey which can be used to generate them from the passwords and the engine ID, and because of RFC 3414 it is formally required to do so anyway.

55 Programming SNMPv3 in Perl Sending an Inform if (defined $session) { my $sys_uptime_oid = ' '; my $snmp_trap_oid = ' '; my $example_oid = ' '; my $varbind_list = [ $sys_uptime_oid, TIMETICKS, 0, $snmp_trap_oid, OBJECT_IDENTIFIER, $example_oid ]; $session->inform_request( -varbindlist => $varbind_list ); else { die "Could not open SNMPv3 session: $error\n"; $session->close; exit; 55 If the NET::SNMP->session call was successful, we now initialise the data to send within the inform and then send it. I chose a very simple inform OID for this example that doesn t take any parameters. If there are parameters, they are simply appended as triples (OID, type, value) to the $varbind_list array. The inform_request method sends the inform to the manager, and it is good style (though not strictly necessary) to close the session before exiting the program. This example uses an inform for three reasons: They are actually easier to set up, there is a confirmation of receipt, and Net::SNMP does not support traps.

56 Programming SNMPv3 in Perl Getting the Value of SysOID use Data::Dumper; if (defined $session) { my $sys_contact_oid = ' '; my $sys_name_oid = ' '; my $varbind_list = [ $sys_contact_oid, $sys_name_oid ]; my $result = $session->get_request( -varbindlist => $varbind_list ); print Dumper $result; else { die "Could not open SNMPv3 session: $error\n"; $session->close; exit; 56 Obviously the session now has to contain the credentials for an SNMP agent on a managed device, but apart from that everything is the same as for an SNMP inform request, so we leave it as an exercise. The get_request itself is even simpler than the inform_request. The $varbind_list just contains a number of OIDs that we re interested in, and the data structure that is returned is a hash containing key/value pairs with the passed OIDs as the keys as shown in the Data::Dumper output.

57 Programming SNMPv3 in Perl Getting the Value of SysOID (cont d) perl-test]#./snmp-get.pl $VAR1 = { ' ' => 'ouce2013-juniper-srx100.example.com', ' ' => ', pe-ouce2013@hindenburgring.com' ; 57 That s all there is to say nearly.

58 Troubleshooting SNMPv3 Some methods to troubleshoot SNMPv3 Using Net-SNMP-Tools (with or without debug) Enabling SNMP4J debugging in OpenNMS Using tcpdump (which is of limited use with encrypted protocols) Using Wireshark (which is much better if you know how to configure it) 58 We ve seen what the Net-SNMP tools can be used for, so we skip directly to SNMP4J.

59 Troubleshooting SNMPv3 Enabling Debugging in SNMP4J ~]# vi /opt/opennms/etc/log4j.properties... # SNMP4J internal logs (if enabled in opennms.properties) log4j.category.org.snmp4j=debug, SNMP4J-INTERNAL log4j.additivity.org.snmp4j=false log4j.appender.snmp4j-internal=org.apache.log4j.rollingfileappender log4j.appender.snmp4j-internal.maxfilesize=100mb log4j.appender.snmp4j-internal.maxbackupindex=2 log4j.appender.snmp4j-internal.file=/opt/opennms/logs/daemon/snmp4j-internal.log log4j.appender.snmp4j-internal.layout=org.apache.log4j.patternlayout log4j.appender.snmp4j-internal.layout.conversionpattern=%d %-5p [%t] %c: %m%n 59 Change the value of log4j.category.org.snmp4j from WARN to DEBUG. It is not necessary to restart OpenNMS, log4j will pick up the change after a few moments and start writing debug messages into the log file specified in log4j.appender.snmp4j-internal.file.

60 Troubleshooting SNMPv3 Sample SNMP4J Output ~]# less /opt/opennms/logs/daemon/snmp4j-internal.log :22:05,604 DEBUG [DefaultUDPTransportMapping_ /0] org.snmp4j.security.privaes: aes decrypt: used key 91:82:b5:34:b9:8f: 14:46:c8:98:62:12:db:04:de:e :22:05,604 DEBUG [DefaultUDPTransportMapping_ /0] org.snmp4j.security.privaes: aes decrypt: used privacy_params 8b:77:7a: 53:c3:35:16:c7... StateReference[msgID= ,pduHandle=PduHandle[ ], securityengineid=80:00:1f:88:80:c4:ce:6b:73:e6:c1:cc:50:00:00:00:00, securityname=opennms-monitor,securitylevel=3, contextengineid=80:00:1f:88:80:c4:ce:6b:73:e6:c1:cc:50:00:00:00:00,contextname=] What we see here is a debug message showing an access to an SNMP agent, listing the engine IDs and keys as well as the user name used to connect to the agent.

61 Troubleshooting SNMPv3 Using tcpdump ~]# tcpdump -i eth0 -Tsnmp port 1162 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size bytes 03:36: IP lagavulin.hindenburgring.com > ouce2013- linux1.example.com.health-trap: F=r U= E= C= GetRequest(14) 03:36: IP ouce2013-linux1.example.com.health-trap > lagavulin.hindenburgring.com.58869: F= U= E= 0x800x000x1F0x880x030x000x0C0x290x110x5B0x66 C= Report(31) S:snmpUsmMIB.usmMIBObjects.usmStats.usmStatsUnknownEngineIDs.0=10 03:36: IP lagavulin.hindenburgring.com > ouce2013- linux1.example.com.health-trap: F=apr U=OpenNMS-Inform [!scoped PDU]f4_8a_fa_66_4d_74_c0_af_90_bd_16_69_1c_fd_e7_58_1c_02_09_c1_89_23_0d_c3_f9_1e _aa_da_b4_f6_b9_ce_33_7b_4a_73_e3_f6_8e_eb_18_c0_a0_8c_e5_ca_49_ec_28_4e_98_ee_50 _ff_a7_3e_1a_ef_b7_5c_2c_c6_59_b5_b6_1e_06_a6_e6_73_38_6a_60_15_b1_75_7a_19 03:36: IP ouce2013-linux1.example.com.health-trap > lagavulin.hindenburgring.com.58869: F=ap U=OpenNMS-Inform [!scoped PDU]4b_fb_b8_7d_31_e0_ee_cd_45_39_55_c8_e0_44_c1_59_f4_17_17_c2_6e_82_2f_2b_e1_6d _84_ee_7d_b4_6f_83_8f_4d_dd_c4_05_26_e3_eb_0f_f1_d4_e0_22_77_e4_20_3f_bf_b0_98_6d _d8_07_0d_8a_ba_6f_50_57_ca_55_f3_06_ea_0b_90_9f_1f_b8_51_1b_4c_3d_27_df_a7 61 You have to be root to use tcpdump. The tool can in most cases only be used to verify that there is SNMP traffic and who participates in it. For encrypted connections, the payload of the logged packets is virtually useless, so we can limit the output to the header fields and the first few bytes of data. If the port we re interested in is not one of the standard ports 161 or 162, the option -T snmp makes tcpdump interpret the packets as SNMP and show some additional header fields that are useful. The output shown is the trace of an SNMPv3 inform request from host lagavulin.hindenburgring.com on the non-standard port 1162 using engine ID discovery (the unencrypted GetRequest and subsequent Report PDU), the actual Inform and the Response PDU. The latter two cannot be decrypted, as the PDU including its headers is part of the encrypted payload. F=apr means authentication + privacy, reportable, the response just has authentication and privacy flags set. The user name used to authenticate was OpenNMS- Inform.

62 Troubleshooting SNMPv3 Using Wireshark Wireshark uses the same packet capturing mechanism as tcpdump, but Wireshark provides a GUI making it easier to visualise large amounts of data Wireshark can decode OIDs provided it has the MIBs Wireshark can decrypt SNMPv3 packets provided the parameters are known! 62 Basically, the technology Wireshark uses is the same as for tcpdump, called libpcap. But in addition to providing a more or less user friendly GUI for the packet capturing functionality, Wireshark has some additional features that make it very useful for the analysis of SNMPv3 traffic: It can decode the packets, provided the credentials used for the authentication and encryption of the packets are known. In addition, one can tell Wireshark to decrypt the OIDs after decrypting, which makes it very easy to get a grasp of what happens on the Network.

63 The Wireshark GUI 63 ok, this slide is pretty useless

64 Configure Wireshark to Decrypt SNMPv3 Packets 64 Click on Edit/Preferences, unfold Protocols in the left sidebar and open SNMP. There you ll find a button named Users Table, which gets you here. For SNMP Get/Set and Trap packet analysis enter the engine ID of and the user name and authentication/privacy parameters for the target host, for SNMP Informs the engine ID of the source host and user name and authentication/privacy parameters for the target host. After confirming the configuration dialogs, Wireshark will decode all the packets and also show encrypted information from the PDU headers.

65 Configure Wireshark to Decrypt SNMPv3 Packets 65 Click on Edit/Preferences, unfold Protocols in the left sidebar and open SNMP. There you ll find a button named Users Table, which gets you here. For SNMP Get/Set and Trap packet analysis enter the engine ID of and the user name and authentication/privacy parameters for the target host, for SNMP Informs the engine ID of the source host and user name and authentication/privacy parameters for the target host. After confirming the configuration dialogs, Wireshark will decode all the packets and also show encrypted information from the PDU headers.

66 Configure Wireshark to Decode OIDs 66 Click on Edit/Preferences, and click on Name Resolution in the sidebar. Select Enable OID resolution and, if necessary, add required MIB module directories by adding MIB paths under SMI (MIB and PIB) paths.