Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Channels ▼
RSS

SNMP Trap Handling with Nagios


SNMP Trap Handling with Nagios

Francois Meehan

My company has been very successful in providing network-monitoring solutions based on Nagios/Netsaint. For environments that required SNMP trap handling, however, the technique suggested in Nagios's documentation was too cumbersome. That technique requires coding for each individual trap message that needs to be monitored and, for those clients, we could only suggest making use of commercial solutions, all of which come with high price tags and complicated implementations.

Recently, however, we discovered Alex Burger's "SNMP Trap Translator" project that extends Net-SNMP. Coupled with Risto Vaarandi's event correlation tool called "SEC", a small Python script, and, of course, Nagios, we have put together a very scalable, efficient alternative. The whole process is pictured in Figure 1.

Please note that we used Red Hat advanced server for this particular installation, but this solution should be adaptable to other modern Linux distributions.

Pre-requisites:

1. Net-SNMP with snmptrapd configured.
2. SNMPTT, SNMP trap translator.
3. SEC -- Simple event correlator.
4. Nagios.
5. Mib definition files for the equipment or software you need to monitor.

One of the beauties of this solution is that we can use the event severity set by the mib designer. Nagios will always report the event status based on this information.

Net-SNMP

Net-SNMP, formally known as UCD-SNMP, is installed by default on most Linux distributions. Here we are specifically interested in configuring the trap receiver portion of the installation. The trap receiver is a daemon that receives its startup configuration in /etc/rc.d/init.d/snmptrapd. We modified the following line:

OPTIONS="-s -u /var/run/snmptrapd.pid"
To this:
OPTIONS="-On -u /var/run/snmptrapd.pid"
As quoted from SNMP Trap Translator documentation: "The -On is recommended. This will make snmptrapd pass OIDs in numeric form and prevent SNMPTT from having to translate the symbolic name to numerical form." We then modified the file /usr/share/snmp/snmptrapd.conf by adding the following line:
traphandle default /usr/sbin/snmptt
When the modification is done, the deamon must be restarted for the changes to take effect. SNMP Trap Translator Install the trap translator by following the supplied instructions and then configure the file /etc/snmp/snmptt.ini by altering the some of the parameters as follows:
mode = standalone
dns_enable = 1
strip_domain = 1
net_snmp_perl_enable = 1
translate_value_oids = 1
translate_enterprise_oid_format = 1
translate_trap_oid_format = 1
translate_varname_oid_format = 1
log_enable = 1
syslog_enable = 1
syslog_level = info
snmptt_conf_files =  /etc/snmp/snmptt.conf
Once the trap translator is successfully installed, you need to feed some mib trap definitions to it. For example, assume that you have the mib for the APC UPS "powernet361.mib":
./snmpttconvertmib --in=/usr/share/snmp/mibs/powernet361.mib --out=/etc/snmp/snmptt.conf
The next time you add mib definitions using this command, the translated mib will be appended to the snmptt.conf file. You can use a different output file but, if you do so, then do not forget to change the snmptt.ini. For example, you can do:
snmptt_conf_files = <<END
/etc/snmp/snmptt.conf.generic
/etc/snmp/snmptt.conf.compaq
/etc/snmp/snmptt.conf.cisco
/etc/snmp/snmptt.conf.hp
/etc/snmp/snmptt.conf.3com
END
When a trap occurs, the trap translator will output the information received from the trap to syslog and, at this point, SEC will take over. SEC SEC was installed before we discovered SNMP Trap Translator. We had used it as an event filter-correlator in a centralized syslog configuration to interface some events with Nagios. It may be possible to have the translator interface directly with Nagios, but we thought we would lose a lot in terms of flexibility if we did this. By using SEC, we can alter the flow of messages and do things such as:
  • For a non-stable machine or non-production equipment, you can delay alarms that occur at night until morning.
  • If you are suddenly bombarded by traps from a device, SEC can be used to regulate the flow.
Note that we chose to install SEC in the "/opt" directory. We start SEC with the following statement in the rc.local file:
/opt/sec/sec.pl -input=/var/log/messages -conf=/opt/sec/sec.conf \
  -detach -log=/var/log/sec.log
In our sec.conf file is the rule that handles SNMP traps:
# Snmptrap event translated by snmptraptt
type=Single
ptype=RegExp
pattern=snmptt.*(Normal|INFORMATIONAL|MINOR|WARNING|SEVERE|MAJOR|CRITICAL) \
  \"Status Events\" (\w+) \- (.*)
desc=snmptrap received from $2
action=shellcmd /opt/nagios/libexec/eventhandlers/snmptraphandling.py \
  $2 $1 "$3"
  
snmptraphandling.py The snmptraphandling.py Python script performs the following actions for us:
  • Formats the output of SEC
  • Obtains the current time in epoch format
  • Translates the severity code
  • Posts the data into the Nagios command file
See Listing 1 for complete code. Nagios This is the delicate part. Great care must be taken when monitoring traps to get it right. Given the high security standards of the customer for which we did this initial setup, one requirement was that all major events had to be audited. Thus, for each Nagios host capable of sending traps, we configured three services: one for critical, one for warnings, and one for "OK" events. This way, when a critical or warning event occurs, the systems administrators must submit a passive check result "OK" and enter a brief description of what action was taken to correct the situation in order to reset the event. This information is then kept in the Nagios event log. Because of the number of SNMP-capable devices, we have made use of service templates as shown below:
define service {
    use    passive-check-template
    name   generic-snmptrap
    service_description    snmp_trap_handling
    is_volatile 1
    check_period    none
    notification_interval    120
    notification_options    w,u,c,r
    notification_period    24x7
    check_command    passive_check_missing
    max_check_attempts    1
    check_freshness    0
    stalking_options    o,w,u,c
    register 0
}
Note the "stalking_options" parameter. Without it, if two or more traps are received quickly one after the other, the Nagios administrator will receive only alerts from the first trap and thus will not get duplicate messages. For each host to be monitored, we created the following services that refer to the previous service template as shown above:
define service {
    use    generic-snmptrap
    host_name     my_example_host
    contact_groups    my_example_group
    service_description    snmp_trap_handling_ok
}

define service {
    use    generic-snmptrap
    host_name    my_example_host
    contact_groups    my_example_group
    service_description    snmp_trap_handling_warning
}

define service {
    use    generic-snmptrap
    host_name     my_example_host
    contact_groups    my_example_group
    service_description    snmp_trap_handling_critical
}
To test the whole solution, we used an APC UPS with an on-board management card. After we configured the trap destination and community parameters, we began our tests. We simply disconnected the UPS from the mains, and Nagios immediately started to receive events and generate alarms. We then fed mib definitions for most of our major equipment vendors and, since then, have witnessed many different alarms for disks, cooling fans, and a host of other events.

Because we are using Nagios to monitor processes, it only makes sense to use it to monitor the presence of SEC and snmptrapd processes. This gives Nagios a brand new range of opportunities and makes our customer really happy.

This solution is only possible because smart individuals take the time to write high-quality software, such as Nagios, SNMP Trap Translator, Net-SNMP, and SEC.

References

Nagios -- http://nagios.org/

Net-SNMP -- http://www.net-snmp.org/

SEC -- http://kodu.neti.ee/~risto/sec/

SN MP Trap Translator -- http://www.snmptt.org/

Francois Meehan has more than 20 years of experience in computer operations, covering midrange computers to PCs, from AIX to OpenBSD, Linux, and Windows operating systems. He founded his own consulting company, CEDVAL Info Inc., specializing in network monitoring and open source solutions integration. CEDVAL Info customers are mostly in the pharmaceutical and manufacturing industries in the Montreal region and the United States. Francois, his wife, Lise, and their two children live in Notre-Dame-de-l'Ile-Perrot, an island on the St. Lawrence River west of Montreal. Francois can be contacted at: [email protected].


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.