February 2005

Shell Corner: Disk Space Checking with SNMP

Hosted by Ed Schaefer

diskchecker.sh
by Marc Skinner

This bash script — diskchecker.sh — is an SNMP poller that polls the SNMP OID of the disk partitions defined in the snmpd.conf file on Linux boxes. It polls the disks of hosts defined in the diskchecker.conf file. You can specify the drive and the minimum percentage you want free. Once the percentage is exceeded (e.g., a partition is set for 25% and suddenly only 20% of the partition is free), an email is generated.

I suggest monitoring all partitions. Include the disk partition information in the snmpd.conf file and the hosts you want to monitor in the diskchecker.conf file. If you wanted to monitor the disk partitions on host 10.10.10.5 and wanted to be alerted if the available disk space on partition /var had exceeded 75%, you would follow the examples below. Remember the % listed in the snmpd.conf file is the desired minimum disk space free before the threshold is exceeded.

SNMP stands for Simple Network Monitoring Protocol. It is a management tool all systems administrators should be using. It is a must for tools like MRTG, RRDTool, and a variety of other graphing and monitoring tools. SNMP is an extensible way that systems, programs, and environments can be monitored. SNMP can not only respond to tools with statistics like "Network card octets sent" or "Linux 5 minute load average", but it can also be used to change settings when using the Read-Write mode.

Making changes using SNMP is an advanced topic and will not be covered in this article. I will look only at the Read-Only mode. This script relies on the SNMP libraries and daemon to be installed on all the servers that you choose to monitor. Most of today's Linux distributions ship with SNMP packages (as do most networking equipment) and are ready to work out of the box.

On an RPM-based system such as Red Hat or Mandrake, run the command rpm -qa | grep snmp. This will display all the RPMs installed on your system that are SNMP-related. The most common SNMP libraries are ucd-snmp and net-snmp. Ucd-snmp was renamed to net-snmp when it changed major versions from 4.x to 5.x. If SNMP isn't installed, your distribution CD will have it. Install it from the CD or find it at http://rpmfind.net. If you need to build your own from source, follow these steps:

• Obtain the source for Net-SNMP at: http://net-snmp.sourceforge.net
• tar -xzf source.tar.gz
• ./configure --enable-ucd-snmp-compatibility --with-defaults --with-mib-modules="host" --with-default-snmp-version="2"
• make
• make install (as root)

Once SNMP is installed, you must configure it. A basic snmpd.conf file might look like this:

 
########################################################################### 
# 
# snmpd.conf 
# 
# - created by the snmpconf configuration program 
# 
########################################################################### 
# SECTION: System Information Setup 
# 
# This section defines some of the information reported in 
# the "system" mib group in the mibII tree. 

# syslocation: The [typically physical] location of the system. 
# Note that setting this value here means that when trying to 
# perform an snmp SET operation to the sysLocation.0 variable will make 
# the agent return the "notWritable" error code. IE, including 
# this token in the snmpd.conf file will disable write access to 
# the variable. 
# arguments: location_string 

syslocation TEST DATACENTER 
 
# syscontact: The contact information for the administrator 
# Note that setting this value here means that when trying to 
# perform an snmp SET operation to the sysContact.0 variable will make 
# the agent return the "notWritable" error code. IE, including 
# this token in the snmpd.conf file will disable write access to 
# the variable. 
# arguments: contact_string 

syscontact [email protected] 

# sysservices: The proper value for the sysServices object. 
# arguments: sysservices_number 

sysservices 76 

 

########################################################################### 
# SECTION: Access Control Setup 
# 
# This section defines who is allowed to talk to your running 
# snmp agent. 
 
# rocommunity: a SNMPv1/SNMPv2c read-only access community name 
# arguments: community [default|hostname|network/bits] [oid] 

rocommunity public 

# rwcommunity: a SNMPv1/SNMPv2c read-write access community name 
# arguments: community [default|hostname|network/bits] [oid] 

rwcommunity private 
 
disk / 25% 
disk /boot 15% 
disk /home 25% 
disk /tmp 25% 
disk /usr 15% 
disk /var 25% 
disk /raid1 25% 
load 90 80 75 

 
##################################################### end SNMPD.CONF file 

03 * * * * /path-to-script/diskchecker.sh >> /dev/null 2>&1 

First the script scans through the diskchecker.conf file (where all your hosts are listed, one per line), and then it does an SNMPWALK of the HOST MIB OID that corresponds with DISKS. An OID is an Object Identifier. SNMP identifies all the information it monitors with an OID. The two OIDs we are looking at are "1.3.6.1.4.1.2021.9.1.100 and 1.3.6.1.4.1.2021.9.1.101" from the HOST MIB.

An MIB (Management Information Base) is the database of objects that SNMP clients can access. There are two standard MIBs defined — MIB I defined in RFC 1156, and MIB II defined in RFC 1213. As I said before, SNMP is extensible; anyone can request and define an OID.

Many companies design custom SNMP objects that are private and specific to their needs. For example, a company might design an application that is monitored via SNMP. The company requests and obtains their own OID from the IANA (Internet Assigned Numbers Authority) and then defines the objects and structure to fit their specification. This is outside the scope of this column, but I thought I'd mention how SNMP can be used. SNMPWALK is a client query tool that is passed a host, SNMP community string, and, optionally, an OID. In the script, I first need to know whether any of the partitions defined in a hosts snmpd.conf file exceed the miniumum disk % threshold specified.

If a threshold is exceeded, the OID "1.3.6.1.4.1.2021.9.1.100" returns a value of 1 and the script then does an SNMPWALK of OID "1.3.6.1.4.1.2021.9.1.101". This actually returns the status of all monitored partitions with their current disk utilization and the disk threshold set in the snmpd.conf file.

To discard the data not required, the script does a grep for the keyword "less", since the partition flagged with less than the threshold is what it is concerned about. The rest of the script formats the email and sends it to the appropriate email address.

SNMP is a very powerful tool and should be part of every administrator's tool kit. This column has only scratched the surface of what SNMP can do. I hope you'll take the time to learn its ins and outs; I still am.

Marc Skinner lives in Minneapolis where he is a Linux Engineer for Agiliti, Inc. He can be reached at: [email protected]