July 2001

Survey of Log Analysis Tools for Snort

by Yen-Ming Chen

Snort is a lightweight network intrusion detection system capable of logging every possible trace of intrusion attempts into a text file, syslog, XML, libpcap format, or a database. This article introduces current tools that can help systems administrators analyze different log formats generated by Snort. This is not a complete analysis of all possible tools available for Snort. Because Snort is a robust freeware program, new tools are continuously developed and updated by users from around the globe. More tools are available for similar purposes but are not specifically designed or written for Snort.

The Snort tools covered fall into three different categories of output formats: text-based, libpcap-based, and database. The installation methods and functions will be briefly described for each tool in its own section. This article is written with Snort-1.8 beta 5 build 19. The version of tools used in this article will be noted in later sections.

What Is Intrusion Detection?

Intrusion detection is "The process of monitoring the events occurring in a computer system or network, analyzing them for signs of security problems"[1]. Most Intrusion Detection Systems (IDS) can be categorized into: network-based, host-based, or hybrid. A network-based IDS monitors the traffic on the network to find possible traces of an intrusion or an attack. A host-based IDS monitors the activities on the host (user commands, logon/logout process, data usage, etc.) to determine possible intrusion attempts on the host level. A hybrid IDS usually monitors both network and host environment by applying distributed sensors or agents technology. A central monitoring station will collect information from agents placed on other computers in the network to monitor network traffic and intrusion attempts.

Methods used by various IDS to discover possible intrusion attempts are categorized into signature-based and anomaly detection. The signature-based method finds the trace of known exploits in captured data. For example, some special forms of URLs, which contain malformed Unicode encoded characters, are the signatures of possible IIS Unicode attacks. Various pattern-matching techniques are applied in this area to increase the precision of this signature-based method. Anomaly detection compares current captured data with a baseline (which is defined as normal) set of information. Then, the system will determine whether there is any anomaly in the current captured data. For example, if a secretary usually works from 9 a.m. to 5 p.m., and one night his computer records him logged on to the company's mail server at midnight, then that can be an anomaly and worth further investigation. A lot of statistical methods and machine learning techniques have been applied in this area to help improve the precision of anomaly detection.

What is Snort?

Snort is a lightweight network intrusion detection system based on libpcap. It runs on a "sensor" machine (which can be a simple Pentium 100 PC running FreeBSD with at least one network card) and listens to the traffic on the wire. Snort can find traces of possible intrusion attempts by pattern matching with existing rule files specified at initial setup or by detecting statistically anomaly on network traffic (using the SPADE plug-in).

With a flexible modular architecture, Snort can also use plug-ins contributed by various authors and performs various functions including HTTP decoding, TCP stream re-assembly, portscan detection, disconnecting suspicious connections (FLEXRESP), and logging to different devices.

Snort is a free, simple, fast, and flexible network IDS. It has been ported to various Unix platforms and also the Win32 platform. It is one of the most active open source projects in the field of security. It is also one of the best alternatives to other similar expensive commercial products.

What is Log Analysis?

When you install an IDS (e.g., Snort), you are nowhere near complete. After you start your Snort, your hard drive space will be filled up with hundreds, if not thousands, of alerts indicating possible intrusion attempts or incidents. From this point on, you need a tool to analyze your log or alert files and prevent your pager or mailbox from being bombarded with those alerts (an alternate kind of DoS attack). Depending on your needs and security posture, you may just want a summary of what's in the logs, or your manager might ask you to periodically present a report of the top 15 attacks. It is also possible that you want to find out more information about a certain alert when you see it pop up in the log for further investigation. Whatever the purposes, log analysis helps you extract useful information from logs, which can enable you to act on alerts that actually pose a risk to your organization. Useful security measures take into account practical steps that enable systems administrators to more efficiently perform their jobs.

Snort Log Format

When running Snort, you can ask it to present the alerts in the way you specified in the configuration file. The formats as mentioned earlier are text-based, libpcap and database. I will briefly describe each format and how to configure it in your Snort configuration file. Then I will introduce different tools for log analysis on various data formats.

Text-Based Format

• Alert file--By using -A [fast|full|none] switch when you start Snort, you can make Snort write each alert into a file as in the following example:

    [**] INFO - ICQ Access [**] 
    [Classification:  content:"MKD / "] [Priority: 0]  
    05/10-10:02:31.953089 10.1.1.1.:54835 -> 10.2.2.5:80  
    TCP TTL:127 TOS:0x0 ID:13690 IpLen:20 DgmLen:482 DF   
    ***AP*** Seq: 0x112BDD12  Ack: 0x11B38D8A  Win: 0x4510  
    TcpLen: 20            

  The [Classification: content:"MKD / "] [Priority: 0] part is added in Snort 1.8 beta to support classification and priority of alert.
• Syslog file--By uncommenting the following line in the snort.conf file, you will enable Snort to log the data to your system log files: output alert_syslog: LOG_AUTH LOG_ALERT The output will look like the following:

  May 10 00:03:38 xxxxxx snort: INFO - ICQ Access [Classification:               
  content:"MKD / "   Priority: 0]: 10.1.1.1:54352 -> 10.2.2.5:80 

• CSV file--Brian Caswell contributed the CSV output plug-in that makes Snort able to log data in the CSV format. The configuration is very easy. To log to the CVS format, add the following line to the snort.conf file: output CSV: /your/filename timestamp,msg,proto,src,dst The output plug-in will generate the following result in the file /your/filename: 05/10-10:02:31.953089, INFO - ICQ Access, TCP,10.1.1.1,10.2.2.5 You can use the "default" without specifying the fields you want explicitly. You can use multiple CSV outputs in your configuration file to create the output you need for your own customized scripts.
• XML format--The XML output plug-in for Snort was developed by Jed Pickel and Roman Danyliw as part of the AIRCERT project (Automated Incident-Reporting CERT) in CERT CC. This plug-in enables Snort to output the logged data or an alert in XML format so the data can be stored in a local file and sent to a centralized database or CERT for further correlation and process. The data is in the format of SNML--Simple Network Markup Language (a.k.a. SNort Markup Language). More information is available at: http://www.cert.org/kb/snortxml The XML output plug-in supports HTTP, HTTPS, and IAP (Intrusion Alert Protocol). It can encode the data in hex, base64 or ASCII format.
  Examples for setting up XML output are:
  output xml: log, file=/var/log/snort/snortxml
  This line will generate the outputs from the rules with the log facility to the file called /var/log/snort/snortxml-MMDD@HHMM where "MMDD" is month and date, "HHMM" is hour and minute.

  output xml: alert,   protocol=https host=your.server.org file=yourfile              
  cert=mycert.crt key=mykey.pem ca=ca.crt server=srv_list.lst 

  This line generates the outputs from the rules you have defined with the alert facility to a file called "yourfile" on the remote server your.server.org through the HTTPS protocol. The "cert," "key," and "ca" are information for SSL. The "server" parameter specifies a list of servers to connect to.
  One example of output is:

   
	  <event version="1.0"> 
      <sensor encoding="hex" detail="full"> 
      <interface>fxp0</interface> 
      <ipaddr version="4">10.3.3.3</ipaddr> 
      <hostname>test.someserver.org</hostname> 
      </sensor> 
      <signature>RPC portmap listing</signature> 
      <timestamp>2001-05-09 19:43:05+00</timestamp> 
      <packet> 
      <iphdr saddr="192.89.3.5" daddr="10.3.3.3" proto="17" ver="4" hlen="5" \
               len="64" id="32085" ttl="239" csum="47239"> 
      <udphdr sport="34959" dport="111" len="44" csum="22602"> 
      <data>5A97E73C0000000000000002000186A00000000200000004000000000000000000000000</data> 
      </udphdr> 
      </iphdr> 
      </packet> 
      </event> 

Tcpdump format The Network Research Group at Lawrence Berkeley National Lab developed the Packet Capture Library. Many packet-capturing programs are based on this library. For example, tcpdump and Snort are based on this library. By using the following line in the snort.conf file, you can log the data in tcpdump binary format to the file "yourfile":

output log_tcpdump: snort_dump.log

You can later use Snort or tcpdump to read back the packets in the file.

Database Jed Pickel contributed the Snort database output plug-in in early March 2000. This output plug-in supports logging to different databases including MySQL, Postgresql, unixODBC, and Oracle.

To configure database logging, enter the following line in your snort.conf file:

output database: log, mysql, user=snortuser password=snortpass dbname=snortdb host=localhost

This line enables Snort to log the data into the MySQL database called "snortdb" on localhost using the user "snortuser" and password "snortpass" to authenticate. By changing "mysql" into "postgresql" or "unixodbc," you can log the data to different database formats.

Before you can really log data into the database, create the database and the user/password. Then grant proper rights (usually Select, Insert, Delete, Update, and maybe Create) for that user only to the database you created for Snort. Then run the proper SQL commands in the "contrib" directory of a normal Snort distribution to create the tables needed for logging purposes.

Tools

In this section, I will introduce existing tools for analyzing Snort logs. I will categorize them according to the type of log they process. A brief introduction to the functions and a simple how-to will be described in each paragraph.

For Text-Based Logs

• Snort_stat Perl script -- I wrote this Perl script in 1999 because I needed a program to generate some statistical result from the Snort log files periodically. This script will generate the following information:
  • The count of source and destination pairs with the same signature
  • The count of signatures with the same source and destination pair
  • The count of attempts with the same source and signature
  • The count of destinations with the same signature
  • Signature distributions
  • Portscan and SPADE logs

  The script will periodically send such information to the administrator as an email or store it in an HTML format file. Adding the following entry to your crontab file can perform an automated process of executing this script:

  59 23 * * * root cat /your/snort/logfile | /your/snort_stat.pl -r | sendmail someone@somewhere

  Then, the script will parse /your/snort/logfile at 11:59 p.m. every day. Do the reverse DNS lookup, and send the result in email to "someone@somewhere." Using -h switch can generate HTML output. You can get this script through the Web page:

  http://xanadu.incident.org/snort/

  And an example of HTML output is at:

  http://xanadu.incident.org/snort/example.html

  The latest version is 1.15. I am in the process of updating it for Snort 1.8.

• SnortSnarf: SnortSnarf is a Perl program that will read your log files and generate HTML output. This lets you look into your Snort log to find possible problems. The output contains links to WHOIS lookup and DNS lookup. You can also choose to find out what attacks are associated with certain IP addresses.

  SnortSnarf comes with other functions besides log analysis. It contains nmap2html that will turn the result of a nmap scan into a HTML page. The latest addition is SISR (SnortSnarf Incident Storage and Reporting mechanism), which will help you to create a report for an incident and store that report or send the report by email efficiently.

  To install SnortSnarf, first download it from the Web site (the latest version is 041501.1):

  http://www.silicondefense.com/software/snortsnarf/index.htm

  Make sure you have the Perl module you need. The Time::JulianDay module is included in the tar-ball. Prepare a directory for the HTML output, and then issue the following command:

  snortsnarf.pl -d /where/is/html/output /var/log/messages /var/log/snort/alert

  This command will parse /var/log/messages and /var/log/snort/alert files, and then generate the HTML output in /where/is/html/output directory. There are other possible options listed in the snortsnarf.pl file.

  The example output of SnortSnarf can be found at:

  http://www.silicondefense.com/software/snortsnarf/example/index.html

  SnortSnarf has a problem with printing correct destination and source IP addresses for Snort 1.8 at this point.

• Snort-sort: Andrew R. Baker contributed this script that produces a sorted list of Snort alerts from a Snort alert format log. You can find the script under the contrib directory within the Snort distribution. This script can do reverse DNS lookups for you and also put a link on each IP address to do the whois lookup. This script currently does not handle 1.8 alerts well.

  To run it, simply use:

  snort-sort.pl /your/alert/file > result.html

  It will generate the HTML output in the result.html file.

• Snort2html: Daniel Swan wrote this Perl script to turn Snort alerts in your syslog files into HTML format, with links to Max Vision's ArchNIDS database, from which you can get more information about the signature. You can also click on the source host and look up its information in the ARIN database. The script does not perform any further analysis on the log. The script does not handle the new format of Snort 1.8 well at this point.

  To install it, simply change $logfile and $outputfile in the Perl script, then run it in your crontab every day or periodically by executing snort2html.pl. No command switch is needed.

• Snortlog.pl: Snortlog.pl was written by Angelos Karageorgiou, and performs similar function as Snort-sort and Snort2html. It converts a Snort alert file into HTML pages. The usage is:

  Snortlog.pl /var/log/snort/alert machinename

  Then snortlog will process /var/log/snort/alert and generate HTML pages. It could not handle the alert format from Snort 1.8. It can be found at: http://packetstorm.securify.com/sniffers/snort/snortlog.pl.

• ARIS Extractor: ARIS stands for "Attack Registry and Intelligence Service" and is a free service from SecurityFocus that provides a large-scale correlation of IDS and firewall logs. ARIS Extractor is the program to extract the information from your Snort log and send it in XML format to their ARIS database for further analysis. You must register with SecurityFocus first to obtain a username and password in order to send your data to them. The data will be sanitized to keep your privacy, though certain information will still be exposed in order for the correlation to happen. It is not meant for personal use, but to create a worldwide database that can be used to trend attacks and perform analysis on a macro level. More details are available at: http://aris.securityfocus.com/.

For libpcap Format Logs There are tools to analyze a tcpdump binary file--tcpreplay and tcpshow, for example. These tools can read the file and decode the packet to reassemble the network traffic captured in that file. None of these tools is specifically written or designed for Snort.

For Database

• ACID: ACID stands for "Analysis Console for Intrusion Databases." It is an application written in PHP by Roman Danyliw as part of the AIRCERT project. The latest version is 0.9.6b9, and the homepage is at:

  http://www.cert.org/kb/acid/

  ACID is capable of performing searches, grouping, maintaining, and plotting the data in the database. The data can be either Snort log/alert or log from some other firewall product; ipchains is the one currently supported. It displays various statistical data including:

  • Traffic profile by protocol
  • Unique alerts
  • Most recent/frequent alerts
  • Most frequent addresses

  ACID also provides a link to the WHOIS lookup for IP addresses. You can group alerts and delete alerts in the database. It runs well with Snort 1.8 beta 5. To install it, download the tarball from their Web site first. You also need ADODB for supporting multiple databases. If you want to enable the graphic plotting functions, you will need GD 1.8 and PHPlot 4.4.6. The order of the procedures should be:

  • Make sure your Web server supports PHP
  • Install ADODB, GD, and PHPlot
  • Install ACID

  In the document for ACID, set up the following variables in acid_conf.php:

  • $DBlib_path: Full path to the ADODB install. For example: /usr/local/www/php/adodb.
  • $ChartLib_path: Full path to the PHPlot install, if you want to use it.
  • $DBtype: Type of database used ("mysql", "postgres").
  • $alert_dbname: MySQL database name where the alerts are stored. For example, snort.
  • $alert_host: Host where the database is stored.
  • $alert_port: Port where the database is stored.
  • $alert_user: Username into the database.
  • $alert_password: Password for the username.

After setting up these variables, you can run acid_main.php from your Web browser to create some tables in the database to help optimize ACID's performance.

Figure 1 shows the main page of ACID.

Conclusion

Apparently, there is no killer app in this area as yet. When you want to deal with your text-based logs, you can use SnortSnarf to gain a more thorough understanding of your logs and the ability to further search through them. Use the snort_stat perl script if you just need a periodic summary (in email or HTML) about what incidents are happening in your network environment. Other text-based tools simply convert the log files into HTML output without analyzing them first.

For databases, ACID is currently your best choice and the only choice if you are running Snort 1.7 or higher. With some other Perl scripts to maintain your Snort database, you can manage your intrusion detection infrastructure fairly easy and automatically. If you are interested in sending out your data so it will be correlated in someone else's server, register with SecurityFocus and use the ARIS Extractor.

The log analysis tools are just there to help you better track down possible intrusion attempts--or actual incidents. You still need to keep an eye on the logs as often as possible.

References

1. Rebecca G. Bace, "Intrusion Detection", MTP Technology Series, 2000. (Return to text.)

2. Snort homepage: http://www.snort.org/

3. AIRCERT Project: http://www.cert.org/kb/aircert/

4. Snortdb plug-in: http://www.incident.org/snortdb/

5. SnortSnarf: http://www.silicondefense.com/software/snortsnarf/

6. Snort_stat: http://xanadu.incident.org/snort/

7. ARIS: http://aris.securityfocus.com/

8. ACID: http://www.cert.org/kb/acid/