An Introduction to Logsurfer

Originally published in SysAdmin magazine, March 2004

      Introduction
      Downloading and Installing
      Rules
      Contexts
      Alerting if there are no messages
      Detecting port scans
      Conclusion
      Resources

Introduction

There are many useful open source tools available. So many in fact, that it can be hard to choose which ones to use to solve a particular problem. Logsurfer is one of the most useful tools that I know of to monitor system log files, in fact, it is quite surprising just how much can be done with this small C program.

Systems administrators often dismiss problems which are difficult to do. For example, if you had 100 systems logging to a central server at the rate of around 100 megabytes of messages a day in total, how would you configure a log monitoring system to alert an operator if any one of those systems stopped sending syslog messages? An elegant solution would be difficult, and probably involve a custom-written perl program. But for Logsurfer, such a function would be almost trivial.

Another example is port scans. Lets say you log your firewall packet drops into syslog and you want to alert someone if there is a high rate of dropped packets from a single source IP address. Again, you would probably be looking for a specialised program, and again, Logsurfer can do this fairly easily.

In this article I will be examining a version of Logsurfer called Logsurfer+ which has been modified slightly with a few extra features such as the ability to alert when certain messages stop coming in, and being able to specify the minimum number of messages required to trigger an alert. Logsurfer+ is an extended version of the standard Logsurfer, and can be downloaded from http://www.crypt.gen.nz/logsurfer

Downloading and Installing

Download the latest version of Logsurfer+ from http://www.crypt.gen.nz/logsurfer ( currently version 1.6 ) :

  wget http://www.crypt.gen.nz/logsurfer/logsurfer+-1.6.tar.gz
  tar xvfz logsurfer+-1.6.tar.gz

Configure Logsurfer+ with default options, compile and install it in /usr/local/bin:

  ./configure
  make
  make install

and that's pretty much all there is to install. Logsurfer is a single small C program which is very easy to install and manage. You will also want to setup the start-up and shutdown scripts depending upon your system. For Solaris there is one available :

  wget http://www.crypt.gen.nz/logsurfer/logsurfer_init_solaris
  mv logsurfer_init_solaris /etc/init.d/logsurfer
  chmod 755 /etc/init.d/logsurfer
  ln -s /etc/rc3.d/S98logsurfer /etc/init.d/logsurfer

Edit /etc/init.d/logsurfer and set local options for log file locations, etc. Then symlink it into the /etc/rc*.d directories to be invoked when the system starts up and shuts down.

There is also an init.d script for RedHat Linux available.

The init script defines some options to Logsurfer including which log file to monitor the location of the configuration file which contains the monitoring rules.

Rules

Logsurfer's rules are simply instructions on what to do when it sees a particular line in the incoming stream of log messages. A rule line has the following fields:

  match_regex not_match_regex stop_regex not_stop_regex timeout [continue] action...

where match_regex is a regular expression defining which lines match the rule, not_match_regex is a regular expression defining lines which are NOT to match the rule. If a line matches stop_regex then the rule will be removed from the run-time list of active rules, and not_stop_regex defines lines which will not apply to stop_regex. The timeout entry defines how long the rule will be active for, or specify 0 for no timeout. The continue keyword, if present, tells Logsurfer to continue processing lines which match this rule ( by default it will stop processing a line as soon as there is a match ).

The "-" character can be used to mark unused fields.

The action field specifies what to do when a successful match is found. The action can be one of :

     ignore - ignore the line
     exec - run a progam
     pipe - like exec, but the log line is sent to stdin
     open - open a new context rule
     delete - delete an open context
     report - run program, piping in context data
     rule - create new rule

Its easy to see what rules can do with a few simple examples.

  'last message repeated' - - - 0
    ignore

This rule simply ignores all lines containing "last message repeated". Note that if there is a lot of log messages going into a file then it is a good idea to explicitly ignore unwanted lines at the top of the Logsurfer configuration file.

  ' ([^ ]+) unix: ' - - - 0
    pipe /usr/local/bin/start-mail operator@example.com "Unix message from $2"

This rule simply sends a copy of the log line in an Email to the operator ( operator@example.com ). Unfortunately, the operator will get a separate message for each log line containing "unix:" which could be a large number if a system is booting. A better rule for doing this is defined in the next section on contexts.

The start-mail script is in the contrib directory of the Logsurfer package, it is simply a sendmail wrapper script to add a subject and place the Email alert into the sendmail queue.

Contexts

Contexts are dynamically created rules which have the ability to group log lines together according to pattern matches. Contexts have the ability to group lines together even when there are unrelated log entries interspersed between them. For example, you would use a context to group together all messages relating to the ntp daemon from a certain host. All contexts are created by rules using the "open" rule action. The syntax for creating a context from a rule is to use the "open" action followed by the context definition:

  match_regex not_match_regex line_limit timeout_abs timeout_rel [min_lines] action

where match_regex - lines to match not_match_regex - lines to ignore line_limit - maximum number of lines to match timeout_abs - absolute timeout, from first matching line timeout_rel - relative timeout, from last matching line min_lines - optional minimum number of lines to match action: ignore - ignore the line exec - run a progam pipe - like exec, but the log line is sent to stdin report - run program, piping in all log lines collected A better "unix:" system message rule could be defined as follows :

  ' ([^ ]+) unix: ' - - - 0
    open " $2 unix: " - 500 1200 600
      report /usr/local/bin/start-mail operator@example.com "Unix message from $2"

This rule will look for lines containing "unix:" with a host name in the preceding field. When a line matches, it will create a new context which will collect further matching lines from the same host with a maximum number of lines of 500, an absolute timeout of 1200 seconds ( 20 minutes ) and a relative timeout of 600 seconds ( 10 minutes ). This means that we will collect up all of the matching log lines and Email them to the operator when we don't see a matching line within 10 minutes of the last matching line, or within 20 minutes of the first matching line. We put a line limit of 500 lines in just to prevent run-away messages exhausting our memory space.

Alerting if there are no messages

Lets say you have a lot of systems all logging into a single file on a central log server. One useful function you want to setup is to configure a cron job on each remote system to send a "ping" syslog message every 15 minutes or so just to let you know that the system is up and running. The cron table entry could look like this:

  0,15,30,45 * * * * /bin/logger -t syslog_ping -p local7.info `/bin/uptime`

in this example, I also send the output from uptime which logs the number of users logged in and a brief summary of the system load. A Logsurfer rule to detect if a host stops logging looks like the following :

  ' ([^ ]+) syslog_ping:' - - - 0
    open " $2 syslog_ping:" - - - 2100
      exec /usr/local/bin/start-mail operator@example.com 
       "Alert: syslog pings from $2 have stopped"

The first line defines a rule which will match the syslog_ping log message from hosts. The expression '([^ ]+)' simply matches the host name from the log message and places it into $2. The action from the rule creates a new context which monitors all future syslog messages from the host and will send an Email alert if there are no messages seen in a span of 35 minutes - which means that we are allowed to drop one syslog message accidentally but if we don't get two then the monitored system may be having problems. Note that you need Logsurfer+ v1.6 to perform this properly - the standard Logsurfer v1.5b would store all matching lines in memory, possibly causing the system to run out of memory over time.

The nice thing here is that we don't have to define a rule for every host that is logging. The first rule will automatically start up a monitoring context as soon as it sees a syslog_ping message from a host which it is not already monitoring. In just a few lines of configuration, we have created a relatively complex function.

Detecting port scans

Another feature of Logsurfer+ is that you can configure a minimum number of lines to match in a context before the action is triggered. If the minimum is not reached, and the context times-out then no action is performed. This can be used to alert when a serious security event occurs, such as when a firewall system receives a large number of attempted connections. You don't really want to see alerts when there are just 10 denied attempts - but you would want to get an alert if there were 100 or more in rapid succession.

The following rule alerts the administrator if a Cisco router logs a high number of denied packets from a single IP address :

  ' ([^ ]*) %SEC-6-IPACCESSLOGP: .* ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\([0-9]+\) ->' - - - 0
    open " $2 %SEC-6-IPACCESSLOGP: .* $3\\([0-9]+\\) ->" - 10000 600 300 100
      report "/usr/local/bin/start-mail operator@example.com
        \"Port scan from $2\"" " $2 %SEC-6-IPACCESSLOGP: .* $3\\([0-9]+\\) ->"

This is a quite complex rule. Basically, for every denied packet it opens a context which begins tracking further denials from the same source. If there are more than 100 denied packets ( up to a limit of 10,000 ) with an absolute time-span of 600 seconds and a relative timeout to 300 seconds, then an alert message is Emailed which contains the logged messages. Of course, this can be tuned to meet your needs and can easily monitor other types of firewalls such as iptables, PIX and CheckPoint Firewall-1.

Conclusion

Logsurfer is a simple, small, and efficient program which can monitor log files in real-time and intelligently send alerts when anomalies occur. Unlike other monitoring systems, such as swatch, Logsurfer can be tweaked and tuned to only send single alerts containing all of the relevant information and not send a deluge of Emails to the operator. Logsurfer can be used to detect faults and security events before they develop into serious problems. Some additional tips:

Be careful how you process syslog messages. It is sometimes possible for an attacker to influence the contents of syslog messages, sometimes inserting control and escape character sequences.
Its a good idea to install and run Logsurfer under a non-root userid, such as "logsurfer".
Remember to kill -HUP the logsurfer process whenever you roll-over your syslog files, in the same way that you need to kill -HUP your syslog daemon.
Of course, your alerts don't need to be Emailed. You can just as easily use a pager interface to send alerts.
Logsurfer can be used to monitor any log file - not just syslog files. It is also often used to monitor web server log files and logs belonging to applications.

Resources

Kerry Thompson's Logsurfer+ page, for Logsurfer+ download and information:
http://www.crypt.gen.nz/logsurfer

emf's Logsurfer configuration page, containing many useful Logsurfer configurations:
http://www.obfuscation.org/emf/logsurfer.html

CERT-dfn official Logsurfer site:
http://www.cert.dfn.de/eng/logsurf/

Crypt.Gen.NZ