An Introduction to LogsurferOriginally published in SysAdmin magazine, March 2004
Downloading and Installing
Alerting if there are no messages
Detecting port scans
wget http://www.crypt.gen.nz/logsurfer/logsurfer+-1.6.tar.gz tar xvfz logsurfer+-1.6.tar.gzConfigure Logsurfer+ with default options, compile and install it in /usr/local/bin:
./configure make make installand that's pretty much all there is to install. Logsurfer is a single small C program which is very easy to install and manage. You will also want to setup the start-up and shutdown scripts depending upon your system. For Solaris there is one available :
wget http://www.crypt.gen.nz/logsurfer/logsurfer_init_solaris mv logsurfer_init_solaris /etc/init.d/logsurfer chmod 755 /etc/init.d/logsurfer ln -s /etc/rc3.d/S98logsurfer /etc/init.d/logsurferEdit /etc/init.d/logsurfer and set local options for log file locations, etc. Then symlink it into the /etc/rc*.d directories to be invoked when the system starts up and shuts down. There is also an init.d script for RedHat Linux available. The init script defines some options to Logsurfer including which log file to monitor the location of the configuration file which contains the monitoring rules.
match_regex not_match_regex stop_regex not_stop_regex timeout [continue] action...where match_regex is a regular expression defining which lines match the rule, not_match_regex is a regular expression defining lines which are NOT to match the rule. If a line matches stop_regex then the rule will be removed from the run-time list of active rules, and not_stop_regex defines lines which will not apply to stop_regex. The timeout entry defines how long the rule will be active for, or specify 0 for no timeout. The continue keyword, if present, tells Logsurfer to continue processing lines which match this rule ( by default it will stop processing a line as soon as there is a match ). The "-" character can be used to mark unused fields. The action field specifies what to do when a successful match is found. The action can be one of :
ignore - ignore the line
exec - run a progam
pipe - like exec, but the log line is sent to stdin
open - open a new context rule
delete - delete an open context
report - run program, piping in context data
rule - create new rule
Its easy to see what rules can do with a few simple examples.
'last message repeated' - - - 0 ignoreThis rule simply ignores all lines containing "last message repeated". Note that if there is a lot of log messages going into a file then it is a good idea to explicitly ignore unwanted lines at the top of the Logsurfer configuration file.
' ([^ ]+) unix: ' - - - 0 pipe /usr/local/bin/start-mail firstname.lastname@example.org "Unix message from $2"This rule simply sends a copy of the log line in an Email to the operator ( email@example.com ). Unfortunately, the operator will get a separate message for each log line containing "unix:" which could be a large number if a system is booting. A better rule for doing this is defined in the next section on contexts. The start-mail script is in the contrib directory of the Logsurfer package, it is simply a sendmail wrapper script to add a subject and place the Email alert into the sendmail queue.
match_regex not_match_regex line_limit timeout_abs timeout_rel [min_lines] actionwhere match_regex - lines to match not_match_regex - lines to ignore line_limit - maximum number of lines to match timeout_abs - absolute timeout, from first matching line timeout_rel - relative timeout, from last matching line min_lines - optional minimum number of lines to match action: ignore - ignore the line exec - run a progam pipe - like exec, but the log line is sent to stdin report - run program, piping in all log lines collected A better "unix:" system message rule could be defined as follows :
' ([^ ]+) unix: ' - - - 0 open " $2 unix: " - 500 1200 600 report /usr/local/bin/start-mail firstname.lastname@example.org "Unix message from $2"This rule will look for lines containing "unix:" with a host name in the preceding field. When a line matches, it will create a new context which will collect further matching lines from the same host with a maximum number of lines of 500, an absolute timeout of 1200 seconds ( 20 minutes ) and a relative timeout of 600 seconds ( 10 minutes ). This means that we will collect up all of the matching log lines and Email them to the operator when we don't see a matching line within 10 minutes of the last matching line, or within 20 minutes of the first matching line. We put a line limit of 500 lines in just to prevent run-away messages exhausting our memory space.
0,15,30,45 * * * * /bin/logger -t syslog_ping -p local7.info `/bin/uptime`in this example, I also send the output from uptime which logs the number of users logged in and a brief summary of the system load. A Logsurfer rule to detect if a host stops logging looks like the following :
' ([^ ]+) syslog_ping:' - - - 0 open " $2 syslog_ping:" - - - 2100 exec /usr/local/bin/start-mail email@example.com "Alert: syslog pings from $2 have stopped"The first line defines a rule which will match the syslog_ping log message from hosts. The expression '([^ ]+)' simply matches the host name from the log message and places it into $2. The action from the rule creates a new context which monitors all future syslog messages from the host and will send an Email alert if there are no messages seen in a span of 35 minutes - which means that we are allowed to drop one syslog message accidentally but if we don't get two then the monitored system may be having problems. Note that you need Logsurfer+ v1.6 to perform this properly - the standard Logsurfer v1.5b would store all matching lines in memory, possibly causing the system to run out of memory over time. The nice thing here is that we don't have to define a rule for every host that is logging. The first rule will automatically start up a monitoring context as soon as it sees a syslog_ping message from a host which it is not already monitoring. In just a few lines of configuration, we have created a relatively complex function.
' ([^ ]*) %SEC-6-IPACCESSLOGP: .* ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\([0-9]+\) ->' - - - 0 open " $2 %SEC-6-IPACCESSLOGP: .* $3\\([0-9]+\\) ->" - 10000 600 300 100 report "/usr/local/bin/start-mail firstname.lastname@example.org \"Port scan from $2\"" " $2 %SEC-6-IPACCESSLOGP: .* $3\\([0-9]+\\) ->"This is a quite complex rule. Basically, for every denied packet it opens a context which begins tracking further denials from the same source. If there are more than 100 denied packets ( up to a limit of 10,000 ) with an absolute time-span of 600 seconds and a relative timeout to 300 seconds, then an alert message is Emailed which contains the logged messages. Of course, this can be tuned to meet your needs and can easily monitor other types of firewalls such as iptables, PIX and CheckPoint Firewall-1.
- Be careful how you process syslog messages. It is sometimes possible for an attacker to influence the contents of syslog messages, sometimes inserting control and escape character sequences.
- Its a good idea to install and run Logsurfer under a non-root userid, such as "logsurfer".
- Remember to kill -HUP the logsurfer process whenever you roll-over your syslog files, in the same way that you need to kill -HUP your syslog daemon.
- Of course, your alerts don't need to be Emailed. You can just as easily use a pager interface to send alerts.
- Logsurfer can be used to monitor any log file - not just syslog files. It is also often used to monitor web server log files and logs belonging to applications.
http://www.crypt.gen.nz/logsurfer emf's Logsurfer configuration page, containing many useful Logsurfer configurations:
http://www.obfuscation.org/emf/logsurfer.html CERT-dfn official Logsurfer site: