Developer.com content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Network Security is hot these days. There are plenty of offerings, both commercial and free. Usually, a good network security model employs more than a single security product. However, not many commercial or free security utilities play nice with each other. Luckily, you can use perl to glue them together to get more meaningful data from your network.
To build something flexible and extendable, you’re going to need to use a well-known integrated path to relay messages to the central server. Syslog-ng will handle that. You’ll use a simple program in perl as a destination for some Snort messages relayed over syslog-ng. The perl program will use a PostgreSQL database to store the messages in a very custom fashion. You’ll employ perl again in the frontend utilizing Mason to throw together a simple console to view messages. From the example in the article, a user should be able to incorporate other syslog enabled daemons into the security framework and begin correlating events systematically.
System Logging
Most packages on the market support “syslog” logging. Syslogging began as an extension to the sendmail project to handle the logging needs. Because then it has blossomed into the standard for logging messages on UNIX systems. There are even a few notable ports to Windows.
The concept is simple. It’s a client/server architecture, usually confined to receiving and sending messages either over the loopback device or UNIX Sockets. The main reason for this is to not expose an additional service to the unfriendly and often aggressive internet. The client follows RFC3164 guidelines for sending a message, including its facility and criticality and the syslog server dispatches the message, usually to a file on the system.
What does Syslog have to do with security or programming?
Funny you should ask. To gather data from a network of hosts in some cohesive fashion, you’ll have to talk to each other. You could either re-invent the wheel or leverage existing technologies. Syslog presents you with an interesting array of benefits:
- Syslog is included with all modern UNIX Operating Systems, including Mac OS X.
- Several free Windows-compatible syslog servers exist.
- Syslog messages can be transmitted over TCP or UDP, allowing you to send messages from all over to a central server.
- Syslog support is built into most network daemons and security packages.
Central Syslog Server
Yes, you’re going to to set up a central syslog server to receive messages from your network. This server will only receive messages; it won’t get any special privileges on your network. It’s important to realize that attackers could potentially spoof messages to your server. Another risk is that some legacy systems only support UDP syslogging, which means your server may never get the client’s message. Basically, this is not a 100% perfect solution, but the bang for the buck is quite refreshing.
On the upside, syslog functionality is usually built into most networked Operating Systems. This means that most key events such as authentication, hardware messages, login attempts, program crashes, and such go through this channel already. Additionally, most network daemons support logging through syslog in a rather painless way. Also, there’s probly an API in your favorite language to write messages to syslog. Given the benefits, ease of use, and re-use of current components on the systems, syslog seems to be a good transport mechanism for your framework.
Choosing a Daemon
What are you looking for in a syslog daemon for a central server? Well, your primary concern needs to be compatibility and cost. Additionally, you’re gonna want some modularization with the central syslog server. You’d love fine grain control of message dispatch as well. All things considered, you’re looking for syslog-ng (http://www.balabit.com/products/syslog_ng/). It fits the bill nicely.
There’s tons of documentation on syslog-ng. I invite you to indulge yourself in a deluge of well-crafted documentation: http://www.campin.net/syslog-ng/faq.html.
Configuring syslog-ng
There are a few key things to note about the system configuration on the syslog server. First, if you’re going to use a TCP port to listen on, and you have a decent sized network, you’re going to need to set the max-connections paramter:
source s_tcp {
tcp(ip(0.0.0.0) port(514) max-connections(1000));
};
As I mentioned earlier, some legacy systems only support UDP, so you’ll need to listen for that as well. You just add another source definition like so:
source s_udp {
udp(ip(0.0.0.0) port(514));
};
The primary reason I decided to use syslog-ng is its modular ability to use external programs as “destinations” for logging. Any program you write will be executed once at startup and continue running so long as syslog-ng is running. Data is sent to STDIN on the external program and that handle is kept open the entire time syslog-ng is running.
This model makes perl an excellent candidate for use as a parser. Perl is great with handling text. Perl’s major performance concern lies in the loading and startup of the perl interpretter itself. With syslog-ng simply starting up the interpreter once, with a bit of smart programming, a perl program works well as an event analyzer and data correlator for the infrastructure.
Perl Glue
To receive data from the syslog-ng server, all the perl program needs to do is read from STDIN. You can achieve this cleanly by using a while() loop.
while( local $_ = >< ) {
chomp; # Strip the new line
next if /^s*$/; # Skip empty lines
#
# Do something fancy with your data.
}
Extracting the data from the syslog messages using perl is easier than you might expect. A search of CPAN reveals quite a few syslog modules. I looked at quite a few syslog modules and they were very well put together. However, on my network, I’m dealing with several different client formats that behave slightly differently. I’m also more interested in the messages in the context of where they come from, what program sent them, and what that program was trying to tell me. So, although the syslog parsing modules on CPAN were good, I’d end up having to write my own regular expressions to extract context sensitive information from the log files anyway.
I knew I could handle writing the regular expressions to parse the logs, but I was more concerned with the performance of the log parser. Syslog-ng’s “program” destination has no queuing mechanism. Messages are sent as fast as they come in, and if I’m unable to process them in time, I might end up dropping a few on the floor. To speed up my matching, I decided to use a technique known as “precooking” to save a step in the Perl regex matching.
Precooking regular expressions tells the perl interpreter to compile the regular expression just once, instead of the default behavior to compile every time it’s used. Here’s some regex to extract some key data from the syslog messages:
my %regex = (
'meta' => {
empty => qr/^s*$/
},
'mesg' => {
date => qr/(w+s+d+s+d+:d+:d+)/,
host => qr/s+d+:d+:d+s+(S+)/,
facility=> qr/s+d+:d+:d+s+S+s+([^:]+):/,
mesg => qr/s+d+:d+:d+s+S+s+[^:]+:s+(.*)/
}
);
Syslog-ng Parser example
A really simple syslog-ng compatible parser might look something like this:
use strict;
#
# Precook regular expressions to save recompile time.
my %regex = (
'meta' => {
empty => qr/^s*$/
},
'mesg' => {
date => qr/(w+s+d+s+d+:d+:d+)/,
host => qr/s+d+:d+:d+s+(S+)/,
facility=> qr/s+d+:d+:d+s+S+s+([^:]+):/,
mesg => qr/s+d+:d+:d+s+S+s+[^:]+:s+(.*)/
}
);
while( local $_ = <> ) {
chomp; # Strip the new line
next if /$regex{meta}{blank}/; # Skip empty lines
#
# Build a hash from our regex:
my %mesg = ();
foreach my $field (keys %{ $regex{mesg} }) {
if(/$regex{mesg}{$field}/) {
$mesg{$field} = $1;
}
}
#
# Now %mesg might look something like this:
#
# %mesg = (
# date => 'Oct 18 22:02:50',
# host => 'idssensor',
# facility => 'snort',
# mesg => '[1:2466:6] NETBIOS SMB-DS IPC$ unicode
# share access
# [Classification: Generic Protocol Command Decode]
# [Priority: 3]:
# {TCP} 10.0.2.150:1372 -> 10.0.2.154:445'
}
Calling the Parser with Syslog-ng
Earlier, I mentioned the syslog-ng “program” destination. It’s actually far too easy to configure. First, you need to make the perl script executable; for me it was ‘chmod +x parser.pl’. Then, you just add it to the syslog-ng config file:
destination d_perl { program("/path/to/the/parser.pl");};
All that line does is tell syslog-ng to start the program during startup. Incidentally, no messages are passed through to it. To send messages to the parser, you need to create a “log” entry in the config file. For your main example, you’re going to focus on messages from snort, so you’ll also use the following “filter” to just send your parser your snort messages.
filter f_snort { program("snort"); };
Now, you put your sources through a filter and out to a destination.
log { source(s_tcp); source(s_udp); filter(f_snort);
destination(d_perl); };
Now, if you restart syslog-ng, incoming messages from TCP and UDP client from snort will be passed to your parser.
Don’t Touch that Dial
Okay, you’re bored. Well, it’s about to get more interesting. Now that you have the basic concepts, the next article will dig into extracting data from the snort messages, and storing them somewhere in a way that makes it possible to build on.
Build on? Yes, currently I’m integrating my network’s DHCPD logs and Windows Event Success Audit for the login system into my database. Once things are in databases, I can correlate, analyze, and start incorporating all kinds of external data. NMS, Ticketing Systems, E-Mail, and commercial or open source firewalls can all be interfaced with through perl. Individually, these systems are useful, but when working together, they can eliminate tedious tasks and leave a network administrator time to focus on solving problems and contributing more crazy ideas to the programmer’s patchwork Network & Security Management Console.
Please contact me with input because I will try to answer questions and incorporate your ideas into the next few articles in this series. At the end the series, I’ll include an archive of source code for you to play with.
About the Author
Brad Lhotsky is a Software Developer whose focus is primarily web based application in Perl and PHP. He has over 5 years experience developing systems for end users and system and network administrators. Brad has been active on Perl beginner’s mailing lists and forums for years, attempting to give something back to the community.
