June 23, 2018
Hot Topics:

Super Glue: Using Perl to Develop a Cheap Network Framework

  • November 16, 2005
  • By Brad Lhotsky
  • Send Email »
  • More Articles »

Network Security is hot these days. There are plenty of offerings, both commercial and free. Usually, a good network security model employs more than a single security product. However, not many commercial or free security utilities play nice with each other. Luckily, you can use perl to glue them together to get more meaningful data from your network.

To build something flexible and extendable, you're going to need to use a well-known integrated path to relay messages to the central server. Syslog-ng will handle that. You'll use a simple program in perl as a destination for some Snort messages relayed over syslog-ng. The perl program will use a PostgreSQL database to store the messages in a very custom fashion. You'll employ perl again in the frontend utilizing Mason to throw together a simple console to view messages. From the example in the article, a user should be able to incorporate other syslog enabled daemons into the security framework and begin correlating events systematically.

System Logging

Most packages on the market support "syslog" logging. Syslogging began as an extension to the sendmail project to handle the logging needs. Because then it has blossomed into the standard for logging messages on UNIX systems. There are even a few notable ports to Windows.

The concept is simple. It's a client/server architecture, usually confined to receiving and sending messages either over the loopback device or UNIX Sockets. The main reason for this is to not expose an additional service to the unfriendly and often aggressive internet. The client follows RFC3164 guidelines for sending a message, including its facility and criticality and the syslog server dispatches the message, usually to a file on the system.

What exactly does this have to do with security or programming?

Funny you should ask. To gather data from a network of hosts in some cohesive fashion, you'll have to talk to each other. You could either re-invent the wheel or leverage existing technologies. Syslog presents you with an interesting array of benefits:

  • Syslog is included with all modern UNIX Operating Systems, including Mac OS X.
  • Several free Windows-compatible syslog servers exist.
  • Syslog messages can be transmitted over TCP or UDP, allowing you to send messages from all over to a central server.
  • Syslog support is built into most network daemons and security packages.

Central Syslog Server

Yes, you're going to to set up a central syslog server to receive messages from your network. This server will only receive messages; it won't get any special privileges on your network. It's important to realize that attackers could potentially spoof messages to your server. Another risk is that some legacy systems only support UDP syslogging, which means your server may never get the client's message. Basically, this is not a 100% perfect solution, but the bang for the buck is quite refreshing.

On the upside, syslog functionality is usually built into most networked Operating Systems. This means that most key events such as authentication, hardware messages, login attempts, program crashes, and such go through this channel already. Additionally, most network daemons support logging through syslog in a rather painless way. Also, there's probly an API in your favorite language to write messages to syslog. Given the benefits, ease of use, and re-use of current components on the systems, syslog seems to be a good transport mechanism for your framework.

Choosing a Daemon

What are you looking for in a syslog daemon for a central server? Well, your primary concern needs to be compatibility and cost. Additionally, you're gonna want some modularization with the central syslog server. You'd love fine grain control of message dispatch as well. All things considered, you're looking for syslog-ng (http://www.balabit.com/products/syslog_ng/). It fits the bill nicely.

There's tons of documentation on syslog-ng. I invite you to indulge yourself in a deluge of well-crafted documentation: http://www.campin.net/syslog-ng/faq.html.

Configuring syslog-ng

There are a few key things to note about the system configuration on the syslog server. First, if you're going to use a TCP port to listen on, and you have a decent sized network, you're going to need to set the max-connections paramter:

source s_tcp {
   tcp(ip( port(514) max-connections(1000));

As I mentioned earlier, some legacy systems only support UDP, so you'll need to listen for that as well. You just add another source definition like so:

source s_udp {
   udp(ip( port(514));

The primary reason I decided to use syslog-ng is its modular ability to use external programs as "destinations" for logging. Any program you write will be executed once at startup and continue running so long as syslog-ng is running. Data is sent to STDIN on the external program and that handle is kept open the entire time syslog-ng is running.

This model makes perl an excellent candidate for use as a parser. Perl is great with handling text. Perl's major performance concern lies in the loading and startup of the perl interpretter itself. With syslog-ng simply starting up the interpreter once, with a bit of smart programming, a perl program works well as an event analyzer and data correlator for the infrastructure.

Perl Glue

To receive data from the syslog-ng server, all the perl program needs to do is read from STDIN. You can achieve this cleanly by using a while() loop.

while( local $_ = >< ) {
   chomp;                  # Strip the new line
   next if /^s*$/;        # Skip empty lines

   # Do something fancy with your data.

Extracting the data from the syslog messages using perl is easier than you might expect. A search of CPAN reveals quite a few syslog modules. I looked at quite a few syslog modules and they were very well put together. However, on my network, I'm dealing with several different client formats that behave slightly differently. I'm also more interested in the messages in the context of where they come from, what program sent them, and what that program was trying to tell me. So, although the syslog parsing modules on CPAN were good, I'd end up having to write my own regular expressions to extract context sensitive information from the log files anyway.

I knew I could handle writing the regular expressions to parse the logs, but I was more concerned with the performance of the log parser. Syslog-ng's "program" destination has no queuing mechanism. Messages are sent as fast as they come in, and if I'm unable to process them in time, I might end up dropping a few on the floor. To speed up my matching, I decided to use a technique known as "precooking" to save a step in the Perl regex matching.

Precooking regular expressions tells the perl interpreter to compile the regular expression just once, instead of the default behavior to compile every time it's used. Here's some regex to extract some key data from the syslog messages:

my %regex = (
   'meta' => {
      empty   => qr/^s*$/
   'mesg' => {
      date    => qr/(w+s+d+s+d+:d+:d+)/,
      host    => qr/s+d+:d+:d+s+(S+)/,
      facility=> qr/s+d+:d+:d+s+S+s+([^:]+):/,
      mesg    => qr/s+d+:d+:d+s+S+s+[^:]+:s+(.*)/

Page 1 of 2

Comment and Contribute


(Maximum characters: 1200). You have characters left.



Enterprise Development Update

Don't miss an article. Subscribe to our newsletter below.

By submitting your information, you agree that developer.com may send you developer offers via email, phone and text message, as well as email offers about other products and services that developer believes may be of interest to you. developer will process your information in accordance with the Quinstreet Privacy Policy.


We have made updates to our Privacy Policy to reflect the implementation of the General Data Protection Regulation.
Thanks for your registration, follow us on our social networks to keep up-to-date