Linux Log Monitoring and Watcher - About Watching UNIX Log Files

Linux Log Monitoring and Watching - How can you monitor your Linux log files without having to spend hours writing a script for the task or searching the internet endlessly for a log monitoring solution?

Linux log file monitoring and watching is a duty that is in a different dimension all by itself. It is unique. The reason why I say this is that, log files in Linux often vary in format which makes it difficult for many users/admins to accomplish the task of monitoring their various logs. This is why it is impossible to find one good monitor that can be set up to watch custom log files, which by the way, are very common in UNIX environments.

But what exactly do I mean by "format"? By format, I'm referring solely to the format of the dates on the lines in the logs which is usually at the beginning of each line.

For instance, a typical /var/log/messages file on a Linux server, will look something like this:

[nagios.kedy0:501]
[nagios.kedy0:501] tail -6 /var/log/messages
Sep 18 08:23:51 nagios snmpd[4539]: Received SNMP packet(s) from UDP: [10.10.*.*]:47725
Sep 18 08:23:54 nagios snmpd[4539]: Connection from UDP: [10.10.*.*]:47725
Sep 18 08:24:11 nagios sshd[13078]: Authorized to root, krb5 principal dadmin/root@NETNET
Sep 18 08:24:11 nagios sshd[13078]: permit_root_login: PERMIT_GSSAPI_ONLY method: gssapi-with-mic
Sep 18 08:24:11 nagios sshd[13078]: GSSAPIII authenticated jbowman login accepted
Sep 18 08:24:11 nagios sshd[13078]: Accepted gssapi-with-mic for root from 10.10.*.* port 5345 ssh2
[nagios.kedy0:502]
[nagios.kedy0:502]

Notice the first three columns of each line of the above output. They represent the date and time.

Another log file, usually an application log file (not a system log like the one above), can look like this:

[nagios.kedy0:516] tail -3 /prod/app.log
2011/01/20 14:26:35 UTC [SMTPProper,48088609,69.*.*.*] Receiving message for delivery: from=17777551333@vim.blah.com to=['17777551333@vim.blah.com']
2011/01/20 14:26:35 UTC [-] Attempting 'attach' (promo is False) delivery for 17777551333
2011/01/20 14:26:35 UTC [-] Starting factory
[nagios.kedy0:517]

Again, focus on the columns; in this case, it is the first 2 columns that matter.

The day, month and year are separated with forward slashes and they're numerical. Notice the order they're in. The year comes first, then the month, then the day. Other log files may have the order reversed; with the Day coming first, followed by the month, then followed by the year. Other logs may have the fields separated by hyphens instead of slashes. The scenarios here are endless.

Do you now see how trying to monitor a UNIX linux log can be a hellish task for a UNIX user to take on without the proper efficient tool?