Facebook
TwitterQmail Log Analysis and Reports
There have been many posts asking about email statistics, so here's a little report on the tools I've tried. All of the following are relatively simple schemes (no programming skills required) which readily produce email stats from Plesk's existing Qmail logs.
Note: The paths shown are for Plesk 8.3 with CentOS 5. For other systems there may be some variation in paths.
First a couple of simple, reliable approaches... these are my favorites:
1) This first scheme uses awk for filtering and qmailanalog (http://cr.yp.to/qmailanalog.html) for the log analysis.
Note: When compiling qmailanalog on CentOS 5, it was necessary to first apply the 0.70-errno.patch, see for example http://www.qmailrocks.org/qmailanalog.htm
Note: If using qmailanalog results in an error message such as 'sort: open failed: +2: No such file or directory', then you'll need to edit the three files zdeferrals, zfailures and zsuccesses in /usr/local/qmailanalog/bin, replacing 'sort +2' with 'sort -k 3', in zrhosts and zrecipients replace 'sort +4' with 'sort -k 5', and in zsuids replace 'sort -n +7' with 'sort -n -k 8'. See 'info sort' for more information about the sort problem (clue: the old sort used zero based numbering). To make these changes to the qmailanalog source before compiling, use the patch file at http://www.korsten.org/misc/qmailanalog-sort.patch .
To see a text report from the current log:
cat /usr/local/psa/var/log/maillog | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup | /usr/local/qmailanalog/bin/zoverall | more
To see an overall report from a zipped log:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup 5>/dev/null | /usr/local/qmailanalog/bin/zoverall | more
To see percentage distribution of delays:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup | /usr/local/qmailanalog/bin/zddist | less
To see delivery delay to each recipient:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup | /usr/local/qmailanalog/bin/zrxdelay | less
There do not seem to be any features included in qmailanalog to automatically capture the fragments of messages which begin in one log and are resolved in the following log. Nonetheless, the various different qmailanalog reports are quite useful, and the number of missed emails at log rotation should typically be small compared to the daily total.
2) If you just want to run a quick sanity check, to get an idea if anything in the email system has gone berserk, here's a quick way using awk. This scheme might be used, for example, in a simple script which decides to send an email to an admin if the number of emails suddenly exceeds some reasonably expected threshold. (remember that unless the logs happen to be changed at precisely midnight, the log file will only include a portion of the first day and a portion of the last day of that log)
To see the total number of successes and failures included in the log:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:.*success:/ {s++} /qmail:.*failure:/ {f++} END {print s, "successes\n", f, "failures"} ' | more
Or, to see the number of qmail successes for each of the individual days which are included in the log:
outgoing success:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:.*success:.*accepted_message/ {print "qmail outgoing success on", $1, $2} ' | uniq -c | more
incoming and outgoing, success and failure:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:.*success:.*accepted_message/ {print "qmail outgoing success on", $1, $2} /qmail:.*success: did_/ {print "qmail incoming success on", $1, $2} /qmail:.*failure: Unknown_R/ {print "qmail incoming failure on", $1, $2} /qmail:.*failure:.*Remote_host_/ {print "qmail outgoing failure on", $1, $2}' | sort | uniq -c | more
Next, here are a couple of approaches that I've ended up abandoning.
3) I've also experimented with isoqlog http://www.enderunix.org/isoqlog/. The output is html, and provides only the most basic information about number of messages sent and received. Isoqlog is easy to install, especially using the instructions at http://www.qmailrocks.org/extra/isoqlog.htm
The html reports produced by isoqlog are very limited, showing only sent, received and total for each day. And worse yet, while isoqlog apparently works fine for some Plesk users, even with a cron call every hour on my system isoqlog is only counting about 10% of my server's actual mail. So... perhaps it may work for you, or perhaps not. Personally, I've given up on isoqlog.
4) Another scheme uses the lire (http://logreport.org/dl.html) routines for both the filtering and the log analysis. The log analysis package called lire-2.0.3 happily accepts the Plesk qmail log file.
Lire can create reports in many formats including text, HTML and PDF. If you install lire with Ploticus, the HTML will even include plots. The reports are quite detailed with regard to number of messages to/from various domains and various users, and also include some simple statistics such as delay.
Lire includes support for saving the log analysis in an xml Dlf Store so that various reports can be made, including merging analyses for weekly, monthly and yearly reports.
However the poor documentation, tedious setup, cryptic error messages, and the fact that there has not been any further development work on lire since 2004 have all led me to abandon further use of lire.
Typical usage to view a command-line text report:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | /usr/local/bin/lr_desyslog qmail | /usr/local/bin/lr_log2report qmail | less
Some other email log analysis options to consider:
5) It would be nice if somebody would offer a simple and efficient package to get graphics as detailed and nice looking as shown in the article at http://forums.cacti.net/post-145427.html#145427 ... very nice apearance, a lot of info in a compact presentation.
6) Awstats provides detailed information about the number of emails being sent and received by the server, as well as the senders and recipients. However, the majority of the stats lump all of the incoming mail and outgoing mail together, simply calling it "sent mail", which makes the whole report rather useless. That's a pity, it would be quite uesful to see separate graphs of incoming and outgoing mail.
Unfortunately, Awstats does not provide any information at all about message delays, which are very important when trying to understand queue and delivery issues. Another unfortunate situation is that the plesk maillog does not contain any information about the number of spam rejects or the queue size, both of which would be nice to be able to keep an eye on. Nonetheless, since it's already installed, awstats does provide a convenient way to get some very basic email stats.
You can see a sample of a typical awstats email report at http://awstats.sourceforge.net/awstats.mail.html .
Enjoy!
Please feel free to add additional email log processing tips to this thread.
There have been many posts asking about email statistics, so here's a little report on the tools I've tried. All of the following are relatively simple schemes (no programming skills required) which readily produce email stats from Plesk's existing Qmail logs.
Note: The paths shown are for Plesk 8.3 with CentOS 5. For other systems there may be some variation in paths.
First a couple of simple, reliable approaches... these are my favorites:
1) This first scheme uses awk for filtering and qmailanalog (http://cr.yp.to/qmailanalog.html) for the log analysis.
Note: When compiling qmailanalog on CentOS 5, it was necessary to first apply the 0.70-errno.patch, see for example http://www.qmailrocks.org/qmailanalog.htm
Note: If using qmailanalog results in an error message such as 'sort: open failed: +2: No such file or directory', then you'll need to edit the three files zdeferrals, zfailures and zsuccesses in /usr/local/qmailanalog/bin, replacing 'sort +2' with 'sort -k 3', in zrhosts and zrecipients replace 'sort +4' with 'sort -k 5', and in zsuids replace 'sort -n +7' with 'sort -n -k 8'. See 'info sort' for more information about the sort problem (clue: the old sort used zero based numbering). To make these changes to the qmailanalog source before compiling, use the patch file at http://www.korsten.org/misc/qmailanalog-sort.patch .
To see a text report from the current log:
cat /usr/local/psa/var/log/maillog | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup | /usr/local/qmailanalog/bin/zoverall | more
To see an overall report from a zipped log:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup 5>/dev/null | /usr/local/qmailanalog/bin/zoverall | more
To see percentage distribution of delays:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup | /usr/local/qmailanalog/bin/zddist | less
To see delivery delay to each recipient:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:/ {$1="";$2="";$3="";$4="";$5="";print} ' | /usr/local/qmailanalog/bin/matchup | /usr/local/qmailanalog/bin/zrxdelay | less
There do not seem to be any features included in qmailanalog to automatically capture the fragments of messages which begin in one log and are resolved in the following log. Nonetheless, the various different qmailanalog reports are quite useful, and the number of missed emails at log rotation should typically be small compared to the daily total.
2) If you just want to run a quick sanity check, to get an idea if anything in the email system has gone berserk, here's a quick way using awk. This scheme might be used, for example, in a simple script which decides to send an email to an admin if the number of emails suddenly exceeds some reasonably expected threshold. (remember that unless the logs happen to be changed at precisely midnight, the log file will only include a portion of the first day and a portion of the last day of that log)
To see the total number of successes and failures included in the log:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:.*success:/ {s++} /qmail:.*failure:/ {f++} END {print s, "successes\n", f, "failures"} ' | more
Or, to see the number of qmail successes for each of the individual days which are included in the log:
outgoing success:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:.*success:.*accepted_message/ {print "qmail outgoing success on", $1, $2} ' | uniq -c | more
incoming and outgoing, success and failure:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | awk '/qmail:.*success:.*accepted_message/ {print "qmail outgoing success on", $1, $2} /qmail:.*success: did_/ {print "qmail incoming success on", $1, $2} /qmail:.*failure: Unknown_R/ {print "qmail incoming failure on", $1, $2} /qmail:.*failure:.*Remote_host_/ {print "qmail outgoing failure on", $1, $2}' | sort | uniq -c | more
Next, here are a couple of approaches that I've ended up abandoning.
3) I've also experimented with isoqlog http://www.enderunix.org/isoqlog/. The output is html, and provides only the most basic information about number of messages sent and received. Isoqlog is easy to install, especially using the instructions at http://www.qmailrocks.org/extra/isoqlog.htm
The html reports produced by isoqlog are very limited, showing only sent, received and total for each day. And worse yet, while isoqlog apparently works fine for some Plesk users, even with a cron call every hour on my system isoqlog is only counting about 10% of my server's actual mail. So... perhaps it may work for you, or perhaps not. Personally, I've given up on isoqlog.
4) Another scheme uses the lire (http://logreport.org/dl.html) routines for both the filtering and the log analysis. The log analysis package called lire-2.0.3 happily accepts the Plesk qmail log file.
Lire can create reports in many formats including text, HTML and PDF. If you install lire with Ploticus, the HTML will even include plots. The reports are quite detailed with regard to number of messages to/from various domains and various users, and also include some simple statistics such as delay.
Lire includes support for saving the log analysis in an xml Dlf Store so that various reports can be made, including merging analyses for weekly, monthly and yearly reports.
However the poor documentation, tedious setup, cryptic error messages, and the fact that there has not been any further development work on lire since 2004 have all led me to abandon further use of lire.
Typical usage to view a command-line text report:
gunzip -c /usr/local/psa/var/log/maillog.processed.1.gz | /usr/local/bin/lr_desyslog qmail | /usr/local/bin/lr_log2report qmail | less
Some other email log analysis options to consider:
5) It would be nice if somebody would offer a simple and efficient package to get graphics as detailed and nice looking as shown in the article at http://forums.cacti.net/post-145427.html#145427 ... very nice apearance, a lot of info in a compact presentation.
6) Awstats provides detailed information about the number of emails being sent and received by the server, as well as the senders and recipients. However, the majority of the stats lump all of the incoming mail and outgoing mail together, simply calling it "sent mail", which makes the whole report rather useless. That's a pity, it would be quite uesful to see separate graphs of incoming and outgoing mail.
Unfortunately, Awstats does not provide any information at all about message delays, which are very important when trying to understand queue and delivery issues. Another unfortunate situation is that the plesk maillog does not contain any information about the number of spam rejects or the queue size, both of which would be nice to be able to keep an eye on. Nonetheless, since it's already installed, awstats does provide a convenient way to get some very basic email stats.
You can see a sample of a typical awstats email report at http://awstats.sourceforge.net/awstats.mail.html .
Enjoy!
Please feel free to add additional email log processing tips to this thread.
