Mail Stats

I wrote this script during swotvac to parse my sendmail logs and report on the number of emails processed and spams tagged during the preceding day. I'd just implemented SPF, so I was interested in determining how widely adopted it is. It would also be interesting to count how much ham and spam would have been reject by SPF - I may add that one day.

It uses a patched version of logtail, but works with the normal one if you don't care about rotation issues.

 #!/usr/bin/perl -w
 # Check maillogs and generate nice spam summary statistics

 use IO::File;

 # Variables

 my $LOGTAIL="/root/logtailng.pl";
 my $LOGTAILSTATE="/root/maillog.state";
 my $LOGFILE="/var/log/mail/mail.log";
 my $LOCALDOMAIN="(localhost|tristesse\.org)"; # Assume these are local

 # Things we want to look out for

 my @userunknown;

 my $spffail = 0;
 my $spfnone = 0;
 my $spfpass = 0;
 my $spfneut = 0;
 my $acceptedcount = 0;
 my $sentcount = 0;
 my $spamcount = 0;

 # Code goes here

 my $fd = new IO::File;

 $fd->open("$LOGTAIL -f $LOGFILE -o $LOGTAILSTATE |");

 while (<$fd>) {
        if (/User unknown/) {
                my ($user) = /\<([^>]+)>/;
                push @userunknown, $user;
                next;
        }

        if (/identified spam/) {
                $spamcount++;
        }

        if (/stat=Sent/ && / sm-mta/) {
                # Locally sent mail will also produce a sendmail line.
                # Ignore duplicates.
                if (/to=[<]*[^\@]+\@$LOCALDOMAIN[>]*/) {
                        $acceptedcount++;
                } elsif (/to=[<]*[^\@]+[>]*[,\s]+/) {
                        # Match things like to=joel, to=root
                        $acceptedcount++;
                } else {
                        $sentcount++;
                }
        }

        if (/Received-SPF: (\w+) /) {

                # Sending locally triggers spfmilter. Filter it.
                next if /localhost is always allowed/;

                my $spfstat = $1;
                if ($spfstat eq "none") {
                        $spfnone++;
                } elsif ($spfstat eq "softfail") {
                        $spffail++;
                } elsif ($spfstat eq "pass") {
                        $spfpass++;
                } elsif ($spfstat eq "neutral") {
                        $spfneut++;
                }
        }

 }

 $fd->close;

 my $spampercent = 0;
 $spampercent = sprintf("%.3f", $spamcount/$acceptedcount * 100) unless $acceptedcount;

 print <<EOF
 In summary:

 Mail accepted $acceptedcount
 Mail sent     $sentcount
 Spam tagged   $spamcount
 Spam %        $spampercent

 SPF Failures: $spffail
 SPF No record $spfnone
 SPF Passes    $spfpass
 SPF Neutral   $spfneut

 Unknown users $#userunknown

 EOF
 ;

Here is a "spam for today" script that reads Procmail logfiles. It is incredibly evil. You will need to customise it, so don't be scared! - WzDD

 #!/bin/sh

 TODAY=`date +"d"`
 TODAY=$TODAY.\*`date +"%Y"`

 TOTAL=`grep "$TODAY" ~/.procmail-logfile |wc -l`
 SPAM=`grep -A 2 "$TODAY" .procmail-logfile |grep "Folder: spam"|wc -l`
 PERCENT=`python -c "print '%2.1f' % (float($SPAM)*100/$TOTAL)"`
 echo Received $TOTAL in total, $SPAM spam \($PERCENT% spam\)