Creating a Linux Mail Server
(Postfix, Procmail, Fetchmail, SpamBayes, Courier-imap, Mutt, SquirrelMail)

What this Document is About

This document covers how I have set up my Linux box to work as a Mail Server. The details will be specific to Arch Linux, but it will be be general enough for any *nix distribution if you have a knowledge of installing software packages, finding the location of configuration files, and starting/stoping services..
My goals in creating this mail system are to have all of my mail, from various email accounts, stored locally on my machine so I can access it anytime via SSH, the web, or imap. I wanted all of my mail delivered to my local user account and I wanted the spam to be sucked out and launched into space.
The backbone of the mail system is the Mail Transport Agent (MTA). It will handle receiving mail addressed directly to your machine, sending it to the correct users mail box, and sending mail out from the machine. I determined Postfix to be the best choice for my MTA and after some reading found Procmail for sorting mail, SpamBayes for detecting spam, and Fetchmail for grabbing mail from my pop mailboxes.
Bellow is a flow diagram for mail moving through the system:

The SpamAssassin and sa-learn can be ignored. I will describe how to use it with SpamBayes since I was not happy with Spam Assassin's performance and had made the diagram when I was still using it.

Software Used

Arch Linux
Postfix 2.1.5
Cyrus SASL 2.1.21
Procmail 3.22
Fetchmail 6.25
SpamBayes 1.1a1
Mutt 1.4.2.1
Courier-imap 4.0.4
Courier-authlib 0.57
SquirrelMail cvs 1.5.1
Pop-before-smtp 1.38

Installing the Software

Installing Postfix

A note for those with dynamic IP addresses:

dyndns.org

$ pacman -Sy postfix

Arch Build System (abs)

http://wiki.archlinux.org/index.php/PostFix_Howto_With_SASL

$ pacman -Sy cyrus-sasl
$ abs
$ cd /var/abs
$ mkdir -p local/postfix
$ cp -v extra/daemons/postfix/* local/postfix

make OPT="${CFLAGS}" || return 1

make OPT="${CFLAGS}" \
CCARGS="-DUSE_SASL_AUTH -I/usr/include/sasl/" \
AUXLIBS="-L/usr/lib/sasl2 -lsasl2" || return 1

$ makepkg
$ pacman -A postfix-*.pkg.tar.gz

Note:

$ mkdir /var/lib/pacman/abs_built
$ cp postfix-*.pkg.tar.gz /var/lib/pacman/abs_built

Installing Fetchmail

$ pacman -S fetchmail

Installing Procmail

$ pacman -S procmail

Installing SpamBayes

$ wget http://dl.sourceforge.net/sourceforge\
/spambayes/spambayes-1.1a1.tar.gz
$ tar zxvf spambayes-1.1a1.tar.gz
$ cd spambayes-1.1a1
$ python setup.py install

Installing Mutt

$ pacman -S mutt

Installing courier-imap and courier-authlib

$ pacman -S courier-imap

Installing SquirrelMail

$ pacman -S squirrelmail

Installing pop-before-smtp

Postfix/TLS

pop-before-smtp

$ wget http://dl.sourceforge.net/sourceforge/popbsmtp/pop-before-smtp-1.38.tar.gz
$ tar zxvf pop-before-smtp-*.tar.gz
$ cd pop-before-smtp-*

Postfix Configuration

main.cf

'/etc/postfix'

myhostname = example.com mydomain = example.com

home_mailbox = Maildir/

mailbox_command = /usr/bin/procmail -a "$DOMAIN"

alias_maps = hash:/etc/postfix/aliases

relayhost = [smtp.sbcglobal.yahoo.com]

# authentication for sbc yahoo
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

That wraps up our changes to main.cf. We just need to create the hash aliases databases that we defined above. Add these lines to the top of /etc/postfix/aliases (replacing hypexr with your user account of course):

root: hypexr

spam: hypexr
ham:  hypexr

Generate the database. From /etc/postfix dir:

$ postalias aliases

Set up smtp server and authentication for SBC mail relay. sasl_passwd:

smtp.sbc.mail.yahoo4.akadns.net    my_username@sbcglobal.net:my_passw

Note: Server name can change from time to time. There is a CNAME that points to this name but CNAMEs do not work here.
Generate the database:

$ ./postmap sasl_passwd

Start the postfix server and we are done with postfix!!!!

$ /etc/rc.d/postfix start

Having problems getting it to start? Check out the Postfix log file usually in /var/log/mail or /var/log/maillog.

Fetchmail Configuration

We are going to run our fetchmail daemon from user root. So create the file /root/.fetchmailrc:

set postmaster "postmaster"
set bouncemail
set no spambounce
set properties ""
set daemon 60
poll pop.example.com with proto POP3
user 'user_name' there with password 'my_password' is 'local_user_to_deliver_to' here options flush
poll pop.gmail.com with proto POP3 port 995
user 'my_gmail_un@gmail.com' there with ssl with password 'my_password' is 'hypexr' here options keep

This tells fetchmail to check every minute for mail at pop.mypop.com and also at my gmail account. The flush option at the end of the mypop entry tells it to remove the mail after it is downloaded. I do not want the mail removed from my Gmail account so I use the keep option. Google also uses ssl, hence the "with ssl". Fetchmail comes with its configuration utility fetchmailconf which makes setting up your .fetchmailrc file easy.
I don't believe that Arch Linux has a rc script for fetchmail. The one that I made (/etc/rc.d/fetchmail) looks like this (stat_busy, stat_fail, stat_done are specific to Arch Linux. You could take these out):

#!/bin/sh

. /etc/rc.conf
. /etc/rc.d/functions

DAEMON_NAME="fetchmail"
DAEMON_CONF="/root/.fetchmailrc"
DAEMON_PATH="/usr/bin/fetchmail"

PID=`pidof -o %PPID ${DAEMON_NAME}`

case "$1" in
start)
   stat_busy "Starting ${DAEMON_NAME}"
   [ -z "$PID" ] && ${DAEMON_PATH} -f ${DAEMON_CONF} -d 60 &> /dev/null
   if [ $? -gt 0 ]; then
      stat_fail
   else
      add_daemon ${DAEMON_NAME}
      stat_done
   fi
   ;;
stop)
   stat_busy "Stopping ${DAEMON_NAME}"
   [ ! -z "$PID" ]  && kill "$PID" &> /dev/null
   if [ $? -gt 0 ]; then
      stat_fail
   else
      rm_daemon ${DAEMON_NAME}
      stat_done
   fi
   ;;
restart)
   # calling 'stop' and 'start' without the $0 fails...
   $0 stop
   sleep 3
   $0 start
   ;;
*)
   echo "usage: $0 {start|stop|restart}"
esac
exit 0

And start the fetchmail daemon:

$ /etc/rc.d/fetchmail start

Or if you don't want to create/modify an rc script you can start the fetchmail daemon like:

$ /usr/bin/fetchmail -d 60

Spambayes and Procmail Configuration

We are going to be using Spambayes to train and determine if mail is spam and Procmail to sort the mail into folders when they enter our mail system. These two go together as far as Procmail is going to use Spambayes to help sort the mail.
I had used SpamAssassin for a year and had not been very happy with its performance. I have only been using Spambayes for a few weeks and have been pleasantly surprised by how well it has worked and how quickly it adapts. In fairness I had been training SpamAssassin on all of my spam, the spam it had caught and the spam it had missed as well. It was also an older version and I was not using all of the features. With that being said if you are curious give it a try. Many other excellent programs exist. A quick search on Google will turn them up.
When Spambayes is installed it is going to move several files that start with sb_ into your /usr/bin directory. We are going to be interested in sb_mboxtrain.py and sb_filter.py. The former trains the spam filter and the later classifies incoming spam. There are two methods I tried for the initial training:

Since I already had thousand of spam and ham (non spam) messages saved up I first tried initially training on this older mail. The results were okay but it was miss sorting mail. These problems would have sorted them selves out with the continued training of the misclassified mail but it would have also took a little while because the database had already been trained so much.
The second method I used was extremely effective. I started out with no training on initial messages. I simply let Spambayes work with no information. I then sent the spam and ham messages that ended up in the wrong mailbox to a designated folder. I then run the Spambayes training program on these miss classified messages.

The results from the second method have been amazing. Without even having to manually classify very many messages Spambayes has been performing at superior levels. Check out Spambayes Wiki for many other training methods. I will discuss easy methods of sorting miss classifications in the Mutt and Squirrel Mail sections.

Spam System Initial Setup

$ mkdir ~/Maildir
$ mkdir ~/Maildir/{new,cur,tmp}

$ mkdir ~/Maildir/{.MissedSpam,.MissedHam,.CaughtSpam,.Unsure}
$ mkdir ~/Maildir/.MissedSpam/{new,cur,tmp}
$ mkdir ~/Maildir/.MissedHam/{new,cur,tmp}
$ mkdir ~/Maildir/.CaughtSpam/{new,cur,tmp}
$ mkdir ~/Maildir/.Unsure/{new,cur,tmp}

#!/bin/sh
# Script to copy mail missed spam and ham into correct folders
# and run sb_mboxtrain.py to train spambayes
# Training will be done only on missed spam and ham

# Files we saved or bounced may be in the new directory
# We want them in cur for training
mv /home/hypexr/Maildir/.MissedSpam/new/* \
 /home/hypexr/Maildir/.MissedSpam/cur 2>/dev/null
mv /home/hypexr/Maildir/.MissedHam/new/* \
 /home/hypexr/Maildir/.MissedHam/cur 2>/dev/null
/usr/bin/sb_mboxtrain.py -d /home/hypexr/.hammie.db \
 -g /home/hypexr/Maildir/.MissedHam \
 -s /home/hypexr/Maildir/.MissedSpam

$ chmod +x ~/Maildir/train_spambayes

$ /usr/bin/sb_filter.py -d $HOME/.hammie.db -n

crontab -e

21 3 * * * ~/Maildir/train_spambayes

Procmail Configuration for Spambayes

Spambayes on Unix or Linux

SHELL=/bin/sh
MAILDIR=$HOME/Maildir
DEFAULT=$HOME/Maildir/
CAUGHT_SPAM=$MAILDIR/CaughtSpam/
UNSURE=$MAILDIR/Unsure/
  
#Spambayes process
:0fw:hamlock
| /usr/bin/sb_filter.py -d /home/hypexr/.hammie.db

:0
* ^X-Spambayes-Classification: spam
${CAUGHT_SPAM}

:0
* ^X-Spambayes-Classification: unsure
${UNSURE}

# Catches everything else.
# Anything left over goes into the DEFAULT folder
:0:
${DEFAULT}

Mutt Configuration

First we need to set the MAIL variable so that Mutt will know where to look for your mail. In your ~/.bashrc (create if does not exist) add the line:

MAIL='~/Maildir'

Exit out of your shell and log back in so that the variable will be set or run export MAIL='~/Maildir'. On some systems mutt will not use $MAIL to determine the location of your mail box. When trying to start mutt if it reports a message that no mail directory exists you can try setting the MAILDIR variable or just create the directory that it wants to use and make it a valid Maildir: mkdir -p ~/.maildir/{cur,new,tmp} where .maildir is the folder that mutt wants to use. If you use a directory other than ~/Maildir remember to substitute that wherever I reference Maildir in this document.
Now for the fun part. Lets make everything look pretty and integrated in Mutt. If you start up Mutt right now you will see any mail that your system has received since you have completed the steps above. If no mail shows up in Mutt and there should be mail you need to first check if Spambayes has filtered it into your .Unsure or .CaughtSpam directories. Type 'c' to change directories and enter ~/Maildir/Unsure and check out the mail thats been classified as spam: 'c' ~/Maildir/CaughtSpam. If you still have not found the mail that you are expecting to have received make sure that you have configured Postfix, Fetchmail, and Procmail correctly. Also, try sending mail directly to your_user_accunt@example.com and see if it shows up in one of the directories above.

Now mail is flowing into your system and we can check how Spambayes has classified it by checking the different mail folders. The miss classified mail just needs to be moved into the correct folders for the training.
To manually move the misclassified spam mail:

From you Maildir inbox tag each message that has arrived that is spam. Do this by pressing 't' and an asterisk will appear by the messages. After all of the messages have been tagged, press '; m' and enter ~/Maildir/MissedSpam and the spam messages in your inbox will be moved to the MissedSpam mailbox for training by spambayes.
When looking at the mail that it has classified as spam in '~/Maildir/CaughtSpam', tag all of the messages it considered spam that were ham and press '; C' and enter ~/Maildir/MissedHam. Now tag them again and press '; m' and enter ~/Maildir to move them into your inbox, where they should have been placed in the first place.

Of course all of this manual stuff is a huge pain so lets automate all of it in .muttrc.

.muttrc Configuration

.muttrc

# Emacs is my editor of choice.  I have mostly read that people like
#  using emacs-client here and using an emacs server so that the
#  editing opens up in an existing emacs session but I prefer
#  loading up a fresh instance. The default editor is some 'vi'
set editor="emacs"

# Setting these macros is going to save us all of that time I was
#  talking about.  Now instead of having to type all of those
#  characters to move the mail around we will can just tag the messages
#  and, for the instance of spam in your inbox, hit 'S'.
  
# Move mail to correct directories
macro   index   S       ";s~/Maildir/.MissedSpam\r\r$\r"
macro   pager   S       ";s~/Maildir/.MissedSpam\r\r$\r"
macro   index   H       ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro   pager   H       ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro   index   M       ";s~/Maildir/\r"
macro   pager   M       ";s~/Maildir/\r"

# This one lets me quickly move job search related emails to the
#  correct directory.  Want to give me a job? :)
macro   index   B       ";s~/Maildir/.Job\r\r$\r"
macro   pager   B       ";s~/Maildir/.Job\r\r$\r
  
# This is how we are going to move around to the different
#  mailboxes.  Hitting 'alt-1' will take us to our inbox
#  'alt-2' and we are looking at the mail Spambayes classified
#  as spam.  etc.
#  The extra mailboxes I have in here are for mailing lists I am on.
#  I will show the procmail recipe that automatically puts
#  the mailing list mail into its correct mailbox further down.
  
# Move to mailboxes quickly
macro   index   <Esc>1  "c~/Maildir\r"
macro   pager   <Esc>1  "c~/Maildir\r"
macro   index   <Esc>2  "c~/Maildir/.CaughtSpam\r"
macro   pager   <Esc>2  "c~/Maildir/.CaughtSpam\r"
macro   index   <Esc>3  "c~/Maildir/.Unsure\r"
macro   pager   <Esc>3  "c~/Maildir/.Unsure\r"
macro   index   <Esc>4  "c~/Maildir/.Tur\r"
macro   pager   <Esc>4  "c~/Maildir/.Tur\r"
macro   index   <Esc>5  "c~/Maildir/.Sbayes\r"
macro   pager   <Esc>5  "c~/Maildir/.Sbayes\r"
macro   index   <Esc>6  "c~/Maildir/.Wikka\r"
macro   pager   <Esc>6  "c~/Maildir/.Wikka\r"
# Training Maildirs for Spam and Ham
macro   index   <Esc>8  "c~/Maildir/.MissedSpam\r"
macro   pager   <Esc>8  "c~/Maildir/.MissedSpam\r"
macro   index   <Esc>9  "c~/Maildir/.MissedHam\r"
macro   pager   <Esc>9  "c~/Maildir/.MissedHam\r"
# Job E-mails
macro   index   <Esc>0  "c~/Maildir/.Job\r"
macro   pager   <Esc>0  "c~/Maildir/.Job\r"

# Setting these colors makes it easy to tell which emails
#  Spambayes has missed.
# Turn spam red and unsure green
color index red default "~h '^X-Spambayes-Classification: spam'"
color index green default "~h '^X-Spambayes-Classification: unsure'"
# Turn gmail mail brightblue
#  (My regular expression is not correct here.  I think that it
#   is looking anywhere in the header for my gmail address
#   instead of what I wanted..  Just in the 'To:'.
#   It still works for the most part)
color index brightblue default "~h '(^)*my_name@gmail.com'

# Lets set more colors to make things look beautiful.  Judging by
#  my title for this section I must have had bad feelings towards
#  colors when I added this.
#Color crap
color index brightwhite default ~N       # color for new messages
color status black yellow
color attachment brightyellow default    # file attachments
color search brightred default           # search matches
color quoted brightyellow default        # quoted text in replies
color quoted1 magenta default            # quoted text in replies
color body cyan default "((ftp|http|https)://|news:)[^ >)\"\t]+"   # URLs
color body cyan default "[-a-z_0-9.+]+@[-a-z_0-9.]+"               # email

# The rest of this stuff is pretty important for functionality and
#  readability.

# Don't ask to move read message
set move=no

# My signature is in this file
set signature="~/.signature"

# Change mail to look like from scott@hypexr.org
#  Stick your email address here.
my_hdr From: Scott Rippee <scott@hypexr.org>

# Header control   h displays header when in pager
# I want to only see the unignored by default
ignore *                                        # weed out all headers
unignored date from: to cc subject organization  # now show me these...

# Thanks to many people whom have written the documents and posts that I
#  have gotten these settings from.

New Mutt Commands

t       Tag messages to be manipulated
S       Send message/s to spam training folder
H       Send message/s to ham training folder and to inbox
alt-1   Change mailbox to Maildir/ (inbox)
alt-2   Change mailbox to .CaughtSpam
alt-3   Change mailbox to .Unsure
alt-8   Change mailbox to .MissedSpam
alt-9   Change mailbox to .MissedHam

SquirrelMail Configuration

To configure SquirrelMail go into your html_base_directory/squirrelmail/config and run ./conf.pl. Edit the options to fit your needs. If you want to test your config cruise over to http://localhost/squirrelmail/src/configtest.php.
Now go to http://localhost/squirrelmail/ and login using your user account. If you have any email in your in box you should see it after you log in. Now click on the Folders option that is along the top, select all of the mailbox folders that we created earlier (they are listed here because we prefixed them with a period), and click on subscribe. Hit refresh and the mailboxes should be listed in the side panel. Now you can use SquirrelMail for almost all of you mail needs including tagging messages and moving them to the correct folders for Spambayes training and checking on the mail that Spambayes has classified as spam and unsure. The only functionality from our mutt setup that can't be duplicated in SquirrelMail is copying missed ham into its training folder and then moving it into your inbox. This is due to the lack of copying ability. So you will have to go into the ham training folder to see some of the mail that should be in your inbox.
!!! IMPORTANT SECURITY NOTE!!! - Using SquirrelMail like this is not secure! Sending the plain text passwords that you use for your Unix account can be accessed by by a 3rd party, which would them access to your account. You should set up a secure server with SSL and have Courier-imap use passwords that differ from the users system password using Courier Authlib, which I will demonstrate in the next session. Excellent documents exist all over the net for setting up your web server with ssl.

Courier Authentication

Now is a good time to address a bit of essential security. As of now when a user connects via the SquirrelMail or IMAP interface the passwords are being sent in plain text and if these are sniffed that users system account is compromised. Without adding secure connections we can minimize the threat by using alternate passwords for imap connections. The snoop would then have access to your mail, but not be able to log in to the system.
Edit your authdaemonrc file. If your on Arch Linux this will be in /etc/authlib. Under authmodulelist I commented out:

authmodulelist="authpam authpwd authuserdb authshadow ..."

And replaced it with:

authmodulelist="authuserdb"

I believe that adding authuserdb to the front of the list may work, but I am only using this method of authentication so it is all I need in my list.
Lets create the user account entry:

$ userdb "some_user_name" set home=/home/user_dir \
  mail=/home/user_dir/Maildir \
  uid=users_userid gid=users_groupid

$ userdbpw | userdb "some_user_name" set systempw

You will now be prompted for this pseudo users new password. Next make sure that /etc/authlib/userdb has read/write/execute access for the owner. If not chmod 700 /usr/authlib/userdb. Now create the user database that courier is going to look to for authentication:

$ makeuserdb

Again look to using Postfix/TLS for a tunneling the connection through a secure connection.

Pop-before-smtp Configuration

To successfully configure pop-before-smtp follow the straight forward instructions in contrib/README.QUICKINSTALL. When you are done and everything is working it is important to make sure that you have not some how made your mail server an open relay! This would cause your mail server to be flooded with traffic and be put on the open relay ban lists.
!!! IMPORTANT SECURITY NOTE!!! - I will not use the pop-before-smtp method because it involves sending a password over an insecure connection. Use Postfix through a SSL connection, Postfix/TLS, to accomplish this. Since your local machines are likely already allowed to use your Postfix as a mail relay the only time that you would be using this method is from a non-trusted server where you password can be sniffed. If you still choose to use this method make sure that you have changed the users courier password as described in Courier Authentication.

Additional Simple Procmail Recipes and Mailbox Config

If you are a member of any news letters/mailing lists you can have them automatically sorted to their own mailbox using Procmail. First create the new mail directory in your ~/Maildir directory. Lets call it .Sbayes and pretend that we are on the Spambayes mailing list. So exactly as before with the spam related mailboxes:

mkdir -p ~/Maildir/.Sbayes/{cur,new,tmp}

Now add this new rule to your ~/.procmailrc:

# spambayes-dev@python.org mailing list
:0
* ^X-BeenThere: spambayes@python.org
${MAILDIR}/.Sbayes/

In the .mutrc example above it shows an example of setting alt-5 as the macro to see your Spambayes emails.

A slightly different method can be used to send spam and ham for training. Adding the following lines will allow you to bounce spam messages to spam@localhost (localhost or your mail_domain.com) and ham messages to ham@localhost:

# Mail has been bounced to spam@localhost
:0
* ^Resent-To:.*spam
${MAILDIR}/.MissedSpam

# Mail has been bounced to ham@localhost
:0
* ^Resent-To:.*ham
${MAILDIR}/.MissedHam

If you want to bounce messages to the spam and ham addresses like this, you will need to make sure that the headers stay intact. Do not try using this or a similar technique for forwarding mails to an address because the email's headers would not be intact and this is a large part of the spam classification. This method works because above we specified spam and ham as aliases for hypexr in the Postfix aliases file.

Conclusion

For Arch Linux users, Add saslauthd authdaemond courier-imap postfix fetchmail to DAEMONS in your /etc/rc.conf file and they will be started automatically when your machine boots. If you are running a web server have the httpd service listed after the services we are adding.
/etc/rc.conf

DAEMONS=(... saslauthd authdaemond postfix courier-imap fetchmail ...)

Enjoy your new mail system and centralized mailbox. If you are handling a large number of users for you email system this is just the tip of the iceberg and you are going to mostly be interested in virtual users, security, and a system wide spam classifier. There are many good documents on various setups to accomplish this, see the links section.

Wednesday, 7 November 2012

Creating a Linux Mail Server (Postfix, Procmail, Fetchmail, SpamBayes, Courier-imap, Mutt, SquirrelMail)