Too Far Afield

Introduction

The Exim mail server comes with out of the box quota support. All you need to do is to set the quota, quota_warn_threshold and quota_warn_message options on your transport and you will be fine (see Chapter 26 of the Exim Specification). Combine this with a few custom database lookups and you’ve got a great way to handle your user’s mailbox quota. By utilizing some additional options and the power of regular expressions you could even define different quotas for each folder of your user’s IMAP mailboxes.

The problem

Using Exim‘s quota management has a major drawback, though: Quota checking takes place after the message has been accepted for delivery. What does that mean in practise?

If the quota limit of a mailbox is exceeded, messages for this mailbox are still accepted for delivery, but deferred temporarily. At each queue run, Exim checks, if the quota limit is still exceeded. If the user has deleted mails from her mailbox since the last queue run, the mail will be delivered. If not, the mail will be deferred again. At some point, Exim will give up and finally bounce the mail back to the sender. How long Exim will try to deliver the message depends on your retry rules. While this is good for your user, because she has some more time to clear up her mailbox before she effectively looses mail, it is bad in a number of ways:

• The deferred mails add up to the queue and clutter it up.
• If the user does not delete messages from her mailbox (e.g. she is no longer using it actively), messages will sit in your queue for a long time, before they are finally bounced back to the sender. During this time, the sender does not know about the status of the mail. You need to provide disk space for these mails. If you set up quota limits to better manage the disk space at your disposal, this may hit you hard.
• If your user gets a lot of spam from faked sender addresses, you may generate a lot of collateral spam by bouncing her messages back to the faked sender addresses.

A possible solution

Exim were not Exim, if there wasn’t some way to get around this limitation and check quota right at RCPT time during the SMTP transaction. Actually, there are a number of ways to implement this. And all have pros and cons. I’d like to show you a way that is relatively easy to implement, but does have some minor issues.

Basically, the idea is to set up a database of those users that are over quota. You’d use an external script to periodically check, if a user exceeded her quota limit, and manage the database accordingly. You could use a flat file to store mailbox names of these users or – if you’re already using a relational database (e.g. MySQL, PostgreSQL) or an LDAP directory for mailbox management – add an additional property (e.g. isOverQuota) to your mailbox object.
Additionally, you need to configure Exim to check our file or database during the SMTP transaction and reject an incoming message, if it is for a mailbox that is over quota. How to do this depends a bit on your current configuration. Chances are, you’re already verifying the recipient of a message before you accept it (check your ACLs for require verify recipient). In this case, the fastest way to implement this in Exim is to add an additional router somewhere before (probably immediately before) the router that handles routing of messages to your user’s mailboxes (the router with the quota option). If you used a MySQL database it might look something like this:

1. virtual_mailbox_overquota:
2. driver = redirect
3. domains = +local_domains
4. condition = ${if eqi{$local_part} {${lookup mysql {SELECT username FROM exim.mailbox WHERE username='$local_part' AND domainname='$domain' AND enabled='1' AND overquota='1' LIMIT 1}}}{yes}{no}}
5. data = :fail: Mailbox is full, quota limit exceeded
6. allow_fail

Note: You’ll probably want to change the domains and condition options before you use this router in your configuration. But you get the idea, don’t you?

By default Exim calculates quota by adding the size of the new message to the size of the mailbox and comparing this to the quota limit of the mailbox (inclusive checking). This means that a user can’t go over quota. The mailbox will always remain below the quota limit. This is great, but it doesn’t work for us. Our script depends on users going over quota, because otherwise it would not detect the mailboxes that are over quota. Again, there are a number of possible solutions to this problem. You could increase the quota limit of every mailbox by 1% and check for mailboxes at 99%. Alternatively, you could disable inclusive quota checking. To do so, add the following option to the router, which handles routing of messages to your user’s mailboxes (the router with the quota option):

1. quota_is_inclusive = false

Testing the new configuration

To test, if the new configuration works, you might use Exim‘s basic address deliverability checking.

1. $exim -C exim.new.conf -f sender@example.org -bt not.over.quota@example.org
2. not.over.quota@example.org
3. router = virtual_mailbox, transport = appendfile
4. $exim -C exim.new.conf -f sender@example.org -bt exceeded.quota@example.org
5. exceeded.quota@example.org is undeliverable: Mailbox is full, mailbox quota exceeded

If you get anything close to the above, you’re probably okay. Of course you need to have a mailbox that exceeded its quota limit for this test to work.

Some issues

There are two minor problems with this solution. Depending on your situation, these might be acceptable or not.

First, there is the checking interval. If a user exceeded her quota limit, mail to her mailbox wouldn’t be blocked at SMTP time until the checking script’s next run. This is not such a big problem, because the message won’t be delivered anyway. Instead it would be deferred and bounced back later (see above). Unfortunately, it’s the same the other way around. If a user deleted messages from her mailbox, this would not be recognized by Exim before the checking script’s next run. Mail would be rejected at SMTP time, although there might actually be enough space in the mailbox. This is a much bigger problem. Generally spoken, the interval between the runs of your checking script should be as low as at all possible. But don’t burn your mail server on the way ;).

Second there is the exclusive quota checking. Because we depend on the mailbox actually going over quota, we need to disable inclusive checking or – alternatively – increase quota limits and flag mailboxes as over quota at 99% or less.
If we do the latter, there is always the possibility, that the quota of a mailbox isn’t above the checking threshold and there are new messages coming in, which are too large for the mailbox. These messages would be queued. Consider an example: A user get’s 30 messages of 1 GB, but his mailbox is only 500 MB. He gets no small messages in between, so our checking script will not flag the mailbox in subsequent runs and the size of the queue will rise a lot. This is especially a problem, because this probably affects mostly messages that are quite large. Also, if you can’t check fast enough and get a lot of small messages, there might be a lot of mailboxes that go way above the checking threshold before you can flag them. If you have a very large user base, this might be a problem. A run of the script probably takes longer, if you have a lot of mailboxes, so this adds to the problem.
If we disable inclusive checking, the last message that brings a mailbox over quota will always be delivered. Depending on the message this might be okay (in case of small messages) or really bad (in case of very large messages). But: The mailbox will be flagged at the next run of our checking script, so the queue will remain relatively clear compared to our scenario above.
If you do not have a lot of users and your resources are limited, you might not like this option. If you’ve got a large user base, exclusive checking is probably better.

Of course, there are other solutions, that do not have these issues. But they are a lot harder to implement and to maintain.