January 25, 2021
Hot Topics:

Building a high capacity e-mail system

  • By Simon Horman
  • Send Email »
  • More Articles »

E-mail continues to be one of the most popular services used on the Internet. According to research from Internet service provider UUNET, "E-mail comprises the bulk of network traffic and has increased an average of 20% a year."

This tremendous growth means that service providers frequently find that the server where users' mail is stored can no longer keep up with the traffic. When upgrading the server is no longer practical, they face the choice of buying a larger machine or deploying multiple mail servers. But can you provide end users with a homogeneous view of their mail when more than one server is handling mail for the domain?

You can--and it's not hard to do. This article will explain how to use existing technologies and protocols to distribute incoming e-mail messages across multiple servers. This approach is designed to handle users who are in a single domain, and not to handle distributing messages to users on a mailing list, which is an entirely different problem.

Multiplexing SMTP

Simple mail transfer protocol (SMTP) traffic can be split across multiple machines by assigning multiple IP addresses to the primary mail exchanger (MX) for a domain. Incoming messages can also be multiplexed using layer 4 switch technology, such as that provided by the Linux Virtual Server Project.

Once e-mail arrives on one of the lowest preference mail exchangers for a domain, it may need to be relayed to another server. There are various ways of dividing up the e-mail, but regardless, the mechanism for determining the end destination for an e-mail is the same.

Rule-based multiplexing

One way to split e-mail between back-end servers is to make an arbitrary division in the address space for users and route mail based on this. With two servers, for instance, you could decide that addresses beginning with the letters A-K have mail delivered to one server and mail for all other addresses is delivered to the other server. Under Sendmail1 a simple rule in the Sendmail configuration file would divide the mail, like this:

Kalf regex -s1 -a<@alf.bigisp.com.> (^[a-k][^@]*)<@bigisp.com.>
Kbarney regex -s1 -a<@barney.bigisp.com.> (^[^a-k][^@]*)<@bigisp.com.>

These maps define how an address will be translated. The alf map looks for addresses that begin with A-K and are addressed to @bigisp.com. If this is found then an address @alf.bigisp.com is returned. The barney map looks for addresses that do not begin with A-K and are addressed to @bigisp.com and returns an address @barney.bigisp.com. To use regex maps you need either Sendmail 8.8 patched with map-regex or Ssendmail 8.9. In either case you need to compile with DBMDEF= -DMAP_REGEX set in the Makefile.

These maps should be placed under the definition of Dn in sendmail.cf or under LOCAL_CONFIG in sendmail.mc if the m4 preprocessor is used to generate sendmail.cf. Now place the following under ruleset 98 in sendmail.cf or LOCAL_RULE_0 in sendmail.mc.

R$*                  $: $(alf $1 $)
R$*                  $: $(barney $1 $)
RERROR $*            $#error $: $1

The first line will apply the alf map which translates users beginning with the letters A-K. The second line translates all other users as defined by the barney map. The RERROR line causes the rules to abort if an error has been encountered.

Per-user_map multiplexing

The rule-based solution is simple to implement, but lacks flexibility. Making more complicated rules soon defeats the elegance of this solution as maintaining the rules would become a major chore.

A more flexible approach is to define a map that can redirect each user's mail to the server that their mail is hosted on. If a user's mailbox needs to be moved to another server, then it is a simple matter of changing the entry for that user. The map should be in the form of a hash, to enable fast retrieval of data for an individual user. Conveniently, many message transfer agents (MTAs) including Sendmail and qmail using the fastforward add-on provide such a map in the form of an alias file.

Hybrid multiplexing

The alias mechanism provides great flexibility in assigning users to a mail server. However, it places a burden on administrators to ensure that all users have an alias. Users who receive mail and do not have a valid alias will -- depending on the mail server setup -- have their mail delivered locally, on whichever mail server it arrives on, or the mail will be rejected. This onus is particularly great when a new user is added to the system, in which case it is preferable for users' mail to work with minimal configuration.

A hybrid system that combines simple rules with aliases that can override the rules provides both simplicity and flexibility. New users to the system are covered by the general rules and users whose mailbox has been migrated for one reason or another are covered by their own alias. This can be achieved with the following;

Kalf regex -s1 -a<@alf.bigisp.com.> (^[a-k][^@]*)<@bigisp.com.>
Kbarney regex -s1 a<@barney.bigisp.com.> (^[^a-k][^@]*)<@bigisp.com.>
Kuser_map hash /etc/mail/user_map

R$+ < @ $+ > $*      $: $(user_map $1 $: $1 < @ $2 > $3 $)
R$*                  $: $(alf $1 $)
R$*                  $: $(barney $1 $)
RERROR $*            $#error $: $1

The rules are the same for rule-based multiplexing, with the addition of a map to allow per-user server assignment. This is analogous to the aliases file but is a separate map for greater flexibility. The flat file user_map is built into a hash using the command:

makemap -v hash per_user.db < user_map

The general rules will only take effect in the absence of a valid entry in the user_map. Per-user server assignments are processed first, and if the address to which the user_map entry points is not @bigisp.com then the more general mapping will not take place.

Page 1 of 2

This article was originally published on July 7, 2000

Enterprise Development Update

Don't miss an article. Subscribe to our newsletter below.

Thanks for your registration, follow us on our social networks to keep up-to-date