Robert is maintainer of the Linux Frequently Asked Questions with Answers FAQ on the Internet. He can be contacted at firstname.lastname@example.org.
Without a doubt, e-mail traffic is on the rise. And partly due to security concerns, the number of available e-mail relay packages has increased as well. But these packages have introduced interoperability challenges for programmers who need compatibility with existing e-mail software.
For instance, there are a handful of free software mail transport agents that provide mail relaying for both individual systems and large backbone sites. In this article, I'll describe the relaying capabilities and idiosyncrasies of sendmail and qmail, two of the most popular mail transport agents.
All popular relay software primarily uses domain addressing, which works best with hosts that have a continuous TCP/IP connection to the Internet. Unfortunately, many users still have dial-up telecommunication reminiscent of UUCP addressing and static host map routing. Unless system administrators have the facilities to provide their own relay hosts and name servers, they are more likely to face incompatible, competing services from different vendors. Adding to this complexity is the lack of standardized authentication methods for SMTP message transfers. And considering that mail relay software has to cope with clients and servers of half a dozen or more message transfer protocols (SMTP, ESMTP, POP3, IMAP, UUCP, HTTP, and the like) as well as spam filters, mailing list servers, mobile dial-up providers, dynamic address pools, and local, user-level routing software, it's no surprise that the dreaded "Relaying Denied" error message is familiar to users and sysadmins. The end result is that many administrators resort to security methods that inadvertently restrict relaying of valid messages.
The standard sendmail program lost its default relay configuration the "promiscuous_relay" option in Version 8.9. However, sendmail still contains legacy code that provides UUCP-style addressing and routing information that has been deprecated in more recent standards. The configuration options are still present, but SMTP envelope addressing is now the default transfer method, and arriving at a configuration that satisfies the needs for one or more delivery protocols can be difficult. To fill the gap, programs like qmail have appeared. These programs contain features designed to facilitate the handling of mailing lists and dial-up SMTP queues. However, even with the simplified options that qmail offers, you still need to be aware of the configurations of other relay hosts.
In this article, I focus on the cause of relay failures incompatibilities between domain-compatible envelope addressing schemes and examine DNS factors only as they relate to specific configurations of e-mail relay software. In the process, I present a Perl/Tk e-mail client called "ec" (available electronically; see "Resource Center," page 5), which provides SMTP and POP3 e-mail communication that adheres as closely as possible to the standards covered in this article.
Internet Standard Address Formatting
Although e-mail clients and servers adhere to the specifications first defined in RFC 822 (Standard for the Format of ARPA Internet Text Messages) and RFC 821 (Simple Mail Transport Protocol), different programs' interpretations of the standards can make e-mail message handling one of the more arcane areas of software design. For instance, while RFC 822 specifies a strict format for e-mail headers, it does not specify routing or delivery methods. The SMTP Standard does not allow host aliasing, and the lack of a standard authentication procedure means that relay servers often overcompensate for the lack of security by making their relaying rules too restrictive. These and other factors make it difficult for e-mail relay programs to deliver all the different, yet valid, e-mail messages it might encounter.
Not all of the possible message fields defined by RFC 822 need to be present for the message to be valid. Before it can be processed by mail relay software, a header need only contain the address of the recipient (the To: field), the address of the sender (often, but not always, in the From: field), and an RFC-compliant date/time (the Date: field) from the originating e-mail program. But even messages with headers that do not contain this information can be processed. If a domain name is absent, e-mail software can try to deliver the message to a mailbox in a local domain, or it can add a domain address depending on the relay software's configuration.
If a message header does not contain a From: field, it is up to the e-mail software to add a Sender: field to the message. Only one of these fields is required, but e-mail programs must add a Sender: field that contains a valid domain address, especially if a domain address in the From: field contains only the sender's user name or if the field is absent. Figure 1 is a typical e-mail message header, before the message is sent toward its destination. The ordering of the fields is only recommended; RFC 822 does not require that fields appear in any particular order.
The first step is determining the sender and recipient addresses (called the "reverse path" and "forward path," respectively) after they are translated into domain addresses. Considering that an e-mail header's address fields can consist of a sender's name and e-mail address, and that multiple forward paths may be specified, the procedure for parsing the address fields can become complicated because the software must take into account the entire grammar specified in RFC 822.
For instance, the Perl subroutine in Example 1 provides the user's full name and the envelope address (the portion that is to be enclosed in angle brackets < and >) that has any of these forms:
- From: Robert Kiesling email@example.com
- From: Robert Kiesling <firstname.lastname@example.org>
- From: email@example.com
- From: <firstname.lastname@example.org>
The subroutine must return at least the envelope address that relay software can recognize and resolve as a valid domain name, which is the argument to the SMTP "RCPT TO:" in Figure 2. The user's real name need not be present in the message field for delivery. The subroutine does assume, however, that the address is composed entirely of ASCII characters, as required by RFC 822.
Additionally, the e-mail software either the mail user agent that creates the message or the local transport software, via aliasing must take into account multiple addresses in fields, as well as different syntactic formats within individual addresses. This formatting, called "address unfolding," requires a careful look at the entire e-mail header so that all of the message's recipients will receive it, even though only one To: field is recognized.
For example, if a "To:" header line spans multiple text lines and contains extra whitespace such as in Figure 3(a), then the e-mail software must be able to concatenate the addresses into something that a relay agent can recognize. Ideally, a correctly formatted list of addresses should look something like Figure 3(b). This can usually be performed with a subroutine like Example 2, which looks at each line of a header for the lines of the field given by the $fieldname argument, and concatenates them and removes excess whitespace with a simple regular expression substitution from the perlre man page.
Of course, address-formatting routines must still take into account the user's full name if it is present, as well as the mailbox name as provided by a routine like Example 1.
As a practical matter, address unfolding must be handled before e-mail software can begin translating user mailbox addresses into envelope addresses; see Example 3. Finally, a remote SMTP server must receive a list of domain address routes, each an argument to the SMTP RCPT TO: command at the server. (The backslashes indicate line breaks.)
Optional Message Fields
The information presented to this point applies equally to optional fields that require one or more mailbox addresses that can be translated into envelope addresses as operands. The most commonly used optional fields include: Cc:, Bcc:, and Reply-To:.
Fields that can replace a To: field include Apparently-To: (which is handled by sendmail's NoRecipientAction option), and Errors-To: (handled by the ErrorsTo option), although their use is deprecated. In addition, RFC 822 defines the fields Resent-to:, Resent-cc:, and Resent-bcc:.
Qmail writes an X-Apparently-To: envelope field to recognize similarly ambiguous envelope addresses and detect circular forwarding loops.
Because RFC 821 does not allow host aliasing, the envelope addresses must be present before a message is sent to a remote host. If the person creating the message specifies a recipient as only "rkiesling," it is up to the local mail software to provide the domain address. This process is complicated by the fact that a host with a dial-up Internet connection may not have access to a domain name server, or a user's login name may not match his or her mailbox name, and the host may need to serve a virtual domain of its own.
The process of envelope rewriting (masquerading) is the cause of much confusion for e-mail software (not to mention system administrators). To add to the confusion, each relay program has its own procedures.
Sendmail Address Rewriting
A quick look at the sendmail man page reveals the -f command-line option, which tells sendmail what address to use for a message's sender. Here's one solution, where a host name can be provided on the command line:
$ cat email.msg | sendmail -f email@example.com
Sendmail then sets the Sender: field to "firstname.lastname@example.org," but it will also add this warning in the message header:
X-Authentication-Warning: rkiesling set sender to email@example.com
A more secure solution is required for standard message handling. Sendmail lets you add option macros when building the sendmail.cf configuration file. In addition, sendmail configurations include "rulesets" to rewrite local and Internet addresses when necessary, although custom rulesets are used primarily to accommodate legacy UUCP addresses.
The process of substituting one host or domain for another is called "masquerading," and is usually defined in the preprocessed source of the sendmail.cf file. Table 1 lists the basic sendmail macros for defining canonical envelope addresses.
In Table 1, messages from hosts within the ftp.mainmatter.com domain would be shown as originating from mainmatter.com, except for mail.mainmatter.com. The file /etc/mail/masq contains a list of host names to masquerade. Any of the macros that take a domain address as an argument can also accept a list of addresses separated by spaces.
The feature always_add_domain adds the domain mainmatter.com even to local addresses. The allmasquerade option adds the MASQUERADE_AS domain to recipient addresses. The limited_masquerade feature masquerades only those hostnames given as arguments to MASQUERADE_DOMAIN macros. However, if the masquerade_entire_domain feature is set, the hostnames in the MASQUERADE_DOMAIN macro arguments are treated as complete domains.
These options, as well as the procedure for generating the sendmail.cf configuration, are described in the cf/README file of the sendmail source-code distribution.
Message Relaying with Sendmail
Sendmail relay configurations must define a map of allowed domains, listed /etc/mail/relay-domains, and an access list, /etc/mail/access by default, which define the remote hosts for whom relaying is permitted. Sendmail uses these features to configure relay behavior:
As mentioned, the promiscuous_relay configuration (which creates a so-called "open relay" host) was the default configuration up to sendmail Version 8.9. Recent versions use a default configuration that is a combination of the relay_mail_from and relay_local_from features.
Address Rewriting and Relaying with Qmail
Qmail uses unique user and group IDs for run-time files, so a secure installation requires less prebuild configuration than sendmail. The features in qmail are designed less for legacy address syntax and polled connections than for consistent domain addressing and mailing lists.
Qmail also operates more closely than sendmail with domain name servers when relaying mail, so if a sendmail program writes an address that a name server does not recognize, a qmail program in its default configuration provides stricter relaying than sendmail. Qmail is more likely to refuse messages from unrecognized domains unless configured specifically to relay them.
However, defining a default hostname requires only that the name be listed in the ~qmail/control/defaulthost file, so qmail is easier to reconfigure for Internet leaf sites that do not have to worry about relaying messages for other domains.
Using Addresses in SMTP Transfers
When using SMTP as the transport protocol between hosts, the server on the receiving host must be able to resolve at least the originator address. SMTP transaction initiation requires that these addresses be resolved before the message itself is sent. The RCPT TO: argument (the envelope address portion of the To: field) is enclosed in angle brackets. This field contains a domain-addressable route to one or more addressee mailbox, and may specify additional routing information. RFC 821 also requires that the return-path argument of the MAIL FROM: argument be an envelope address.
RFC 821 provides a detailed listing of the possible responses at each phase of the dialog, including standard error messages. A relay program may also define additional site-specific error messages.
Internet Standard Date Format
Although SMTP mail relays do not strictly require a Date: field for message delivery, RFC 822 does. The originating host's relay adds a Date: or Resent-Date: header line by default if it is not already present, and messages that do not contain at least an RFC standard date from an e-mail client run the risk of being misidentified as spam.
Formatting and interpreting RFC 822 dates can be as challenging as formatting and interpreting e-mail addresses. A typical RFC 822 date (the output of the GNU date -R command, in this case) has the format Wed, 02 Mar 2001 12:34:25 +0400. However, RFC 822 makes the weekday optional, so an equally valid date string would be 02 Mar 2001 12:34:25 +0400. The time zone is required and can have, respectively, a local differential from Greenwich Mean Time, a North American Time Zone abbreviation that includes daylight savings time, a military zone letter, GMT, or UTC. The dates and times 02 Mar 2001 12:34:25 -0400, 02 Mar 2001 12:34:25 EDT, and 02 Mar 2001 12:34:25 D are equivalent.
Example 4(a), however, is the Perl regular expression to parse an RFC 822 date that does not include the weekday. To parse a date that contains the weekday, the regexp in Example 4(b) is slightly different.
Different e-mail clients transmit the date differently. For example, Microsoft Windows Messaging does not include the time zone. E-mail client software that is not Y2K compliant may transmit a two-digit year, and some clients may not use leading zeros on single-digit dates and times.
Host Relaying and Forwarding
Because RFC 821 does not allow host name aliasing nor does it specify authentication procedures a domain name server must provide routing information to the destination host.
The variables in sendmail that control relaying mail include LOCAL_RELAY and MAIL_HUB. The stickyhost option can bypass a LOCAL_RELAY if desired. Most mail destined for the Internet, however, should be routed to a defined SMART_HOST. Sendmail's relay definitions include:
- The host defined by the LOCAL_RELAY macro sends mail to unqualified mailboxes.
- The MAIL_HUB macro's defined host routes mail qualified with the name of the local host.
- SMART_HOST routes mail with domain addresses qualified by other hosts.
Qmail's relaying configurations are less well documented than sendmail's; fortunately, qmail uses a simpler configuration that makes routing decisions based on the hosts in these configuration files:
It is also easier to define a null client to forward all mail using the route listed in the ~qmail/control/smtproutes file.
If the ~qmail/control/rcpthosts file does not exist, then qmail acts as an open relay. In practice, however, this increases the likelihood that an improperly addressed message will be denied, relaying by a host further up the line.
Forwarding for Individual Users
Sendmail's ForwardPath option lets users forward mail to addresses listed in a ~/.forward file. The ForwardPath option is synonymous with the -o command-line option. Although long a standard option in sendmail, it is considered a security risk as are many other per-user forwarding techniques not the least of which is that a series of forwarders will create a loop. That is, the addresses in two or more .forward files will refer to each other.
Qmail provides loop checking by using a Delivered-To: header field. Qmail provides site-wide default forwarding rules as well as aliasing, and per-user forwarding rules using a ~/.qmail configuration file. Like sendmail, qmail can use host-based filtering software, but the configuration can be performed on a per-user basis.
In many host installations, particularly those that provide virtual domains, procmail is used as a local forwarding agent. Sendmail provides a compile-time option to use procmail as a local delivery agent, and qmail users can specify procmail as a first-stage message filter.
Different software packages address the complicated process of e-mail envelope addressing, and each provides its own configuration scheme to provide more flexible domain addressing to machines and users that use a wide variety of connection methods and e-mail software. Mail software configuration and maintenance, even for administrators of single-user machines, has become more complicated as the Internet has grown in size and complexity, and the increased number of e-mail relay software packages has made it even more important that administrators choose software that suits the needs of their sites and individual users.
Allman, Eric. Sendmail Installation and Operation Guide. Version 8.317.4.56. Sendmail Inc.
Costales, Bryan and Eric Allman. Sendmail, Second Edition. O'Reilly & Associates, 1993.
Crocker, David H. RFC 822: Standard for the Format of ARPA Internet Text Messages.
Johnson, Chris. The Qmail Newbie's Guide to Forwarding and Relaying. http://www.palomine.net/qmail/relaying.html.
Postel, Jonathan B. RFC 821: Simple Mail Transfer Protocol.