| Back to Fetchmail Home Page | To Site Map | $Date: 2002/03/10 19:59:57 $ |
Before accepting responsibility for the popclient sources from Carl, I had investigated and used and tinkered with every other UNIX remote-mail forwarder I could find, including fetchpop1.9, PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc, popmail-1.6 and upop. My major goal was to get a header-rewrite feature like fetchmail's working so I wouldn't have reply problems anymore.
Despite having done a good bit of work on fetchpop1.9, when I found popclient I quickly concluded that it offered the solidest base for future development. I was convinced of this primarily by the presence of multiple-protocol support. The competition didn't do POP2/RPOP/APOP, and I was already having vague thoughts of maybe adding IMAP. (This would advance two other goals: learn IMAP and get comfortable writing TCP/IP client software.)
Until popclient 3.05 I was simply following out the implications of Carl's basic design. He already had daemon.c in the distribution, and I wanted daemon mode almost as badly as I wanted the header rewrite feature. The other things I added were bug fixes or minor extensions.
After 3.1, when I put in SMTP-forwarding support (more about this below) the nature of the project changed -- it became a carefully-thought-out attempt to render obsolete every other program in its class. The name change quickly followed.
This problem only becomes obvious when a reply is generated on a machine different from where the message was delivered. The two machines will have different local username spaces, potentially leading to misrouted mail.
Most MTAs (and sendmail in particular) do not canonicalize address headers in this way (violating RFC 1123). Fetchmail therefore has to do it. This is the first feature I added to the ancestral popclient.
I was able to improve matters significantly by reorganizing most of the program around the `query' data structure and eliminating a bunch of global context. This especially simplified the main sequence in fetchmail.c and was critical in enabling the daemon mode changes.
Once this worked, I rewrote the POP3 code to use the same organization. The POP2 code kept its own driver for a couple more releases, until I found sources of a POP2 server to test against (the breed seems to be nearly extinct).
The purpose of this reorganization, of course, is to trivialize the development of support for future protocols as much as possible. All mail-retrieval protocols have to have pretty similar logical design by the nature of the task. By abstracting out that common logic and its interface to the rest of the program, both the common and protocol-specific parts become easier to understand.
Furthermore, many kinds of new features can instantly be supported across all protocols by modifying the one driver module.
Why mess with all the complexity of configuring an MDA or setting up lock-and-append on a mailbox when port 25 is guaranteed to be there on any platform with TCP/IP support in the first place? Especially when this means retrieved mail is guaranteed to look like normal sender- initiated SMTP mail, which is really what we want anyway.
Clearly, the right thing to do was (1) hack SMTP forwarding support into the generic driver, (2) make it the default mode, and (3) eventually throw out all the other delivery modes.
I hesitated over step 3 for some time, fearing to upset long-time popclient users dependent on the alternate delivery mechanisms. In theory, they could immediately switch to .forward files or their non-sendmail equivalents to get the same effects. In practice the transition might have been messy.
But when I did it (see the NEWS note on the great options massacre) the benefits proved huge. The cruftiest parts of the driver code vanished. Configuration got radically simpler -- no more grovelling around for the system MDA and user's mailbox, no more worries about whether the underlying OS supports file locking.
Also, the only way to lose mail vanished. If you specified localfolder and the disk got full, your mail got lost. This can't happen with SMTP forwarding because your SMTP listener won't return OK unless the message can be spooled or processed.
Also, performance improved (though not so you'd notice it in a single run). Another not insignificant benefit of this change was that the manual page got a lot simpler.
Later, I had to bring --mda back in order to allow handling of some obscure situations involving dynamic SLIP. But I found a much simpler way to do it.
The moral? Don't hesitate to throw away superannuated features when you can do it without loss of effectiveness. I tanked a couple I'd added myself and have no regrets at all. As Saint-Exupery said, "Perfection [in design] is achieved not when there is nothing more to add, but rather when there is nothing more to take away." This program isn't perfect, but it's trying.
Anyone who's acquired the 0600 permissions needed to read your .fetchmailrc file will be able to run fetchmail as you anyway -- and if it's your password they're after, they'd be able to rip the necessary decoder out of the fetchmail code itself to get it.
All .fetchmailrc encryption would do is give a false sense of security to people who don't think very hard.
If one could thread the protocol code so that fetchmail didn't block on waiting for a protocol response, but rather switched to trying to process another host query, one might get an efficiency gain (close to constant loading at the single-host level).
Fortunately, I've only seldom seen a server that incurred significant wait time on an individual response. I judge the gain from this not worth the hideous complexity increase it would require in the code.
To handle this optimally, multiple fetchmails would have to associate a system-wide semaphore with each active pair of a remote user and host canonical address. A fetchmail would have to block until getting this semaphore at the start of a query, and release it at the end of a query.
This would be way too complicated to do just for an "it might be nice" feature. Instead, you can run a single root fetchmail polling for multiple users in either single-drop or multidrop mode.
The fundamental problem here is how an instance of fetchmail polling host foo can assert that it's doing so in a way visible to all other fetchmails. System V semaphores would be ideal for this purpose, but they're not portable.
I've thought about this a lot and roughed up several designs. All are complicated and fragile, with a bunch of the standard problems (what happens if a fetchmail aborts before clearing its semaphore, and how do we recover reliably?).
I'm just not satisfied that there's enough functional gain here to pay for the large increase in complexity that adding these semaphores would entail.
There are two important aspects of the features for handling multiple-drop aliases and mailing lists which future hackers should be careful to preserve.
The code in mxget.c is nasty, no two ways about it. But it's utterly necessary, there are a lot of MX pointers out there. It really ought to be a (documented!) entry point in the bind library.
Unfortunately this means that if a DNS error is permanent a message can be perpetually stuck in the server mailbox. We've had a couple bug reports of this kind due to subtle RFC822 parsing errors in the fetchmail code that resulted in impossible things getting passed to the DNS lookup routines.
Alternative ways to handle the problem: ignore DNS errors (treating them as a non-match on the mailserver domain), or forward messages with errors to fetchmail's invoking user in addition to any other recipients. These would fit an assumption that DNS lookup errors are likely to be permanent problems associated with an address.
Strings that must be subject to translation should be wrapped with GT_() or N_() -- the former in function arguments, the latter in static initializers and other non-function-argument contexts.
struct run,
struct query, or struct hostdata.
rcfile_y.y. Add the token to the grammar. Don't
forget the %token declaration.
rcfile_l.
options.c. Pick a new LA_
value. Hack the longoptions table to set up the
association. Hack the big switch statement to set the option.
Hack the `?' message to describe it.
def_opts near the top of
load_params in fetchmail.c.
fetchmail.c:dump_params.
FLAG_MERGE actions in fetchmail.c's optmerge()
function. For a global option, add an override at the end of
load_params; this will involve copying a "cmd_run." field to a
corresponding "run." field, see the existing code for models.
Before you implement an option, though, think hard. Is there any way to make fetchmail automatically detect the circumstances under which it should change its behavior? If so, don't write an option. Just do the check!
The POP3 UID feature described in RFC1725 to replace LAST is insufficient. The only problem it solves is tracking which messages have been read by this client -- and even that requires tricky, fragile implementation.
The underlying lesson is that maintaining accessible server-side `seen' state bits associated with Status headers is indispensible in a Unix/RFC822 mail server protocol. IMAP gets this right.
This is an advantage not to be despised! Because of it, this project has been interesting and fun -- no serious or persistent bugs, no long hours spent looking for subtle pathologies.
It's gratifying that fetchmail has become so popular. Until just before 1.9 I was designing strictly to my own taste. The multi-drop mailbox support and the new --limit option were the first features to go in that I didn't need myself.
By 1.9, four months after I started hacking on popclient and a month after the first fetchmail release, there were literally a hundred people on the fetchmail-friends contact list. That's pretty powerful motivation. And they were a good crowd, too, sending fixes and intelligent bug reports in volume. A user population like that is a gift from the gods, and this is my expression of gratitude.
The beta testers didn't know it at the time, but they were also the subjects of a sociological experiment. The results are described in my paper, The Cathedral And The Bazaar.
Other significant contributors to the code have included Dave Bodenstab (error.c code and --syslog), George Sipe (--monitor and --interface), Gordon Matzigkeit (netrc.c), Al Longyear (UIDL support), Chris Hanson (Kerberos V4 support), and Craig Metz (OPIE, IPv6, IPSEC).
| Back to Fetchmail Home Page | To Site Map | $Date: 2002/03/10 19:59:57 $ |