Question : Limiting server load and speeding local delivery, when receiving very high volumes of email

I provide my users with email accounts specifically so that they can receive huge numbers of email from lists that they intentionally join.  Things have been running pretty well for 3 years, but lately I am seeing some very high spikes in cpu load that slow the server down, and of course causes sendmail to refuse connections until load is back under control.

This is junk mail, so suggestions can go light on protecting the arriving mail.  

I am running Sendmail on Redhat 7.3.  I provide OpenWebmail and run a cron job every hour that 'touches' each account in order to run email filters to move 'validation emails' out of the inbox.   This may be a load generator.  The filters also run whenever the member opens their account via OpenWebmail.  Each email account also has a cron job that deletes their inbox once every 3 hours.  These accounts accumulate and average of 10 meg of email in three hours with a high of 100meg in three hours.  

For 3 years Sendmail has run pretty well set in Queued Mode, attempting to deliver once an hour, allowing acceptance and delivery of mail when load is as high as 45 (high I know, but need to limit connection refusals, and normally we spike to no more than 15) I have included the entire M4 beneath my questions.  

Most M4 settings look as I set them myself 3 years or so ago, but I did have a consultant touch my system about 2 months ago, and am not sure if he tweaked this file.  Since I am a novice and set this up originally by trial and error, I need an experts eye to these settings.  The file seems smaller than I remember.  Also, I do not remember having both "mailer local" and "mailer procmail" entries before - and I thought there used to be a cancellation of safe mode in here.  The order of entries is different than I originally had them.

My primary concern is for Sendmail to never or very rarely refuse a connection.  My secondary concern is for the mail to be delivered locally as quickly as possible, since I suspect that an extremely full queue contributes to server load.  I realize that adding a second mx server to accept/hold mail when this server is busy will help, and I have one ready to add to the mix as soon as I am sure SendMail is setup as effectively as possible for this situation.

Specific Issues:
1.  YahooGroups bulk mail servers seem to have started delivering their mail in much larger batches this week from multiple servers at a time, and I think this is my primary problem with high load peaks.  Server load average is peaking up to 50 for as much as 10-15 minutes several times a day.  

Is there a way for me to 'throttle' email coming from their servers - either by limiting the number of connections they can make, or the number of emails they can drop before disconnecting?  This might require something done outside of Sendmail - ipTables perhaps?

2.  What Sendmail settings will minimize server load and speed up acceptance of email and local delivery of same?  

For example
a.  using Deferred versus, Queued, versus Interactive mode.  
b.  how to limite dns activity for inbound mail, etc...
c.  limiting 'protective' measures designed to preserve the mail if delivery fails, etc...

3.  Should I consider switching to PostFix for high volumen receipt and local delivery of email?  Assuming it will work with OpenWebmail.
4.  Should I be looking for a better, quicker way to filter the emails than with OpenWebmail, since this might be contributing to server load?

Thanks in advance for your advice.  Mike

=+++++++++++++++++++++++++++++++M4 below+++++++++++++++++++++++++++++
define(`confDEF_USER_ID',``8:12'')dnl
EXPOSED_USER(`root')dnl
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
define(`PROCMAIL_MAILER_PATH',`/usr/bin/procmail')dnl
FEATURE(local_procmail,`',`procmail -t -Y -a $h -d $u')dnl
define(`ALIAS_FILE', `/etc/aliases')dnl
define(`STATUS_FILE', `/etc/mail/statistics')dnl
define(`confUSERDB_SPEC', `/etc/mail/userdb.db')dnl
FEATURE(`mailertable',`hash -o /etc/mail/mailertable.db')dnl
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable.db')dnl
FEATURE(`genericstable')dnl
FEATURE(`domaintable')dnl
FEATURE(use_cw_file)dnl
FEATURE(use_ct_file)dnl
FEATURE(`access_db',`hash -o /etc/mail/access.db')dnl
FEATURE(`no_default_msa',`dnl')dnl
FEATURE(`blacklist_recipients')dnl
define(`confAUTO_REBUILD',`True')dnl
define(`confTO_CONNECT', `1m')dnl
dnl TRUST_AUTH_MECH(`DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl define(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confTO_QUEUEWARN',`7d')
define(`confTO_QUEUERETURN',`6d')
define(`confQUEUE_LA',45)
define(`confREFUSE_LA',45)
undefine(`confDELAY_LA')dnl
dnl This changes sendmail to only listen on the loopback device 127.0.0.1
dnl and not on any other network devices. Comment this out if you want
dnl to accept email over the network.
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')
dnl NOTE: binding both IPv4 and IPv6 daemon to the same port requires
dnl       a kernel patch
dnl DAEMON_OPTIONS(`port=smtp,Addr=::1, Name=MTA-v6, Family=inet6')
dnl We strongly recommend to comment this one out if you want to protect
dnl yourself from spam. However, the laptop and users on computers that do
dnl not have 24x7 DNS do need this.
define(`confMAX_DAEMON_CHILDREN',0)dnl
define(`confDOUBLE_BOUNCE_ADDRESS',`dblbounce')dnl
define(`confUSE_ERRORS_TO*',`True')dnl
define(`confSAFE_QUEUE*',`False')dnl
define(`confDF_BUFFER_SIZE',32768)dnl
define(`confMAX_QUEUE_RUN_SIZE',0)dnl
define(`confMIN_QUEUE_AGE',`1h')
define(`confLOG_LEVEL',4)dnl
define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn,restrictqrun')dnl
define(`confMAX_RCPTS_PER_MESSAGE',5)
MAILER(`local')dnl
MAILER(smtp)dnl
MAILER(procmail)dnl



Answer : Limiting server load and speeding local delivery, when receiving very high volumes of email

Well, Interactive mode is generally only used for testing.

If you don't actually use IPv6, then you could rebuild sendmail without IPv6 support - that'll lighten its load somewhat.

You could try larger DF and XF buffer sizes, like so:

dnl # Sendmail, Chap 24.9.25, Page 967
dnl # Specify the maximum size, in bytes, of buffered df* files (default is
dnl #   4096 bytes; 0 turns this off and is not recommended)
define(`confDF_BUFFER_SIZE',`16384')dnl

dnl # Sendmail, Chap 24.9.120, Page 1077
dnl # Specify the maximum size, in bytes, of buffered xf* files (default is
dnl #   4096 bytes; 0 turns this off and is not recommended)
define(`confXF_BUFFER_SIZE',`16384')dnl

You could disable canonicalization:

dnl # Sendmail, Chap 4.8.28, Page 192
dnl # Turn off E-Mail canonization (should be done by MSA, and this
dnl #     is a mail relay with no local users)
FEATURE(`nocanonify')dnl

You could turn off IDENT support:

dnl # Sendmail, Chap 24.9.109.13, Page 1065
dnl # Disable IDENT (RFC 1413) calls/turn off sending user-host verification
define(`confTO_IDENT',`0')dnl

If you don't need them you could turn off the DECnet and FAX relay supports:

undefine(`DECNET_RELAY')dnl
undefine(`FAX_RELAY')dnl

Finally, consider splitting your functions. You've got a lot heaped on this box, including local delivery and procmail. Consider installing a sendmail relay host - it would do nothing except receive and queue E-Mail, and then feed it to this box more slowly than everyone on the 'Net trying to deliver directly to this one. Also takes the SPAM load off of this one.
Random Solutions  
 
programming4us programming4us