Trying to send many emails, confused about settings

[AdrianC]

Hello.

I have a dedicated server where I am trying to send many emails per day, around one email every 2 seconds by a php script. That is around 43000 emails a day. Godaddy set a larger limit so I don't reach the daily send limit.

The problem is that emails get stuck in queue, I increased concurrencyremote value of qmail to 240 because some say 255 is a limit, I am afraid that setting it over 255 will drop it to a default 20 value. Now I find others saying that there is a compile-time limit of 120 (?!)

The emails I see blocked in plesk queue are normal emails that I sent, just sometimes some spam, but mostly my sent emails.

1. Is it possible that qmail only accepts concurrencyremote value of up to 120 and if I set it to 240 it dropped back to default 20?
2. Is there a command to find the current values/config used by qmail regardless if I set correct values or not? Something to show me current concurrencyremote used.
3. How to find maximum concurrencyremote accepted by my qmail/Linux version?
4. Any other values to increase to avoid emails being blocked in queue? Some mentioned that is needed to increase "maximum opened files" on Linux.
5. Any other debugging tips?

Thank you.

 

[GeertL]

The problem is that emails get stuck in queue, I increased concurrencyremote value of qmail to 240 because some say 255 is a limit, I am afraid that setting it over 255 will drop it to a default 20 value. Now I find others saying that there is a compile-time limit of 120 (?!)

1. Is it possible that qmail only accepts concurrencyremote value of up to 120 and if I set it to 240 it dropped back to default 20?
2. Is there a command to find the current values/config used by qmail regardless if I set correct values or not? Something to show me current concurrencyremote used.
3. How to find maximum concurrencyremote accepted by my qmail/Linux version?

The max setting on Plesk should be 1.000 for concurrencyremote. Bigger values will be capped to 1.000 - but you can check what setting is used by checking the qmail logfile - on Ubuntu thats in /var/log/mail.info - just check the log for the current string:

Nov  8 02:51:39 server qmail: xx.xx status: local 1/100 remote 0/1000

This will show you which setting is used for concurrencyremote and concurrencylocal. In this case I had the concurrenctyremote set to 1200 - and you can see that qmail capped it to 1000 automatically. So just set a high number when you test - look where qmail will cap the setting and then you know the max values.

Remember you will need to restart qmail for the changed settings to be used.

Hope this will help you

 

[AdrianC]

Hello. That should help but the mail.info is not on that location, I did a search and no results returned, I searched by:

find -name mail.info

Do you know where these parameters/logs can be found on Fedora Core systems ?

 

[GeertL]

Do you know where these parameters/logs can be found on Fedora Core systems ?

I would guess it also could be /var/log/maillog or in /var/log/qmail/<something> - otherwise you can always send a mail to a known adress on the server and in /var/log/ just run a:

grep iEr "your\@emailadress\.com" *

Remember the \ before @ and any dots (.) in the adress. This will search all files in /var/log and subdirs for the e-mail adress - this should point you to the exact logfile... I've never used Fedora so not sure if the PSA implementation of logs is different there.

 

[AdrianC]

I found the file, on plesk (8.2 I think) with Fedora Core it is:

/usr/local/psa/var/log/maillog

And while setting concurrencyremote to 1200 I found it 1000 in the above log, so 1000 is max value. Which I think is more than enough.

I think problem is somewhere else.
I sometimes send 40 000 emails a day, and daily I have around 120 new member registrations, when I sent so many emails I see around 30 new users unactivated (of around 120) which means they didn't get the activation code by email. User activation code is not in email queue eider.

My guess is that GoDaddy's mail server (k2smtpout.secureserver.net) which is set in my server configuration files cannot handle so many emails one after another and some are not delivered.

The "unactivated new users" issues always appear when I sent many emails that day.

I send an email around every 2-4 seconds (not 40 000 at once).
I am afraid to send emails more often or more emails a day because I think they just go nowhere.
What if only 5 000 of the 40 000 emails were delivered?!

Any ideas/tips? Maybe I can just change k2smtpout.secureserver.net to another hosting/email service provider?

 

[IgorG]

Here is one solution for increasing Qmail performance - putting qmail’s queue on a ramdisk.

First, add following line into your /etc/fstab:

none /mailqueue tmpfs defaults,size=100m,nr_inodes=999k,mode=775 0 0

This will make a 100MB ramdisk on /mailqueue. Now, just symlink /var/qmail/mqueue to /mailqueue and move your e-mail over:

# mount /mailqueue
# chmod 750 /mailqueue
# chown qmailq:qmail /mailqueue
# mv /var/qmail/mqueue /var/qmail/mqueue-old
# ln -s /mailqueue /var/qmail/mqueue
# rsync -av /var/qmail/mqueue-old /mailqueue

This will significantly cut the iowait for Qmail server during heavy e-mail periods.

 

[AdrianC]

I will try that ramdisk thing, but before that I have some more info that might help, can you tell me if any of this makes a difference? With 1000 concurrencyremote I tried to send 1000 emails at once (with a php loop script), sender and receiver was the same, around 500 went into queue, they were sent all in around 10 hours.
Here are the errors, what does it mean?
I attached larger part of log as txt file.
Thank you.

Nov 10 00:23:15 ffiles qmail: 1257812595.970116 status: local 0/10 remote 511/1000
Nov 10 00:23:15 ffiles qmail: 1257812595.970958 delivery 1058: deferral: qmail-spawn_unable_to_create_pipe._(#4.3.0)/
...
Nov 10 00:23:16 ffiles qmail: 1257812596.016333 status: local 0/10 remote 511/1000
Nov 10 00:23:16 ffiles qmail: 1257812596.017609 delivery 1060: deferral: qmail-spawn_unable_to_create_pipe._(#4.3.0)/
...
Nov 10 01:03:41 ffiles qmail: 1257815021.524909 status: local 0/10 remote 28/1000
Nov 10 01:03:41 ffiles qmail: 1257815021.565756 delivery 1586: deferral: Connected_to_64.202.189.86_but_connection_died._(#4.4.2)/
Nov 10 01:03:41 ffiles qmail: 1257815021.565854 status: local 0/10 remote 27/1000
Nov 10 01:03:41 ffiles qmail: 1257815021.701730 delivery 1588: deferral: Connected_to_64.202.189.86_but_connection_died._(#4.4.2)/
Nov 10 01:03:41 ffiles qmail: 1257815021.701830 status: local 0/10 remote 26/1000
Nov 10 01:03:42 ffiles qmail: 1257815022.017855 delivery 1592: deferral: Connected_to_64.202.189.86_but_connection_died._(#4.4.2)/

 

[IgorG]

Adrian,

Seems this error caused by OS limit, I think it has relation with the max.file descriptor that the system can handle. Try to increase it seriously in /proc/sys/fs/file-max
I hope it will help.

 

[AdrianC]

Current value is of /proc/sys/fs/file-max is 204287
- By what amount should I increase it? Should it be increase by a larger number than emails in queue (5 000 - 10 000) ?
- I read here (link) that "qmail daemons could crash if you set the run-time concurrency higher than [the per-process limit]." I am not sure if this refers to file descriptors but I think I read somewhere that I should increase another parameter once with this file descriptors?!
- That link talks about bad things and crashes if file descriptors are set incorrectly so I dont know what to enter in /proc/sys/fs/file-max .

I hope you have some more tips/answers ^ IgorG, thanks for your patience.

 

[IgorG]

Adrian,

I can not say how much it should be exactly. I can only advise to pick up this value experimentally increasing it gradually and observing of result.

 

[AdrianC]

One more thing.
I noticed the "unable to create pipe" error just occured that one time when I forced it to send 1000 messages at once (1-2 seconds) but I searched the log and I never seen that error in normal usage even if I sent 30 000 e-mails a day, the "but_connection_died" error is there however, so based on this do you still think ramdisk thing or increasing max-file would help?
Thank you.

 

[IgorG]

Adrian,

RAM disk is more preferable way, in my opinion. It considerably will improve I/O disk operations. The increasing of max_file can lead to unpredictable results.