QMail : Slow, added -Rt0 in xine, but another problem

[Swakoo]

Some updates

I have just changed all the domains in PLESK to REJECT mail instead of "bouncing with message"

 

[jamesyeeoc]

'Reject' is good, it will definitely cut down on the number of bounce/doublebounce messages stuck in queue.

Have you seen any of my other posts (or other's posts) regarding qmail control files?

There are additional settings for Qmail which can also help in keeping the queue from getting overloaded.

Edit: Sorry, I didn't mean to sound snapish, just been a hell of a week so far. What I meant was to please do a search to find the posts regarding qmail control files...

 

[Swakoo]

Originally posted by jamesyeeoc
'Reject' is good, it will definitely cut down on the number of bounce/doublebounce messages stuck in queue.

Have you seen any of my other posts (or other's posts) regarding qmail control files?

There are additional settings for Qmail which can also help in keeping the queue from getting overloaded.

Your other posts... nope didn't.

Does it take a while for "reject" totake effect?

One thing for sure, with '-Rt0' my queue fills up with my "failure notice" faster. I just removed it, but my remote queue is still steadily building up.. mostly failure notices...

before this... my remote queue always wavered about 30-50.. never hit so high.. i had disabled it this morning at 830am.. not its almost 6pm.. and it hit 880 in queue.

Where are the failure notices supposed to "go to" after a while being stagnant in queue?

reading my very long post.. what do you reckon?

-

 

[jamesyeeoc]

Where are the failure notices supposed to "go to" after a while being stagnant in queue?

After reaching the 'queuelifetime' value, they should be dropped from the queue.

Does it take a while for "reject" totake effect?

Normally immediate, did you restart qmail or the server?

most arguements work for qmHandle

Strange that only 'most' work, as to the errors on using the qmHandle -R and having to do the -S then -R again, that is weird, never seen that happen before. Queue corruption maybe?

I added one more space before "-Rt0"

Single or double spaces should not make any difference at all. Your server is possessed by the devil!

MAILER-DAEMON@serverhostname, someare even send to "postmaster@serverhostname

This may get back to the other /var/qmail/control files which are not created by default, but you can create them manually. Do a 'man qmail-send' and 'man qmail-control' for details on these files (too much to post here). Ones to specifically note:

me
concurrencylocal
concurrencyremote
queuelifetime
doublebounceto (make it a single line containing just a # sign)
bouncefrom
bouncehost

 

[Swakoo]

is there a way to control queuetimelife?
because from the way i look.. the queue has been steadily increasing with no signs of decrease.. just over 1 weekend.. its hitting 11000 mails already

Also, just realise my emails going out (gmail, yahoo etc..) again acting up.. going to check what you suggested earlier...

urm with regards to the qmail-control config.. its like configuring the sendmail.mc right?

I'll go read up on it furthur....

whew...

Updates: Mail wasn't relayed out. I removed the 11k mail in queues based on "failure notice" and suddenly the mails are off...

does a long mail queue list block othe rmails?

I can't find queuetimelife on the server... i was relying on this i found on the net..

http://www.cyber-sentry.com/index.php?id=127
It mentions using "/var/qmail/control/queuelifetime" but no such file found... am I looking at something too old? cos that's the only next nearest thing I found..

I couldn't monitor the log in time as I was doing the removal of queue (cos don want current mail stuck too long), checking of log at the same time.. test mail too.. silly me.. sheer lack of experience...

now I just left the -Rt0 back there again, and will monitor once again... this time will do slowly to monitor the problem..

but i see the log.. nothing much also... nothing queer...

one school of thought: could it be spammer using my server to send? I read this from http://forum.plesk.com/showthread.php?s=&threadid=16774&highlight=queue+time, it talks about restricting the relay by configuring to 127.0.0.1/32

i trying it.. but don really understand what it means... any idea?


lastly (for now), most of the failure notices are to postmaster@hostname of server.. you able to point me to a quick link as to how should i change to a valid email address.. wanna try catching all the mail so i can monitor them for the time being... cos hostname is not a valid domain name

 

[jamesyeeoc]

me
concurrencylocal
concurrencyremote
queuelifetime
doublebounceto (make it a single line containing just a # sign)
bouncefrom
bouncehost

These are files which are located in /var/qmail/control directory. Not all of them are created by default, so you may have to create some of them. This is just a sampling of the possible control files, not a comprehensive list...

The 'queuelifetime' file (if not present) can be created. You would put a single line in it which consists of the numerical value corresponding to the number of seconds messages should be able to remain in the queue.

Qmail defaults to 604800 seconds if the 'queuelifetime' file does not exist.

Remember, anytime you change any of the /var/qmail/control files you will need to restart Qmail.

For more information, you can look at the man pages:

man qmail-control
man qmail-send
man qmail-smtpd

or read up at
http://www.qmail.org
http://www.qmailrocks.org

and of course, there is always Google...

 

[Swakoo]

i added some updates after you post...

and more updates here:

Questions...

Under Plesk->Server-Mail->Preference,

should Relay be put to closed?

And for Names for POP3/IMAP account, what s the difference between using full and short?

Will carry on the battle against the possessed server (or maybe its just me) tomorrow

thanks james!

 

[jamesyeeoc]

Originally posted by Swakoo
i added some updates after you post...

most of the failure notices are to postmaster@hostname of server.. you able to point me to a quick link as to how should i change to a valid email address.. wanna try catching all the mail so i can monitor them for the time being... cos hostname is not a valid domain name

What is the 'hostname of server'? If you SSH into the server and type the command 'hostname', does it show you the 'hostname of server' as it is in the emails? Just wondering how the hostname is not a valid domain name, or do you mean that that domain is just not created in Plesk, but the actual 'hostname of server' is a valid domain on the internet?

If this is the case, then it may be possible to create an alias for 'postmaster@'hostname of server' and point it to a valid email address such as monitorspam@yourdomain.com
(you would have to create this mail account in your Plesk)
Then edit /etc/aliases to include a line:
postmaster: monitorspam@yourdomain.com

Then edit /var/qmail/control/locals and add the 'hostname of server' on a separate line. Normally this file only contains 'localhost'

Then edit /var/qmail/control/rcpthosts and add the 'hostname of server'

Then restart qmail using the control panel, or from the commandline:
/etc/init.d/qmail restart

Then send a test message to 'postmaster@'hostname of server' and it should then end up in the mailbox for 'monitorspam@yourdomain.com'

does a long mail queue list block other mails?

It causes delays things since it takes time to process through 11000 messages

'talks about restricting the relay by configuring to 127.0.0.1/32'

Yes, you should set this, otherwise it may be possible for someone to spoof an IP as though they were a 'local' address.

and more updates here:

Questions...

Under Plesk->Server-Mail->Preference,

should Relay be put to closed?

You should set it for 'Authorization required' and checkmark 'SMTP'

And for Names for POP3/IMAP account, what s the difference between using full and short?

If you select FULL only, then your clients will be forced to use their full email address (someuser@somedomain.com) as the username in their mail clients. If you set it for short or Full then they can use either 'someuser' or 'someuser@somedomain.com'.

Will carry on the battle against the possessed server (or maybe its just me) tomorrow

thanks james!

Other than creating the queuelifetime file, you should also create/edit the concurrencyremote and the doublebounceto file.

 

[Swakoo]

thats long piece of info for me to digest... hang on..

meanwhile with regards to seting 127.0.0.1/32

Original is 127.0.0.0/8

- is it .0 or 1 are the end?
- if i add the /32 option, do i remove the the /8

what does this really do Don understand...

also an upadte.. my queue is about 2k plus..
i tried sending my mail to my 3 email accounts (local isp, my own domain email account, gmail)

all 3 didn't receive it yet..
I check the 2 logs
/usr/local/psa/var/log/smtp_pendings.log
/usr/local/psa/var/log/maillog

but can't find any trace of the message..

I did a grep -1R "sayitreachyou" *
thats the subject i put....
what does -1R with a * at the back means? R is recursive, * is all files I suppose? Then 1? doesn't say in the --help

Anyway I found it in /var/qmail/queue/mess/21/4671689

How come its still there...?
There are many folders within /var/qmail/queue ... if it is in mess.. meaning...? Tried doing a search (still) on the different folder purpose.. i may be searching wrong.. any pointers?
The mail in a mess? hahahaha
but anyway, outgoing mail now is stuck there... :S

What a headache....

thanks

 

[jamesyeeoc]

Originally posted by Swakoo
thats long piece of info for me to digest... hang on..

meanwhile with regards to seting 127.0.0.1/32

Original is 127.0.0.0/8

- is it .0 or 1 are the end?
- if i add the /32 option, do i remove the the /8

You would replace the entire 127.0.0.0/8 with 127.0.0.1/32

This has to do with IP subnets.
127.0.0.0/1 = 0.0.0.1 to 127.255.255.254
127.0.0.0/2 = 64.0.0.1 to 127.255.255.254
127.0.0.0/4 = 112.0.0.1 to 127.255.255.254
127.0.0.0/8 = 127.0.0.1 to 127.255.255.254
127.0.0.0/16 = 127.0.0.1 to 127.0.255.254
127.0.0.0/24 = 127.0.0.1 to 127.0.0.254
127.0.0.1/32 = 127.0.0.1 only.
As you can see, you do not want to have less than /32 or you will be allowing lots of additional addresses.


what does this really do Don understand...

also an upadte.. my queue is about 2k plus..
i tried sending my mail to my 3 email accounts (local isp, my own domain email account, gmail)

The bigger the queue, the bigger the delay for all messages in the queue.

all 3 didn't receive it yet..
I check the 2 logs
/usr/local/psa/var/log/smtp_pendings.log
/usr/local/psa/var/log/maillog

but can't find any trace of the message..

how can i search if the message is still in the server and not send?

i am currently doing a "grep sayitreachyou" in /var/qmail..

You should be searching in the /var/qmail/queue directories, this is where the queued up mail is kept while waiting for delivery (either local or remote delivery).

Did you ever adjust any of the values in the control files? Such as concurrencyremote (default=20, try 250), and setup doublebounceto file? And set value in queuelifetime to a lower number (default is 604800 seconds = 1 week) Try 86400 (1 day)

thats the subject i put....

by the way, i have not receive the mail yet...
I am still concerned that your Qmail installation is corrupted, not only from the -Rt0, but also qmHandle not 'fully' working. You really may want to consider forcing a reinstall of the qmail related RPM files.
thanks

 

[Swakoo]

hi james..

my last post was updated during the time you replied... i did a search and i found my file.. can you take a look at my previous post? thanks!


ooh i can't put 127.0.0.0/32 cos .0 is a broadcast address yah? so putting /32 is like putting 255.255.255.255 as my subnet?
ok i'm on to it..
Why did they put 127.0.0.0/8 in the first place??

But by putting the ip in whitelist.. it means mails from local host (127.0.0.1) will be allowed, but mails elsewhere (127.0.0.2 for example) will be disallowed?
Or do i need to specify in blacklist?


alot of thing syou have suggested i havent implement.. cos this is just one of the many task i am doing.. but nonetheless high on my piority

i'm setting that, and the queuetimelife thing. so its ok if i create the file myself yah
I set it to 2 days
now going to monitor again

 

[jamesyeeoc]

Originally posted by Swakoo

I did a grep -1R "sayitreachyou" *
thats the subject i put....
what does -1R with a * at the back means? R is recursive, * is all files I suppose? Then 1? doesn't say in the --help

I suppose you found that command on some forum and saw a lowercase L 'l' and thought it was a number one '1'.

From the man pages for 'grep':
-l, --files-with-matches
Suppress normal output; instead print the name of each input
file from which output would normally have been printed. The
scanning will stop on the first match.
-R, -r, --recursive
Read all files under each directory, recursively; this is equiv-
alent to the -d recurse option.

And the trailing * is telling it to search all files.

Anyway I found it in /var/qmail/queue/mess/21/4671689

The reason it listed the path/filename is due to the lowercase L option above.

How come its still there...?

The reason it is still there is because it has not been successfully delivered to the remote server (or local mailbox) yet.

There are many folders within /var/qmail/queue ... if it is in mess.. meaning...?

'mess' stands for 'messages'. The directory structure is how qmail organizes and keeps track of the messages. Not something you have to be concerned with.

Tried doing a search (still) on the different folder purpose.. i may be searching wrong.. any pointers?

You can spend countless hours scouring the www.qmail.org site and www.qmailrocks.org and using google. Search terms? qmail +queue +path +structure +directory and about a hundred different combinations. I am not that curious.

The mail in a mess? hahahaha
but anyway, outgoing mail now is stuck there... :S

What a headache....

thanks

Nothing to thank me for. All I'm doing is answering questions

 

[Swakoo]

I set the queuelifetime to 1 day yesterday..

but just realise my local and remote file quite is in the hundreds... and my load in top is almost hitting 7...

do I need to point the qmail to the file i created, or it will automatically findit?

i stopped the qmail for a while and remove the queuelifetime...

/var/qmail/control/queuelifetime

 

[Swakoo]

more updates...

i did a tail -f on the mail log, keep seeing
"1126235718.064850 warning: trouble opening remote/16/4670925; will try again later" kind of messages for my valid users...

what does it mean?


I read http://www.qmailrocks.org/maintain.php

It says queued message should not interfere with message being sent outm if the address is valid. But so far whenmy queue becomes long, my mail sending seems retarded.. .hmmm

also.. it recommends i delete reject mails rather than bounce/reject them... but plesk don offer such an option. I set to reject.. but seems like more spam coming in ... or is it due to the 127.0.0.1/32? but can't be tooo :S

how do i do it?


More updates

I checked out some documentation..
so I should set concourrencyremote to set how many times it will try before dropping the mail..

so is it the same, i just set the file (and those you mentioned) in /var/qmail/control and restart qmail. it will automatically know how to find?

 

[jamesyeeoc]

Sorry I haven't been around for a couple days.

On Aug 22, I asked if you had kept an eye on your mailllog. If you had these errors back then, I wish you would have posted it then.

Some possible causes for that error (there may be others):

Disk drive corruption: use fsck or e2fsck to check your partitions.

DNS problems: Run a dnsreport.com on your domain and fix any warning/errors
Also what is the contents of your /etc/resolv.conf

Ownership/permissions: check both the ownership and permissions on the /var/qmail/queue directory and it's contents:

drwxr-x--- 11 qmailq qmail 4096 Jul 8 06:13 queue

#ll /var/qmail/queue
drwx------ 2 qmails qmail 4096 Sep 8 04:09 bounce
drwx------ 25 qmails qmail 4096 Jul 8 06:13 info
drwx------ 25 qmailq qmail 4096 Jul 8 06:13 intd
drwx------ 25 qmails qmail 4096 Jul 8 06:13 local
drwxr-x--- 2 qmailq qmail 4096 Jul 8 06:13 lock
drwxr-x--- 25 qmailq qmail 4096 Jul 8 06:13 mess
drwx------ 2 qmailq qmail 4096 Sep 8 18:17 pid
drwx------ 25 qmails qmail 4096 Jul 8 06:13 remote
drwxr-x--- 25 qmailq qmail 4096 Jul 8 06:13 todo

ll remote
total 100
drwx------ 25 qmails qmail 4096 Jul 8 06:13 .
drwxr-x--- 11 qmailq qmail 4096 Jul 8 06:13 ..
drwx------ 2 qmails qmail 4096 Sep 7 11:29 0
drwx------ 2 qmails qmail 4096 Aug 13 12:01 1
drwx------ 2 qmails qmail 4096 Jul 23 04:10 10
drwx------ 2 qmails qmail 4096 Sep 7 04:09 11
drwx------ 2 qmails qmail 4096 Sep 8 04:09 12
drwx------ 2 qmails qmail 4096 Sep 8 18:17 13
drwx------ 2 qmails qmail 4096 Sep 6 14:27 14
drwx------ 2 qmails qmail 4096 Aug 31 04:09 15
drwx------ 2 qmails qmail 4096 Sep 3 04:05 16
drwx------ 2 qmails qmail 4096 Aug 27 09:38 17
drwx------ 2 qmails qmail 4096 Jul 8 18:28 18
drwx------ 2 qmails qmail 4096 Aug 29 04:09 19
drwx------ 2 qmails qmail 4096 Jul 5 04:05 2
drwx------ 2 qmails qmail 4096 Aug 28 08:42 20
drwx------ 2 qmails qmail 4096 Jul 2 04:10 21
drwx------ 2 qmails qmail 4096 Sep 5 23:32 22
drwx------ 2 qmails qmail 4096 Sep 7 02:05 3
drwx------ 2 qmails qmail 4096 Sep 3 04:05 4
drwx------ 2 qmails qmail 4096 Aug 26 04:08 5
drwx------ 2 qmails qmail 4096 Aug 23 01:37 6
drwx------ 2 qmails qmail 4096 Aug 18 04:08 7
drwx------ 2 qmails qmail 4096 Aug 12 03:01 8
drwx------ 2 qmails qmail 4096 Sep 8 06:18 9

I think you get the point.

Concurrencyremote: is how many maximum concurrent (simultaneous) connections to remote mail servers that your server's qmail should attempt to make.

Restarting Qmail, it *should* find it's control files, unless there is something else wrong with the Qmail setup or install.

 

[Swakoo]

Hi james, thanks for your reply once again!
Don worry if you're busy, no need to apologies

The last time when you tell me that, I don remember seeing it thats why.. then again.. things were moving so much i just caught a glimpse thats why.. and i was much more lost compare to now.. though its only 2 weeks.. hell lot of things i have learn along the way thanks to you and the guys here

I have went to dnsreport.. and here some of the warnnings and failures

Warnings
Nameservers on separate class C's My 2 nameservers are in the same datacenter, though 2 different phsycial machine. One on Plesk, the other is not.
SOA Serial Number SOA is not in the format of YYYYMMDDnn, but plesk does it that way... no way to overwrite huh/i]
Multiple MX records It says I only have one MX record. Which is true as I don have multiple email server.. right?
SPF record Never heard of this before.. I did a search and came across a few links... http://www.zytrax.com/books/dns/ch9/spf.html
Is it this?


Fail
Reverse DNS entries for MX records Ok I have to admit, when setting up, I came across this.. but I have no idea what it is.. then it works without.. so it seems I have a problem here.. is a Reverse DNS lookup in the same machine? or should it be a separate one? How to set one?

Acceptance of postmaster address the email address stated is not valid... thats why? Plesk don have option to edit.. unless I edit it myself..?


Permission is ok
fsck can't be run on mounted filesystem as I will risked severe filesystem damage, or so it says

 

[jamesyeeoc]

"Nameservers on separate class C's" - then don't worry about this one.

"SOA Serial Number SOA is not in the format of YYYYMMDDnn" - Yes, plesk insists on using the other method, don't worry about this one either.

"SPF record" - currently not widely enforced, more info found at: http://spf.pobox.com/ It means "Sender Policy Framework"

SPF fights return-path address forgery and makes it easier to identify spoofs.
Domain owners identify sending mail servers in DNS.
SMTP receivers verify the envelope sender address against this information, and can distinguish authentic messages from forgeries before any message data is transmitted.

This is basically a DNS TXT type record, the contents you would use the wizard at pobox.com, but this should not be the cause of your problems.

"Reverse DNS entries for MX records" - In Plesk, under Domains - DNS, do you have an A record which points mail.yourdomain.com to your IP address? If not, then add this. Now I'm wondering if you have any other errors or ommissions in your domain's DNS...

"Acceptance of postmaster address" - Create a mail alias 'postmaster@yourdomain.com' under your primary email account. I generally also make aliases for 'root' and 'abuse' as well. This is not critical, but you should do it in any case.

fsck vs e2fsck - I generally use e2fsck, and yes, you would want to do this without filesystems mounted. Take care of the other problems above first, you may not need to do any (e2)fsck'ing.

 

[Swakoo]

yup I have, mail.mydomain.com points to my mail server ip via the A record.

do I need to set the Reverse record then? Is it using PTR?

 

[jamesyeeoc]

Originally posted by Swakoo
yup I have, mail.mydomain.com points to my mail server ip via the A record.

do I need to set the Reverse record then? Is it using PTR?

You can set the PTR on your server, but unless the ISP who 'owns' the IP has delegated control of the IP to you (probably not), then you will need to have them get involved on this.

Most ISP's (or your DC or Hosting company) who 'own' the IPs normally don't do this by default, unless you are more than a 'small' client to them. Some will do the changes upon request, others won't do it at all for any reason.

To verify if the IP has been 'delegated' to your direct control, use www.arin.net www.ripe.net www.apnic.net (depending on your location) and do a lookup on your IP.

If you only see your ISP/DC/Host listed, and not you or your company, then you will have to have the reverse DNS (rDNS) changed by them.

If you do see your name or company name underneath the ISP/DC/Host, for your IP or IP block, then you would need to make the changes in your /var/named/run-root/var/xx.yy.zz.nn.in-addr.arpa file from a SSH root login shell prompt. As well as setting up Bind/Named to allow your upstream provider's nameservers to retrieve these records regularly...

Clear as mud, right?

 

[Swakoo]

yeah.. thanks

the ip is own by our datacenter equinix. we just got 14 ips off them.

so i just have to get them to do a reverse dns on the ip... ok.. going to contact them..

meanwhile. i set the queuelifetime to 86400 (1 day)

Also.. u asked what is the content of mine /etc/resolve.conf

its as follows

nameserver 127.0.0.1
search localdomain