Issue SMTP/Qmail stop working suddenly without any cause

[Stephen_Stephen]

Hi
I am using a server of centos 6.10 and plesk 12.0.18, from 19th Feb the mail (SMTP/POP3) services stop functioning (everything worked as usual by 19th Feb),
below are some logs showing when it happen:

/var/log/messages:
Feb 19 04:42:11 server56-116-51-32 wdcollect[1327]: SMTP server connection timeout.
Feb 19 04:42:11 server56-116-51-32 wdcollect[1327]: Connection to SMTP server has been closed.
Feb 19 04:42:12 server56-116-51-32 wdcollect[1327]: Failed to connect to all SMTP servers.
Feb 19 04:42:13 server56-116-51-32 wdcollect[1327]: Connection to server has been established.

In Plesk -> Tools&Settings -> Services Management it shows the SMTP (qmail) is stopped (screenshot), Clicking the run botton on the right does not turn it on at all. however in the console it says it's running (screenshot). I tried restart: [service qmail restart] then it says it restarts successfully. But the problem persists. Tried to send from gmail, it shows the failure (screenshot)

I then tried to see if port 25 is open and as expected:

# netstat -plntu | grep ':25'
tcp 0 0 :::25 :::* LIST

Here is log for qmail restart activities:
Feb 21 15:21:24 server56-116-51-32 qmail: 1708528884.287585 status: exiting
Feb 21 15:22:08 server56-116-51-32 qmail: 1708528928.520275 status: local 0/10 remote 0/20

Now after 2 days I still cannot find any useful info online can someone give me some hint or the cause (no one has changed anything on the server, plesk/server should have stopped upgrading either).

Thanks very much

 

[Peter Debik]

Has QMail logged anything to your syslog by which you could tell why it crashes? As far as I remember, qmail-send is something different. You'll want to look for qmail-smtp.

 

[Stephen_Stephen]

thanks after I run the command: ps ax | grep qmail
11135 ? S 0:00 qmail-send
10220 ? Zs 0:00 [qmail-smtpd] <defunct>
21951 pts/1 S+ 0:00 grep qmail

the qmail-smtpd is marked <defunct>

also where is the syslog? I look at the /var/log/maillog cannot find where/how the qmail fails

 

[Stephen_Stephen]

After some finding I think I can address the issue. The above log [qmail-smtpd] <defunct> provides a ppid, which is xinetd process. I then restart xinetd, Issue persisted, but at this point the /var/log/message give an important hint:

=========================================================================
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: bind failed (Address already in use (errno = 98)). service = ftp
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: Service ftp failed to start and is deactivated.
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: bind failed (Address already in use (errno = 98)). service = poppassd
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: Service poppassd failed to start and is deactivated.
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: bind failed (Address already in use (errno = 98)). service = smtp
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: Service smtp failed to start and is deactivated.
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: bind failed (Address already in use (errno = 98)). service = smtps
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: Service smtps failed to start and is deactivated.
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: bind failed (Address already in use (errno = 98)). service = submission
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: Service submission failed to start and is deactivated.
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Feb 22 10:20:30 server56-116-51-32 xinetd[2035]: Started working: 0 available services
=========================================================================

then look for this:

#ps ax | grep xinetd
=========================================================================
2035 ? Ss 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
15739 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
15740 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
...
15771 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
15772 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
15773 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
=========================================================================

the top most pid is the latest xinetd pid started, with the old xinetd pids still acting on the port 25, I triedto find out how this happened, however after looking at the recent /var/log/messages, it didn't give a clue. So I kill the obsolete pids above, then restart xinetd. Email is now back to normal.

If someone knows the cause of the multiple xinetd instance (or which log to look) please let me know in this thread

Thanks