Issue Dovecot too many SNI ssl certificate config files Max open files

[kevinjansen]

Server operating system version

Centos 7

Plesk version and microupdate number

18.0.49

Hello Everyone,
We are having errors with our dovecot default config.

Jan 25 13:40:47 plesk34 postfix-local[57825]: 93BAD6620B: process '/usr/libexec/dovecot/dovecot-lda -d "$DELIVERED_TO" -a "${ORIGINAL_RECIPIENT:-DELIVERED_TO}"' stderr : doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/14-plesk-sni-mail.domain.tld.conf line 0: Couldn't open include file /etc/dovecot/conf.d/14-plesk-sni-mail.domain.tld.conf: Too many open files

Latest Plesk version
Centos 7

But after that config directory reaches more then 1024 configs, then the problem starts. So there is somewhere a default 1024 settings that needs to be override. Any suggestions?

What background information:
We add for every domain a mail.domain.tld, request a ssl certificate and bind that to the mail server. So customers can use mail.domain.tld with a valid certificate. We know that "domain.tld" works, but in nature people uses mail. or smtp. or imap. as mail services. This results that when we reach 512 domains (because each domain creates 2 config files) that Dovecot starts shouting out errors.

[root@plesk34 ~]# grep files /proc/$(pgrep -f 'dovecot -F')/limits
Max open files 65535 65535 files

show dovecot.service | grep -i limitno
LimitNOFILE=65535

I am unable to find where to increase more limits to exceed that 1024 file limit for Dovecot. Anyone knows the magic trick?

Kind regards,

Kevin

 

[kevinjansen]

I have removed some mail. certificates so the other customers don't have issues at this moment.

[root@plesk34 ~]# ls /etc/dovecot/conf.d/ | wc -l
1006

But I like to have it fixed/increased so I can create it again.

 

[Peter Debik]

You will need to increase the maximum allowed number of file handles on the operating system level. This includes at least two changes:

1) /etc/sysctl.conf
fs.file-max = <new higher number goes here>
Example: fs.file-max = 100000

2) /etc/security/limits.conf
dovecot soft nofile <new number goes here>
dovecot hard nofile <new number goes here>
Example:
dovecot soft nofile 100000
dovecot hard nofile 100000

Afterwards:
# systemctl --system daemon-reload

 

[kevinjansen]

Dear Peter,

Thanks for your fast response, but this is not the case. The servers open file limit is allready increased. But to be sure i have increased it again by a factor 10, and when the Dovecot conf reaches over 1024 configs it starts happening again. Even when all the limits on server side are for above that 1024.


cat /etc/sysctl.conf | grep file-max
fs.file-max=241013800

cat /etc/security/limits.conf | grep dovecot
dovecot soft nofile 500000
dovecot hard nofile 500000

I do also not think its a server limit, cause i am able to open more files with nginx. The limit is within Dovecot.

 

[ChristophRo]

what does this command show?

systemctl show dovecot.service | grep -i limitno

I would not be surprised if you are limited by systemd, because as very often with systemd, why bother with existing methods for such stuff, if systemd can reinvent the wheel and do the same things again and on top...just worse.

 

[kevinjansen]

Dear,

The outcome is:
[root@plesk34 conf.d]# systemctl show dovecot.service | grep -i limitno
LimitNOFILE=65535

This has been allready increased with the command systemctl edit dovecot.service
And ofcourse dovecot restarted after.

 

[kevinjansen]

The difference what i can see here is that it does not give me the limit back that i originally set it to. Thats much higher i have set it to: 241013800

 

[kevinjansen]

Also i may have found the magic number 1024:

[root@plesk34 ~]# ulimit -Sn

1024

[root@plesk34 ~]# ulimit -Hn

4096

But thats ofcourse for the user root.

 

[ChristophRo]

it should also show a value for LimitNOFILESoft that seems to be missing in your outputs.
Can you please also run this command for once:

cat /proc/`pgrep dovecot`/limits

 

[ChristophRo]

But thats ofcourse for the user root.

dovecot starts as root, thus it's likely it will retain these ulimit settings for the root user if not overwriten somewhere else

 

[kevinjansen]

[root@plesk34 security]# cat /proc/`pgrep dovecot`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 95643 95643 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 95643 95643 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

[root@plesk34 security]#

 

[kevinjansen]

Dear all,

What i have done so far:

systemctl edit dovecot.service
[Service]
LimitNOFILE=infinity

In /etc/security/limits.conf added
* soft nofile 65535
* hard nofile 65535

Now ulimit -n shows 65535

And:


[root@plesk34 metareg]# cat /proc/`pgrep dovecot`/limits


Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 95643 95643 processes
Max open files 1048576 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 95643 95643 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Rebooted the server.
But the problem remains when there are more then +-1024 configuration files in /etc/dovecot/conf.d/ its start trowing errors about the "Too many open files"

Is there something else i can check on Centos 7?

 

[Peter Debik]

Maybe it is a limit of the container (in case it is a virtualization)?

 

[Peter Debik]

All documentation is saying the same. When you have done all steps mentioned above, things should work. Just to make sure: have you remembered to run
# systemctl --system daemon-reload
after the changes that were applied to files like /etc/security/limits.conf? Did you also make sure that the changes to /etc/security/limits.conf are the last entries of that file or could it be possible that changes were made, but not at the end so that other entries could overwrite such changes in the same file?

 

[Peter Debik]

One of my docs says, a server reboot is required for the changes to security/limits.conf to apply. I don't think that is correct, but think that a daemon-reload should do the same. However, I was wanting to mention it in case you are looking for more options.

 

[kevinjansen]

Dear Peter,

I dont think our VM is limited. Its running on XCP-NG (Xenserver), and for example NGIX/Apache is allowed to load over 1500 vhost configs.
Only Dovecot is having issues with it.

I have done daemon-reload, and even rebooted the server to be sure. When i am root i can see the set value,
And even the command:
[root@plesk34 ~]# sudo -u dovecot bash -c 'ulimit -n'
65535

Shows more then the 1024. So it feels like there is something else in maybe a dovecot config. Or is maybe SELinux blocking something?

 

[Peter Debik]

Could you please look into /etc/systemd/system/multi-user.target.wants/dovecot.service what Limit is set there for the number of open files? If it is high, no further action is required.

I have this new idea:

Jan 25 13:40:47 plesk34 postfix-local[57825]: 93BAD6620B: process '/usr/libexec/dovecot/dovecot-lda -d "$DELIVERED_TO" -a "${ORIGINAL_RECIPIENT:-DELIVERED_TO}"' stderr : doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/14-plesk-sni-mail.domain.tld.conf line 0: Couldn't open include file /etc/dovecot/conf.d/14-plesk-sni-mail.domain.tld.conf: Too many open files

It might not be caused by a Dovecot issue, because Dovecot is not reporting the problem, but Postfix is. If Dovecot is called from with in a Postfix process, and that Postfix process is limited to a number of 1024 open files, that might be the cause why also Dovecot called from within Postfix is limited to that number of open files. Just a theory though.

Here is an idea how to narrow this thought down:

Try to run
# doveconf | less
on a situation where there are >1024 SNI configuration files. Does it return the Dovecot variables (lots of stuff in the result, you'll see)?

Then I found this sequence that reponds with "1" if doveconf finds an error and "0" if it does not.
# doveconf 1>/dev/null && echo $?
In a situation when there are more than 1024 files, does this still deliver a "0"? And is the output of "doveconf" without parameters still an expected verbose output? If so, then there is no problem with doveconf and likely the problem is with Postfix. However, if this also fails to open the files and returns a "too many open files" message, you know for sure that the problem is with Dovecot.

 

[kevinjansen]

Hi,

Thanks Peter for thinking with me.
Doveconf echo's 0 with more then 1024 configs. So i think indeed its related to postfix.

But also this command returns a higher value:

sudo -u postfix bash -c 'ulimit -n'
65535

I am able to login in webmail, so dovecot seems to work fine. But now how can i fix the postfix error.
I have edited:


systemctl edit postfix.service
[Service]
LimitNOFILE=infinity

[root@plesk34 conf.d]# sudo -u postfix bash -c 'ulimit -n'
65535

Did a server reboot. But the mails are not send/delivered in the mailboxes. So i think we are a step closer that it is related to Postfix.

 

[Peter Debik]

My knowledge horizon ends where I know that in Postfix this is related to a parameter FD_SETSIZE, which in some versions must be compiled into Postfix and is else fixed to 1024, while in more recent versions it should be possible to set it (somehow). But I do not know how to do that, never before came across that necessity.

So here my forum support ends ;-) and I can only adivse to either wait on someone here that knows more about that or to submit a ticket to Plesk support. If you choose to submit a ticket, please include a link to this thread so that the supporter can see that we already checked "360015345719: Dovecot configuration failed on a Plesk server: Too many open files" and it has not helped.

 

[kevinjansen]

Hi Peter,

It surely is postfix, when i switch over from postfix to qmail, the error is gone, and everything works again.
But according to Plesk postfix is the only option for SNI email certificates. So i need to increase the open files by Postfix. And that is indeed the FD_SETSIZE

What i have done so far is edit:
/etc/sysctl.conf
fs.file-max = 65535
sysctl -p

systemctl restart postfix
But that does not work. So i think i need to rebuild postfix, but then it requires a new rebuild everytime there is a update.

Maybe some other people have suggestions how to increase FD_SETSIZE on Postfix