ls of /etc/httpd/conf/plesk.conf.d/vhosts/ takes more than 50 seconds to complete

[Miguel_Tellería]

Dear all,

As the title says we are experiencing a severe performance degradation in our machine. It manifests specially on the following cases:
- Listing or accesing directories with many entries (even under low uptime load figures)

On week days, when our clients use our server the most we have further symtoms:
- Uptime/htop load raises to double figures (12, 13 on a quad-core machine)
- Entering the plesk panel as administrators results in long delay to show the start page. Sometimes you have to click on "start" to have the starting info appearing
- Changing a password (email, FTP, panel...) takes a long time
- Adding a subdomain takes a long time even leaving the process half-way done (as we reported in a separate post).

Here are some data about our server installation:

- We have 421 domains and subdomains (entries in /etc/httpd/conf/plesk.conf.d/vhosts/) + 93 in forwarding and 406 in webmails/roundcube
- The server is a quad-core machine with Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz and 16 Gb of RAM.
- I attach an output of /proc/meminfo and /proc/cpuinfo
- Kernel: 3.14.43 standard without grsecurity
- Distro CentOS 6.6 with SELinux disabled
- Plesk 12
- We use linux software RAID1 (mirror with mdadm) for the root partition.

Work previously done:
- The machine has been placed in "secure mode" and the disks filesystems were inspected:
- fsck did not reveal anything (done at both RAID and separated mirrored partitions)
- We tested speed on both disks by doing dd's to /dev/null and they performed ok
- SMART test passed also on both disks
- Atop complains about disk usage (only magnitude in red color)
- Stopping services (httpd, postfix, dovecot, mysql) returns the machine to "calm state" and directory listing works very fast.
- A similar machine with the same hardware, memory and Plesk and OS versions (although half the number of subscriptions) is not experiencing any problems.

Any hints or diagnosis clues will be welcome.

Regards,

Miguel Telleria

 

[Miguel_Tellería]

I have a little news for this:

Checking the kernel documentation for virtual memory tuning[*] I decided to change the value of

/proc/sys/vm/vfs_cache_pressure

from 100 to 50.

[*] http://lxr.free-electrons.com/source/Documentation/sysctl/vm.txt?v=3.14

This gives a higher priority to dentry_cache and inode_cache storage and now the directories are accessed inmediately at the expense of less caching other data (file, pages or other data).

I am watching the dentry line in slabtop to check that it doesn't increase too much.

With this change the ls works again inmediately and Plesks runs a little smoother. However there are still issues to solve.