Plesk Onyx ProFTPD very slow directory listings

[Chopper27]

A few weeks ago I changed my dedicated server to a new hardware. Plesk 11 on the old, Plesk Onyx on the new one. The migration of the customers hosting is almost done.

Since a few days, the directory listings are coming extremely slow when you work with FTP. Directories with about 10 thousand images need more than 10 seconds to be shown. A directory with 14 thousand files needs more than 25 seconds.
On the old server, the listings come under 1 second. So what could be wrong. My customer told me, that about one week ago, everything worked fine and quick.
Ubuntu 16.04.4 LTS‬
Plesk Onyx
Version 17.8.11 Update #11

Could it be, that an update of the last days could be the problem? Have you any hints for me?

Thank you
Horst

 

[trialotto]

@Chopper,

I am not sure (given the lack of sufficient information), but I am almost certain that there is some network issue: in the best case scenario, there is some (probably unintended) throttling (or something similar) ........ and in the worst case scenario, there are a lot of (network) packets in process or awaiting processing.

I would recommend to shut down some unnecessary services and/or to restrict (external) access to some degree, on both servers (!).

Anyway, ProFTP is not the bottleneck here, it will never be.

It is very (very) likely the case that there is some kind of resource overusage on the new server: this can have many root causes of the problem.

Just try to eliminate the most common pitfalls: slow migration processes, Apache using too much memory, CPU overusage etc.

That way, you will probably find one unique root cause of the problem encountered.

Hope the above helps a bit............. but you can always ask for further assistance.

Regards....

PS I would recommend to do the migration process in steps (read: on a subscription base) and finish the process during the nightly (read: quiet) hours. That would help a bit.

 

[Chopper27]

I thank you very much for trying to help me.

But there is really almost no load on the server. All migrations are finished.

top - 14:03:50 up 16 days, 22:04, 1 user, load average: 0,11, 0,21, 0,23
Tasks: 243 gesamt, 1 laufend, 242 schlafend, 0 gestoppt, 0 Zombie
%CPU(s): 0,1 be, 0,1 sy, 0,0 ni, 99,8 un, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Spch : 8133980 gesamt, 1747460 frei, 1614796 belegt, 4771724 Puff/Cache
KiB Swap: 7999480 gesamt, 7856804 frei, 142676 belegt. 5873504 verfü Spch
PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
15554 www-data 20 0 297728 39588 9404 S 0,3 0,5 0:00.56 /usr/sbin/+
16071 hunde 20 0 430080 23252 14124 S 0,3 0,3 0:00.26 php-fpm
16085 hunde 20 0 430080 23072 13932 S 0,3 0,3 0:00.19 php-fpm
1 root 20 0 185828 6256 3984 S 0,0 0,1 6:33.51 systemd
2 root 20 0 0 0 0 S 0,0 0,0 0:00.48 kthreadd
3 root 20 0 0 0 0 S 0,0 0,0 0:07.14 ksoftirqd/0
5 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/0:+
7 root 20 0 0 0 0 S 0,0 0,0 11:28.00 rcu_sched
8 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0,0 0,0 0:00.74 migration/0
10 root rt 0 0 0 0 S 0,0 0,0 0:03.68 watchdog/0
11 root rt 0 0 0 0 S 0,0 0,0 0:03.90 watchdog/1
12 root rt 0 0 0 0 S 0,0 0,0 0:00.70 migration/1
13 root 20 0 0 0 0 S 0,0 0,0 0:09.15 ksoftirqd/1
15 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/1:+
16 root rt 0 0 0 0 S 0,0 0,0 0:03.77 watchdog/2
17 root rt 0 0 0 0 S 0,0 0,0 0:00.72 migration/2

The time between (FileZilla)
Status: Retrieving directory listing of "/httpdocs/images"...
and
Status: Directory listing of "/httpdocs/images" successful
is awfully long.
10 sec for 10,000 files
26 sec for 14,000 files
50 sec for 25,000 files
Just to list the directory!

I searched several forums to find a solution. Nothing...

 

[Bonsai78]

Did you try yourself with Filezilla? Is there something in the status log of Filezilla?

 

[Chopper27]

Yes, I did.

Here ist the status log of Filezilla:
Status: Connecting to 81.169.180.49:21...
Status: Connection established, waiting for welcome message...
Status: Logged in
Status: Retrieving directory listing...
Status: Directory listing of "/" successful
Status: Retrieving directory listing of "/httpdocs"...
Status: Directory listing of "/httpdocs" successful
Status: Retrieving directory listing of "/httpdocs/images"...
Status: Directory listing of "/httpdocs/images" successful

The time between the last log entries (from retrieving... to Directory listing ...) is more than 50 seconds. (25,000 files)

What I tried out in the last few days.
I took my old server (which I still own until the end of month) and installed a completely new system on it (provided by Strato)
Ubuntu 16.04.4 LTS‬
Plesk Onyx Version 17.8.11 Update #11
and migrated just one domain on it.
-> same problem

Later a new installation with
CentOS Linux 7.5.1804
Plesk Onyx Version 17.8.11 Update #14
and migrated just one domain on it.
-> same problem

I think the problem is Plesk Onyx!?!?

 

[Bitpalast]

No way that this can be a Plesk issue. If it is an issue at all, it is with the FTP service, but I doubt that this is a problem at all. Having 50,000 files in a single directory is a rather questionable setup. It is very well possible that reading and transferring this information takes a while plus Filezilla on the local machine will need some while to build a selector list with 50,000 entries.

For comparison: Try to simply list files via shell, e.g.
# ls -la
and measure the time that needs. This won't finish in a snap either, but take a long while to go through all hem 50,000 entries and display them - although here you are "directly" on the machine.

 

[Bitpalast]

On the old server, the listings come under 1 second.

P.S.: Because Filezilla caches the directory lists.

 

[burnley]

Sorry to dig up a nearly 12 months old thread, but I think there's an issue in ProFTPD 1.3.6 shipped with 17.8.11: Bug 4360 – slow directory listing in 1.3.6 in comparison to 1.3.5d
We've just seen it on a CentOS 7.6 server running 17.8.11. Comparing the results on a directory containing 164289 files:

1. psa-proftpd-1.3.6-cos7.build1708180220.17.x86_64
time ( lftp -u 'user,pass' -e 'ls /large_folder ; quit' plesk.server )
^C
real 32m10.471s
user 0m0.102s
sys 0m0.036s
Had to stop it, lftp kept reconnecting after timing out

2. psa-proftpd-1.3.5d-cos7.build1705170314.14.x86_64, after yum downgrade using the package from Index of /PSA_17.5.3/dist-rpm-CentOS-7-x86_64/opt/hosting/proftpd
time ( lftp -u 'user,pass' -e 'ls /large_folder ; quit' plesk.server )

real 0m21.141s
user 0m1.106s
sys 0m1.542s

 

[AYamshanov]

Hi,

The issue really exists, a few links about it:

• Bug 4360 – slow directory listing in 1.3.6 in comparison to 1.3.5d (bugs.proftpd.org)
  
• Listing of a directory with more than 20000 files via FTP takes a long time (support.plesk.com/)

 

[burnley]

Yep, thanks. How can I get the psa-proftpd src.rpm package as built by Plesk? I'd like to recompile it using test several 1.3.6 master branch iterations. It's important for us to run a fast 1.3.6 build without compromising the security by downgrading to 1.3.5.

 

[AYamshanov]

I have asked the developers; we don't build src.rpm but I have attached a zip-file with patches that are used when we build the psa-proftpd package. I hope it will help you.

[root@ayamshanov-test-vm proftpd]# md5sum *
592b1abd452977da29402f6092bf9460  plesk-proftpd-patches.tar.gz
13270911c42aac842435f18205546a1b  proftpd-1.3.6.tar.gz

 

[burnley]

Excellent, thanks

 

[zwankie]

Hi guys, any feedback or success with fixing this?

 

[HostaHost]

Seeing same issue; psa-proftpd-1.3.6-cos7.build1708180220.17 takes nearly five minutes to output the contents of a 200,000+ file directory, ls -al takes 3 seconds.

 

[HostaHost]

The change in proftpd 1.3.6 is fairly crazy; it does a per-character read of /etc/group, multiplied by every file in a directory, as it builds the output. So if your /etc/group is a kilobyte, and you have 10,000 files in a directory, it's going to do a minimum of ten million read() function calls. Must context switch like crazy. Plesk should really downgrade 17.8's version back to 1.3.5.