Issue Unexpected Disk Usage At The Same Time Everyday

[watchturks]

Server operating system version

AlmaLinux 8

Plesk version and microupdate number

18.0.57

Hello, I have a weird problem. My server's disk usage increases by 75%-100% at the same time every day. Meanwhile, freezing and slow loading problems occur on the server. Since I broadcast videos via the server, loading and buffering problems occur due to this problem.
There is no cron or backup set for this time. I've added the image below showing disk usage. This happens at the same time every day.

CleanShot 2024-01-05 at 10.32.07 - Photo 1
CleanShot 2024-01-05 at 10.33.06 - Photo 2

We suspected that raid check might occur at this time and we disabled it. However, the problem still persists.

I found some topics similar. It was mentioning that plesk maintenances may occur it. Is there anyone help me out?

 

[Peter Debik]

Have you checked
# ls -la /etc/cron.d
and the content of these files? Are you sure no backup is created?
# ls -la /var/log/plesk/PMM

 

[watchturks]

Yes, nothing wrong with them. There isn't any backup.

 

[WebHostingAce]

How about if we try find the directory sizes and go from there?

# find / -maxdepth 1 -mindepth 1 -type d -exec du -h --max-depth=0 {} \;

 

[hschramm]

if the problem occours on the same time every day run iotop to look what processes are writing/reading data

 

[watchturks]

if the problem occours on the same time every day run iotop to look what processes are writing/reading data

Thank you. I'll run it tonight and let you know the results.

 

[watchturks]

I have added some screenshots. Can you please help?

 

[Peter Debik]

What is that rsync process for? It looks like an extra backup of files. What software starts it on your server?

 

[watchturks]

What is that rsync process for? It looks like an extra backup of files. What software starts it on your server?

I move all data to another server. It happens when resyn is closed too.

I have seen md2 raid 5, md3 raid 5 processes. I think it checks disks as there are software raid. We tried to close it but it still continues.

 

[Peter Debik]

But would that rsync not be the main cause for the high disk I/O load?

 

[watchturks]

No, It was not open for 7 days and I had the same issue every day. I/O was still increasing.

 

[Peter Debik]

In that case I suggest testing again when that special one process is not active. The measurements above are useless if they do not represent the typical load situation. All your screenshots clearly suggest that the cause of an increased disk I/O load is the rsync process. If that process is missing, no high I/O load will be measured.

 

[hschramm]

@watchturks: I fully agree to Peter. Your screens are now useless, because they show not the typical situation. I have looked at the screens in your first post. vhostmng-find is known for causing i/o problems due to the searching through all vhosts. maybe your hoster is doing backups or sth at this time.

 

[watchturks]

Thanks @Peter Debik and @hschramm . I understand your explanation. Yes, it seems that rsync process is affecting.However, It's closed for 2 days and something still increases the I/O. I sent you last nights monitoring.

I talked to Hetzner and they say that there isn't any backup or check on their end. They only offer changing disks or server. However, I have 40TB data and it is impossible to do it in a short time.

Last night, there was also intense buffering with USA users. I could watch anything perfectly but USA users had very bad buffering.

That's really becoming weird.

 

[AYamshanov]

Based on the screenshot from the last post (spikes on the same time when "vhostmng-find" is shown on the "Top Processes" graph), what about the issue described in Question - vhostmng-find processe consume CPU Sounds similar. And probably the process is not started by hoster, it could be a part of Plesk...

 

[watchturks]

Based on the screenshot from the last post (spikes on the same time when "vhostmng-find" is shown on the "Top Processes" graph), what about the issue described in Question - vhostmng-find processe consume CPU Sounds similar. And probably the process is not started by hoster, it could be a part of Plesk...

Hi, I have checked this post. There isn't any solution on that post.

 

[AYamshanov]

In that thread abashurov suggested enabling debug to collect mode data.
Do you use "ext-composer" extension on your server? If it possible, as a quick test, I think, you can disable it for a night and check graphs on the next morning.

 

[watchturks]

I have checked and we don't have that extention. When I check last nights monitoring, I have realised vhost-find.

CleanShot 2024-01-11 at 11.01.36

Screenshot uploaded to CleanShot Cloud

share.cleanshot.com

 

[watchturks]

I still have the same issue. I couldn't fix it. I just got an alert regarding swap usage. It's the same time I have issues. Does it related to it?