Issue Alma 8/9 increasing number of server crashes for apparently "no reason"

[Bitpalast]

We are seeing an increasing number of random server crashes on Alma 8 and 9 systems since about April 2025. It seems that the load on servers strongly increases within seconds so that the system becomes unresponsive. However, there is no traffic increase seen, no suspicious log entries, just nothing that points to a cause in websites, database or PHP usage. We suspect that the wave of updates that were delivered during the past months by Alma could have something to do with it, but it's just a guess, because that's the only thing that has changed.

My question here is whether others on Alma 8 or 9 have experienced similar symptoms lately.

 

[Sebahat.hadzhi]

Peter, FWIW, I cannot see any unusual activity in the support queue reporting abnormalities with the server load and random crashes.

 

[manni]

Hello Peter, maybe a Problem on the Infrastructure Level? We had such situations already at HE and H…

 

[Sguk]

Yes, we are seeing the same since upgrading a cpanel server to Alma last month.
Prior to that no crashes in months, now at least once a week it just stops responding.
Can’t log into whm, nothing. We have to do a reset in vcenter.

 

[Bitpalast]

Hello Peter, maybe a Problem on the Infrastructure Level? We had such situations already at HE and H…

I think that on one server we might have caught a real hardware error with the PCIe interface on the RAID controller. But: We still saw this on two other machines lately. On one of them it might have been caused by too many processes, but we'll still need to see what will happen on that machine during the next few weeks. We're doing snapshots of the process list there every five seconds, so hopefully we'll gain some insights if the issue appears again. However, on one other machine, RAID controller is totally different and operating system is different, too, plus there are no signs for any extensive use by customers. It still feels unsafe, but with nothing logged it'll be awfully hard to find out more.

 

[tanasis]

Do you recommend the new Plesk update 18.0.70 or not on Alma because of this problem?

 

[Bitpalast]

Do you recommend the new Plesk update 18.0.70 or not on Alma because of this problem?

It doesn't matter. The issue is probably not caused by Plesk.

 

[kevinjansen]

Hi Bitpalast,

I know this is a old topic but do you still have this issue, or did you manage to fix it?
We also expierence the same on some of our servers. Within seconds the memory gets filled, cpu starts to max-out because swapping, and the server is not responding anymore, all we can do is hard reset the virtual machine.

We mainly use Almalinux 8, never seen it on Centos, and i dont think i have seen it happen on our Debian versions.
I am now planning to upgrade to Almalinux 9, or maybe even move to Debian.

And instead of freeing memory with out-of-memory it just freezes.

 

[Bitpalast]

The issue could be that since a few months, an increasingly large number of extremely stupid bad bot attacks are hitting servers, specifically from Google User Space ip ranges, but also from other ranges. They are now scanning for all kinds of .env files in websites and other file types such as .gz, .tar.gz hoping to find leftover files that contain login credentials. These attacks are carried out within seconds across many websites on a server so that typically dozens of domains can be affected and hundreds of requests sent. Depending on your configuration this can instantly lead to memory exhaustion, especially when websites are configured to respond not with a 404, but with a rewrite to the site's homepage.

What you should definitely be looking into is your log files. I also recommend to add a small section to the "Additional Nginx Directives" that block the stupid attacks by responding with 444, so that these never reach Apache or any PHP processing. Example how to block such requests, not only limited to .env in the request string:

if ($request_uri ~* "/(\.env|\.git|composer\.(json|lock)|some-php-file\.php)(/|$)") {
    return 444;
}