Issue Server is permanently down and unavailable

[Kaspar]

200k of posts don't necessarily have to be a problem. But if they are queried all at once then that could very well cause a high load. Are all those 200k post loaded every hour when the RSS feed(s) get retrieved?

Are those RSS feed important? Or can you disable them?

 

[fab5freddie]

200k of post don't necessarily have to be a problem. But if they are queried all at once then that could very well cause a high load. Are all those 200k post loaded every hour when the RSS feed(s) get retrieved?

Are those RSS feed important? Or can you disable them?

there are 200k posts in total. maybe 10 - 30 articles are downloaded per hour on average. unfortunately i can't deactivate the feeds because that's the core task of the website. it provides news snippets from various websites with a link to the post.

 

[Peter Debik]

do you have any ideas on how to get the server stable, even if it slows down a bit? the main thing is that it doesn't go down.

-->

You've already did a analysis of the website with New Relic and discovered that (most of) the load is caused with the wp_posts database table. Which is really useful information, as it (likely) pin points to cause of the load of your server. ... why not optimize the website...? ... For example by experimenting with replacing plugins or the theme the reduce load.

@fab5freddie What did you actually try so far in website optimization or analysis to tackle the issue?

 

[Kaspar]

there are 200k posts in total. maybe 10 - 30 articles are downloaded per hour on average. unfortunately i can't deactivate the feeds because that's the core task of the website. it provides news snippets from various websites with a link to the post.

I see. So if I understand the situation correctly the RSS feed(s) of your website are probably causing (very) high load on your server when they are getting aggregated. However these RSS feed(s) are very important (as they are the core of your website). So you can't disable or reduce them.

With that in mind, I don't think using cgroups to limit resource usage is going to help you much. Because if the RSS feed(s) of your websites are indeed causing the high load every time they are aggregated the cgroups limit would kick in and access to the RSS feed(s) will likely fail. I guess that's of no use.

The only real solution I can think of is still to optimize your site. In particular the database queries. I know this is probably not what you wanted to hear, but based on the information in this topic this seems like the only solutions to me. As others already give great pointers on optimizing your server.

At the same time maybe this would be a good opportunity for your to reconsider if hosting this website yourself is such a good idea? Your already mentioned it's causing you a headache. Maybe your website would be better fit for a specialized Wordpress hosting provider?

 

[fab5freddie]

-->


@fab5freddie What did you actually try so far in website optimization or analysis to tackle the issue?

as i mentioned, i worked my way through a lot of tutorials on the internet. i tried the bot filters in the beginning because i thought the high traffic was caused by bad bots.

then i used cloudflare to see if it gets better because they have better defense mechanisms.

then i got into new relic and started analyzing with it. when i saw that it had to do with the wp_posts table, i rented a second server and moved the website, which took me days because i couldn't even do a successful move with duplicator pro because the DB is almost 4GB.

by the time I managed to make a backup and move, the server was too small. 4 cores and 8GB RAM were permanently used to the limit. so i rejected the server and ordered a new one with 8 cores and 16GB RAM. i carried out the move with the backup and again had maximum load.

after that i optimized the DB, deleted wp plugins and tried to slim down the site as much as possible and deleted the borlabs cache, which had grown to about 28GB.

currently i've installed cgroups and defined limits, but that doesn't have the effect i want yet, i'm still far away from my goal. i'm now reducing to 8GB RAM and 400% CPU.

let's see if it helps to not quite reach the maximum load so that the server remains accessible

 

[Peter Debik]

With only few dozen articles downloaded per hour, I'd deem it impossible to create a high load. If there is no activity in between, there should not be any load. If there is still a high load although nothing visible happens on the website what is logged to the error_log and access_ssl_log file at these times?

 

[fab5freddie]

With only few dozen articles downloaded per hour, I'd deem it impossible to create a high load. If there is no activity in between, there should not be any load. If there is still a high load although nothing visible happens on the website what is logged to the error_log and access_ssl_log file at these times?

thank you very much peter and everyone else who has tried to help me, but i can't anymore. i give up.
i'm going to kill this project after a year and a half. i've lost over a month now and can't get it solved. then i guess it's not meant to be.

thanks anyway guys. all the best.

 

[fab5freddie]

With only few dozen articles downloaded per hour, I'd deem it impossible to create a high load. If there is no activity in between, there should not be any load. If there is still a high load although nothing visible happens on the website what is logged to the error_log and access_ssl_log file at these times?

peter, one last question before i delete everything. is there any chance that someone from plesk support could take a look at my problem?

 

[Peter Debik]

peter, one last question before i delete everything. is there any chance that someone from plesk support could take a look at my problem?

Sure, that is actually recommended, because Plesk support is staffed with great engineers and highly experienced experts. To sign-in to support please go to https://support.plesk.com

If you experience login issues, please see this KB article:
https://support.plesk.com/hc/en-us/...rt-plesk-com-and-password-reset-does-not-work

If you bought your license from a reseller, your reseller should provide support for you. If the reseller does not provide support, here is an alternative:
https://support.plesk.com/hc/en-us/articles/12388090147095-How-to-get-support-directly-from-Plesk-
This comes with a 30 days free trial period.

 

[mow]

the only way i can explain the peaks is that 50 rss feeds are retrieved once an hour to feed the site with news, then sent to google and bots crawl the site. the interval used to be even shorter before and retrieved the rss feeds every 15 minutes.

Do you retrieve all those feeds sequentially or in parallel? How do you do that anyway, do you use a WP plugin or is it something you programmed yourself?

attached is a screenshot of "htop" and "glances".

What I notice is that grafana uses a lot of CPU. Like, really a lot.
Phusion Passenger Core too, what do you use that for?

 

[Peter99]

Are these RSS feeds generated - live - on the fly, as people request it by URL?
I have had success with other projects, where feeds are updated once a hour and people just download the generated feed by URL.

This way you don't have XX people generating a live feed at the same time, but they just request a feed that you already generated by a cronjob - saves you a lot of CPU and Memory.

 

[fab5freddie]

Do you retrieve all those feeds sequentially or in parallel? How do you do that anyway, do you use a WP plugin or is it something you programmed yourself?

honestly, i don't know. it's a wp plugin called "wp content pilot" that i use.

What I notice is that grafana uses a lot of CPU. Like, really a lot.
Phusion Passenger Core too, what do you use that for?

if i remember correctly, grafana was necessary to be able to display the plesk monitoring. i don't know what the phusion passenger core is installed for. at least i didn't install it knowingly. maybe it was installed as a package in dependency.

Are these RSS feeds generated - live - on the fly, as people request it by URL?

no. the server downloads the feeds and then publishes them as a post on the wordpress site.

 

[Peter Debik]

i don't know what the phusion passenger core is installed for. at least i didn't install it knowingly. maybe it was installed as a package in dependency.

Passenger is a server software required for NodeJS.

 

[fab5freddie]

i have now hired a linux admin on fiverr. apart from the fact that i got rid of money, he has solved the problem so to speak and created a new problem. supposedly he only made php-fpm settings.

the server is now running reasonably stable and all websites remain online, but you can no longer click on anything on the website (apart from external links). no matter what you click on, the page loads briefly and remains on the homepage.

how can i solve the problem now? i'm really going mad!!!