Question Grafana/Monitoring reported boot partition usage warning today, but no usage change lately

[Bitpalast]

Server operating system version

Alma 8 and 9

Plesk version and microupdate number

18.0.77 #2 and 18.0.78 latest

This morning on all our Alma 8 and Alma 9 servers, Grafana Monitoring tool reported that the threshold of boot partition disk usage was exceeded. However, when I skip backwards to previous days, nothing has changed on the boot partitions. This should only leave the conclusion that the Grafana threshold value was changed so that all servers suddenly reported an overuse.

Is this a known issue/behavior?

As I have not visited this report for a very long while (because nothing should really change on the boot partition unless a crash dump is created or a new kernel installed) I am not sure what the threshold was before. I did check whether new files are shown in the boot partition, but the last changes were in May when a kernel update was mandatory due to the Alma CVEs. Nothing changed after that update, yet today was the first time the Grafana Monitoring warnings came in.

 

[Bitpalast]

I also saw that the warning was reported twice: Once at night (e.g. 5.09 am) when obviously something was checked and then again when I opened the panel to check what's going on (at around 9 am). Normally, a disk usage warning only appears once.

 

[Sebahat.hadzhi]

Hi, Peter. There was a recent Grafana upgrade and according to our team's test the thresholds should have been migrated successfully (if legacy alerting was used). I will double-check with our team if that could potentially be the reason for the issue you are noticing.

 

[Kaspar]

For what it's worth, I had almost the same happening on one Alma 9 server server, for me the threshold notifications went off every 4 hours (4 AM, 7 AM, 11 AM) for everything (Memory usage, Disk partition, Services Memory usage), except for anything CPU related. The monitoring dashboard also showed data couldn't be loaded, but I forget what the exact error message was. Oddly enough, the graphs on the Home Dashboard where still working and didn't show any resource usages spikes. I could not be bothered to investigate and re-installed the Monitoring and Grafana extensions instead. Which did resolve the issue for me.

 

[Sebahat.hadzhi]

@Bitpalast @Kaspar could you please confirm if you had legacy alerting was enabled (Grafana extension 1.7.0 re-enabled alerting via unified alerting. The previous version, 1.6.6, had legacy alerting disabled unless it was manually enabled via grafana.ini?

Also, could you please double-check if any errors/warning appear in /var/log/plesk/panel.log and /var/log/grafana/grafana.log?

 

[Kaspar]

I don't recall manually enabling legacy alerting on this particular server, but I am not 100% sure. I deployed the server only recently, without making much modifications to it. The current grafana.ini file has the option commented out (;enabled = false), but this might have been set or influenced when I re-installed the Grafana extension.

Noting related in the panel.log. This is an excerpt from the grafana.log:

ogger=renderer.manager t=2026-05-19T16:08:34.531063207+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=renderer.manager t=2026-05-22T17:54:14.607962779+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=plugin.sources t=2026-05-22T17:54:14.610037317+02:00 level=error msg="Failed to load external plugins" error="failed to open plugins path"
logger=renderer.manager t=2026-05-22T17:54:14.767376366+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=renderer.manager t=2026-06-06T23:18:23.117428146+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=plugin.sources t=2026-06-06T23:18:23.122561231+02:00 level=error msg="Failed to load external plugins" error="failed to open plugins path"
logger=renderer.manager t=2026-06-06T23:18:23.272659213+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=plugin.signature.key_retriever t=2026-06-06T23:18:23.276962227+02:00 level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": dial tcp: lookup grafana.com on 212.227.123.17:53: write udp XX.XX.XX.XX:39767->212.227.123.17:53: write: operation not permitted"
logger=renderer.manager t=2026-06-08T12:38:44.28053585+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=plugin.sources t=2026-06-08T12:38:44.287291633+02:00 level=error msg="Failed to load external plugins" error="failed to open plugins path"
logger=renderer.manager t=2026-06-08T12:38:44.464938+02:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"

logger=plugin.plesk-json-backend-datasource t=2026-06-16T03:40:32.701945213+02:00 level=error msg="plugin process exited" plugin=/var/lib/grafana/plugins/plesk-json-backend-datasource/dist/gpx_plesk-json-backend-datasource_linux_amd64 id=268750 error="signal: terminated"
logger=plugin.plesk-json-backend-datasource t=2026-06-16T03:40:34.749640133+02:00 level=error msg="plugin process exited" plugin=/var/lib/grafana/plugins/plesk-json-backend-datasource/dist/gpx_plesk-json-backend-datasource_linux_amd64 id=269315 error="signal: terminated"
logger=plugins.validation t=2026-06-16T03:40:34.983327999+02:00 level=error msg="Plugin validation failed" pluginId=grafana-exploretraces-app error="plugin 'grafana-exploretraces-app' has an invalid signature"
logger=context userId=1 orgId=1 uname=admin t=2026-06-16T03:40:36.952392465+02:00 level=error msg="Request Completed" method=PUT path=/api/datasources/1 status=500 remote_addr=127.0.0.1 time_ms=3 duration=3.372334ms size=41 referer= handler=/api/datasources/:id status_source=server error="data source not found"
logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=d011f8aaac80116d now=2026-06-16T03:40:40+02:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-16T03:40:40.003976065+02:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=d011f8aaac80116d now=2026-06-16T03:40:40+02:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-16T03:40:40.004047713+02:00 level=error msg="Failed to evaluate rule" attempt=1 max_attempts=3 next_attempt_in=923.407742ms error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=d011f8aaac80116d now=2026-06-16T03:40:40+02:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-16T03:40:40.928282265+02:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=d011f8aaac80116d now=2026-06-16T03:40:40+02:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-16T03:40:40.928363856+02:00 level=error msg="Failed to evaluate rule" attempt=2 max_attempts=3 next_attempt_in=1.631457719s error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=d011f8aaac80116d now=2026-06-16T03:40:40+02:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-16T03:40:42.56078673+02:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=c43e7f7d6ec79d7c now=2026-06-16T03:41:20+02:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-16T03:41:20.001092492+02:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=c43e7f7d6ec79d7c now=2026-06-16T03:41:20+02:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-16T03:41:20.001196244+02:00 level=error msg="Failed to evaluate rule" attempt=1 max_attempts=3 next_attempt_in=961.254609ms error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=c43e7f7d6ec79d7c now=2026-06-16T03:41:20+02:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-16T03:41:20.963570481+02:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=c43e7f7d6ec79d7c now=2026-06-16T03:41:20+02:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-16T03:41:20.963661686+02:00 level=error msg="Failed to evaluate rule" attempt=2 max_attempts=3 next_attempt_in=1.56764737s error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mysql org_id=1 version=1 fingerprint=6ca5c31dec8c9021 now=2026-06-16T03:41:20+02:00 rule_uid=monitoring__memory_usage__mysql org_id=1 t=2026-06-16T03:41:22.001816621+02:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mysql org_id=1 version=1 fingerprint=6ca5c31dec8c9021 now=2026-06-16T03:41:20+02:00 rule_uid=monitoring__memory_usage__mysql org_id=1 t=2026-06-16T03:41:22.001916749+02:00 level=error msg="Failed to evaluate rule" attempt=1 max_attempts=3 next_attempt_in=1.050551012s error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"

 

[websavers]

Same issue here. Getting alerts about every single item that's monitored, yet each one is nowhere near its threshold.

 

[Bitpalast]

@Bitpalast @Kaspar could you please confirm if you had legacy alerting was enabled (Grafana extension 1.7.0 re-enabled alerting via unified alerting. The previous version, 1.6.6, had legacy alerting disabled unless it was manually enabled via grafana.ini?

Also, could you please double-check if any errors/warning appear in /var/log/plesk/panel.log and /var/log/grafana/grafana.log?

I am sorry, but I cannot deliver any further data on the case. I only installed Grafana Monitoring with the default settings once, then, years ago, configured thresholds, then never again touched it. Whether or not there was or is a setting called "legacy alerting" I don't know.

 

[websavers]

The server we are getting this from has been around for many years, so chances are there is legacy alerting configured. Reinstalling Grafana did not help, neither did reinstalling the monitoring extension. Every few hours we get alerts that thresholds are exceeded, yet in looking none of them have been. We have not experienced this with any other server.

 

[websavers]

could you please confirm if you had legacy alerting was enabled (Grafana extension 1.7.0 re-enabled alerting via unified alerting. The previous version, 1.6.6, had legacy alerting disabled unless it was manually enabled via grafana.ini?

A search of grafana.ini for 'legacy' shows nothing but comments and none of the connected options are enabled (they're commented out).

Also, could you please double-check if any errors/warning appear in /var/log/plesk/panel.log and /var/log/grafana/grafana.log?

Just failed logins in panel.log

Whole bunch like this in grafana.log:

logger=ngalert.scheduler rule_uid=monitoring__memory_usage__web org_id=1 version=1 fingerprint=f79a5e1c85c0d5de now=2026-06-17T14:52:20-03:00 rule_uid=monitoring__memory_us
age__web org_id=1 t=2026-06-17T14:52:26.945326666-03:00 level=error msg="Failed to evaluate rule" attempt=2 max_attempts=3 next_attempt_in=1.605994785s error="server side e
xpressions pipeline returned an error: failed to build query 'query': data source not found"

 

[ScottGoddard]

I have 2 different servers that have started sending alerts from every metric, saying it has been exceeded. Both show 'Access Denied' for every monitoring screen.
Logs are full of thousands of these:

Jun 17 22:14:21 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=28c927b23de47adf now=2026-06-17T22:14:20+01:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-17T22:14:21.030125086+01:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
Jun 17 22:14:20 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=28c927b23de47adf now=2026-06-17T22:14:20+01:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-17T22:14:20.001671768+01:00 level=error msg="Failed to evaluate rule" attempt=1 max_attempts=3 next_attempt_in=1.027580234s error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"
Jun 17 22:14:20 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__memory_usage__mail org_id=1 version=1 fingerprint=28c927b23de47adf now=2026-06-17T22:14:20+01:00 rule_uid=monitoring__memory_usage__mail org_id=1 t=2026-06-17T22:14:20.001575176+01:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
Jun 17 22:13:49 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=7600c928e7dec089 now=2026-06-17T22:13:40+01:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-17T22:13:49.16252123+01:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
Jun 17 22:13:47 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=7600c928e7dec089 now=2026-06-17T22:13:40+01:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-17T22:13:47.609038599+01:00 level=error msg="Failed to evaluate rule" attempt=2 max_attempts=3 next_attempt_in=1.551961322s error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"
Jun 17 22:13:47 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=7600c928e7dec089 now=2026-06-17T22:13:40+01:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-17T22:13:47.608903783+01:00 level=error msg="Failed to build rule evaluator" error="failed to build query 'query': data source not found"
Jun 17 22:13:46 grafana[1518988]: logger=ngalert.scheduler rule_uid=monitoring__hdd_util__root org_id=1 version=1 fingerprint=7600c928e7dec089 now=2026-06-17T22:13:40+01:00 rule_uid=monitoring__hdd_util__root org_id=1 t=2026-06-17T22:13:46.669515911+01:00 level=error msg="Failed to evaluate rule" attempt=1 max_attempts=3 next_attempt_in=938.611251ms error="server side expressions pipeline returned an error: failed to build query 'query': data source not found"

 

[Sebahat.hadzhi]

Thank you all for the details. I have forwarded them to our team and I will update you as soon as I have more information.

 

[nmdpa3]

For us its just on Windows servers.

 

[Sebahat.hadzhi]

Unfortunately, our developers were unable to replicate the issue on a test environment. If anyone has the option to open a support ticket and provide us with direct server access it will be really helpful.