• Please be aware: Kaspersky Anti-Virus has been deprecated
    With the upgrade to Plesk Obsidian 18.0.64, "Kaspersky Anti-Virus for Servers" will be automatically removed from the servers it is installed on. We recommend that you migrate to Sophos Anti-Virus for Servers.
  • The Horde webmail has been deprecated. Its complete removal is scheduled for April 2025. For details and recommended actions, see the Feature and Deprecation Plan.

Resolved high CPU usage by 'sw-engine' with task 'ext-log-browser-parse-maillog'

V

vospy

New Pleskian
Server operating system version
Ubuntu & CentOS
Plesk version and microupdate number
18.0.61 #6
Since 2 days I have high CPU usage on my Plesk server, a sw-engine task runs constantly at 100% CPU usage.

Screenshot_20240718_012320_plesk_cpuload_monitoring.png

Code: 
psaadm   29290 94.7  2.4 371724 103596 ?       Rs   Jul17 1300:12 /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/plib/scripts/task-async-executor.php -task-id 5857

Screenshot_20240718_011327_plesk_cpuload_htop.png

With command plesk db "SELECT id,type,status,finishTime FROM longtasks WHERE status <> 'done'" I found out, that the task 5857 is ext-log-browser-parse-maillog.

Code: 
> plesk db "SELECT id,type,status,finishTime FROM longtasks WHERE status <> 'done'"
+------+-------------------------------+-------------+---------------------+
| id   | type                          | status      | finishTime          |
+------+-------------------------------+-------------+---------------------+
| 5855 | ext-log-browser-parse-maillog | error       | 2024-07-16 03:28:42 |
| 5856 | ext-log-browser-parse-maillog | error       | 2024-07-17 03:28:04 |
| 5857 | ext-log-browser-parse-maillog | started     | 0000-00-00 00:00:00 |
| 5858 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5861 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5862 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5863 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5864 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5865 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5866 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5867 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5868 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5869 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5870 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5871 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5872 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5873 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5874 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5875 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5876 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5877 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5878 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5879 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5880 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5881 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5882 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5883 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5884 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5887 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5888 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5889 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5890 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5891 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5892 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5893 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5894 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5895 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5896 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5897 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5898 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5899 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5900 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5901 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5902 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5903 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5904 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5905 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5906 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5907 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 5908 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
+------+-------------------------------+-------------+---------------------+

I'm using Plesk Obsidian - Version 18.0.61 Update #6
This problem occurs on two different servers, one is running with Ubuntu, the other one with CentOS.

How to fix the problem?
 
Hi @vospy,

Thank you for the provided details!
We also would love to get any additional information that will help us to investigate the issue (e.g.: how many email addresses it serves? how big log files are on the servers? Any errors related to the long tasks in the system/Plesk logs?)

vospy said:
How to fix the problem?
Click to expand...
As a temporary fix/workaround, you can disable the "Track Email Delivery" feature on affected servers with the next panel.ini option
Code: 
[ext-log-browser]
mailQueue.customerAccess = false
 
Thanks for your reply.

In /var/log/plesk/panel.log I found:

Code: 
[2024-07-16 03:28:41.845] 4271:6695ccc99fc3c ERR [panel] Task is not responding: id=5855, pid=1006, type=ext-log-browser-parse-maillog
[2024-07-16 03:28:42.051] 4271:6695ccc99fc3c ERR [panel] Task is not responding: id=5855, pid=1006, type=ext-log-browser-parse-maillog, clientRemoteAddr=127.0.0.1, rotateMail=1
[2024-07-17 03:28:04.050] 10756:66971e23e655b ERR [panel] Task is not responding: id=5856, pid=10870, type=ext-log-browser-parse-maillog
[2024-07-17 03:28:04.052] 10756:66971e23e655b ERR [panel] Task is not responding: id=5856, pid=10870, type=ext-log-browser-parse-maillog, clientRemoteAddr=127.0.0.1, rotateMail=1
[2024-07-18 03:28:48.438] 32750:66986fd05466f ERR [panel] Task is not responding: id=5857, pid=29290, type=ext-log-browser-parse-maillog
[2024-07-18 03:28:48.438] 32750:66986fd05466f ERR [panel] Task is not responding: id=5857, pid=29290, type=ext-log-browser-parse-maillog, clientRemoteAddr=127.0.0.1, rotateMail=1

The number of mail addresses is 90 (determined with plesk db "SELECT COUNT(*) FROM mail;").

The maillog file size:

Code: 
ll -h /var/log/maillog*
-rw-r----- 1 syslog adm  543K Jul 18 11:28 /var/log/maillog
-rw-r----- 1 root   root 9,7M Jul 18 03:12 /var/log/maillog.processed
-rw-r----- 1 root   root 1,2M Jul 12 03:16 /var/log/maillog.processed.1.gz
-rw-r----- 1 root   root 1,1M Jul  6 03:15 /var/log/maillog.processed.2.gz
-rw-r----- 1 root   root 1,2M Jul  2 03:12 /var/log/maillog.processed.3.gz

In syslog I can't find an error which I can associate with the maillog-parse

Is there an other logfile I can check?
 
This extension uses the systemd-journal to get it's log data. It's not reading log files in /var/log/. I recommend checking that your systemd-journal isn't corrupted:

Code: 
journalctl --verify

Journald log retention options are controlled in file /etc/systemd/journald.conf. Here is a KB article about them:

How to Automate Journalctl Log Cleanup on Linux – TecAdmin

Journalctl is a command-line utility for viewing system logs in Linux distributions that use Systemd. It’s an essential tool for system administrators to diagnose problems, monitor system performance, and track changes to the system. However, journal logs can consume large amounts of disk space...
tecadmin.net

The SystemMaxUse and MaxRetentionSec options control the journal data retention limits in /etc/systemd/journald.conf . If you make changes remember to restart the systemd-journal with the command:

Code: 
systemctl restart systemd-journald

Pruning the systemd-journal examples:

Code: 
// Remove all files older than 7 days
journalctl --vacuum-time=7d

// limit the journal to 1 GB
journalctl --vacuum-size=1G

// for more help about all the options
man journalctl
 
I did the --verify, there were some old files which failed, but after a vacuum all problematic files are gone, so the verify runs without an error, now.
Checking journalctl --disk-usage the total amount is now at 504 M.
Then I killed the sw-engine process, running since 20 hours with 100%.

Plesk starts a new sw-engine process with the next task-id (as listed in the first post) with CPU usage of 100%,
so the problem still exists.
 
Just to be sure, did you disable "Track Email Delivery" feature as suggested above? If so, are there still new errors logged like these?
Code: 
[2024-07-16 03:28:41.845] 4271:6695ccc99fc3c ERR [panel] Task is not responding: id=5855, pid=1006, type=ext-log-browser-parse-maillog
 
I had not yet deactivated this because it was titled with "temporary fix/workaround" and I'am interested to find a sustainable solution.
Now, I've created the file /usr/local/psa/admin/conf/panel.ini with the content
Code: 
[ext-log-browser]
mailQueue.customerAccess = false

Hope that's the correkt full path? I think so, because in this directory is already the file panel.ini.sample.

Do I have to restart any service after changing the panel.ini file?

Since I didn't know if I had to restart any service, I restarted the whole server, but the problem seems to be present.
Now, after the reboot, the sw-engine process is running again with 100% CPU usage with task-id 5888.
Code: 
psaadm     634 87.5  1.9 351244 82904 ?        Rs   00:03  26:49 /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/plib/scripts/task-async-executor.php -task-id 5888
 
The newest entries in /var/log/plesk/panel.log:

Code: 
[2024-07-19 03:34:54.844] 788:6699c2bebc456 ERR [panel] Task is not responding: id=5858, pid=19129, type=ext-log-browser-parse-maillog
[2024-07-19 03:34:54.845] 788:6699c2bebc456 ERR [panel] Task is not responding: id=5858, pid=19129, type=ext-log-browser-parse-maillog, clientRemoteAddr=127.0.0.1, rotateMail=1
[2024-07-19 03:34:54.858] 788:6699c2bebc456 ERR [panel] Task is not responding: id=5861, pid=32490, type=ext-log-browser-parse-maillog
[2024-07-19 03:34:54.859] 788:6699c2bebc456 ERR [panel] Task is not responding: id=5861, pid=32490, type=ext-log-browser-parse-maillog, clientRemoteAddr=127.0.0.1, rotateMail=1

[2024-07-19 23:53:38.751] 28220:669ae062b7382 ERR [panel] Unable to find row with id 5863 in longtasks table.:
0: /opt/psa/admin/plib/Db/Table/Abstract.php:117
        Db_Table_Abstract->getOne(string '5863')
1: /opt/psa/admin/plib/Task/Async/Executor.php:33
        Task_Async_Executor->execute()
2: /opt/psa/admin/plib/scripts/task-async-executor.php:6
[2024-07-19 23:53:38.752] 28220:669ae062b7382 ERR [panel] Unable to find row with id 5863 in longtasks table.
[2024-07-19 23:53:38.909] 28231:669ae062ddd29 ERR [panel] Unable to find row with id 5864 in longtasks table.:
0: /opt/psa/admin/plib/Db/Table/Abstract.php:117
        Db_Table_Abstract->getOne(string '5864')
1: /opt/psa/admin/plib/Task/Async/Executor.php:33
        Task_Async_Executor->execute()
2: /opt/psa/admin/plib/scripts/task-async-executor.php:6
[2024-07-19 23:53:38.909] 28231:669ae062ddd29 ERR [panel] Unable to find row with id 5864 in longtasks table.

...
» the same for task IDs: 5865, 5866, 5867, 5868, 5869, 5870, 5871, 5872, 5873, 5874, 5875, 5876, 5877, 5878, 5879, 5880, 5881, 5882, 5883, 5884
 
Server operating system version: Ubuntu
Plesk version and microupdate number: 18.0.62 Update #1


Hey everyone,

I have the same problem at our server.
  • task /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/plib/scripts/task-async-executor.php -task-id xxx running with 125% CPU
Code: 
# plesk db "SELECT id,type,status,finishTime FROM longtasks WHERE id = 25648"

+-------+-------------------------------+---------+---------------------+
| id    | type                          | status  | finishTime          |
+-------+-------------------------------+---------+---------------------+
| 25648 | ext-log-browser-parse-maillog | started | 0000-00-00 00:00:00 |
+-------+-------------------------------+---------+---------------------+


# plesk db "SELECT id,type,status,finishTime FROM longtasks WHERE status <> 'done'"
+-------+-------------------------------+-------------+---------------------+
| id    | type                          | status      | finishTime          |
+-------+-------------------------------+-------------+---------------------+
| 25451 | ext-log-browser-parse-maillog | error       | 2024-07-19 06:05:17 |
| 25500 | ext-log-browser-parse-maillog | error       | 2024-07-19 06:05:17 |
| 25512 | ext-log-browser-parse-maillog | error       | 2024-07-20 05:53:41 |
| 25553 | ext-log-browser-parse-maillog | error       | 2024-07-20 05:53:41 |
| 25555 | ext-log-browser-parse-maillog | error       | 2024-07-20 05:53:41 |
| 25557 | ext-log-browser-parse-maillog | error       | 2024-07-21 06:02:56 |
| 25600 | ext-log-browser-parse-maillog | started     | 0000-00-00 00:00:00 |
| 25648 | ext-log-browser-parse-maillog | started     | 0000-00-00 00:00:00 |
| 25651 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 25653 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 25655 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 25657 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 25659 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
| 25661 | ext-log-browser-parse-maillog | not_started | 0000-00-00 00:00:00 |
+-------+-------------------------------+-------------+---------------------+

I did
Code: 
# journalctl --verify
# journalctl --vacuum-time=7d
# journalctl --vacuum-size=1G
but it didn't change.

Also I created /usr/local/psa/admin/conf/panel.ini from panel.ini.sample and added
Code: 
[ext-log-browser]
mailQueue.customerAccess = false
and restarted plesk with
Code: 
service sw-engine restart && service sw-cp-server restart
but nothing changed.

/var/log/plesk/panel.log shows
Code: 
[2024-07-21 06:02:56.928] 13163:669c8870d6c8c ERR [panel] Task is not responding: id=25557, pid=16020, type=ext-log-browser-parse-maillog
[2024-07-21 06:02:56.930] 13163:669c8870d6c8c ERR [panel] Task is not responding: id=25557, pid=16020, type=ext-log-browser-parse-maillog, clientRemoteAddr=127.0.0.1, rotateMail=1
[2024-07-21 06:03:04.742] 13207:669c887453420 ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/filemng' 'root' 'exec' '/' 'bash' '-c' '/usr/bin/pro status --format json' '--allow-root'] with exit code [127]
[2024-07-21 06:03:04.744] 13207:669c887453420 ERR [panel] '/opt/psa/admin/bin/filemng' 'root' 'exec' '/' 'bash' '-c' '/usr/bin/pro status --format json' '--allow-root' failed with code 127.

stdout:


stderr:
--allow-root: /usr/bin/pro: No such file or directory


So I am looking for any update on this topic.
Thanks, Matthias
 
I am sorry that this causing some much inconvenience for you guys. We are working on a fix to drastically reduce load, but there on ETA when this will be released.

Disabling the mailQueue.customerAccess in the panel.ini file should alleviate load, but it seems that in you case it doesn't (or not enough at least). What you additionally can try is to kill any background tasks related to the log browser (ext-log-browser-parse-maillog). This article discribes how to manually kill background taks: https://support.plesk.com/hc/en-us/articles/12377460333463--How-to-cancel-a-stuck-task-in-Plesk

If that does not work either you can consider to disable the log browser extension plesk bin extension --disable log-browser until we've released a fix.
 
We have the exact same issue with some of our servers.
Plesk Obsidian Web Host Edition
Version 18.0.61 Update Nr. 6
Problem exists since
Code: 
[2024-07-17 03:25:18.815] 68942:6697399ec6cf1 ERR [panel] Unable to find row with id 1446 in longtasks table.:
0: /opt/psa/admin/plib/Db/Table/Abstract.php:117
        Db_Table_Abstract->getOne(string '1446')
1: /opt/psa/admin/plib/Task/Async/Executor.php:33
        Task_Async_Executor->execute()
 
Same issue, started 2 days ago. I used the TASK MANAGER in the GUI and manually eliminated all the hanging processes for ext-log-browser-parse-maillog. After finishing stopping all the processes, resources went back to normal.
 
A few minutes ago, a new version of the Log Browser extension was released, it contains fixes to reduce CPU usage and fixes for other known issues.
Let us know if it does not help.
 
After one day of cleaning the Task Manager, the problem started again. So I have updated PLESK to Obsidian 18.0.63 (I was in Obsidian 18.0.62). Will let you know if the problem persists.
 
After the update the process is using less resources, but still using a lot of resources. The server is at 165%, and I need to keep killing the process ext-log-browser-parse-maillog for the server to go back to normal (it was usually at 1-2% load).
 
Processing the logs for the first time may consume extra server resources. Afterward, the extension will process only changed data every hour. If you kill the process before it finishes the first pass through the logs, the service will try again next time.

I would recommend you to decrease the period of time that need to cache with the next option in the `panel.ini` file,
Code: 
[ext-log-browser]
mailQueue.firstEntryOffset = 2678400 ; in seconds
 
Thanks. Things are running smoothly now, after the updates, and killing the process. Did not have to change the panel.ini.
 
You must log in or register to reply here.

Similar threads

Liew CheonFong
Issue Health Manager: High Apache CPU alarms (sw-engine)
Replies
5
Views
2K
iA001
I
haaid
Issue CPU Load sw-engine
Replies
13
Views
3K
Bitpalast
Bitpalast
Back
Top