Resolved High IO by sa-learn / spamtrain

[CGI1979]

Dear Plesk-friends,

at 6 in the morning sa-learn / spamtrain starts to learn from junk mails.
This causes high load on the system und takes about 2-3 hours.

Is there any way to change the time to some hours earlier to avoid high load in office hours?

Thanks for your help.

 

[IgorG]

I believe this issue is known to us – please see if KB - http://kb.odin.com/en/124060 solves it for you.

 

[Bobbbb]

I believe this issue is known to us – please see if KB - http://kb.odin.com/en/124060 solves it for you.

Hello,

I have reviewed the KB article (http://kb.odin.com/en/124060). On Plesk 11 (CentOS 6.6), I only have one Spamassassin related file (60sa-update) in "/etc/cron.daily", which simply runs "/usr/bin/sa-update". I do not see where or how sa-learn/spamtrain is executed from those files. As a result, I am unable to try the 3rd suggested work-around listed in the KB article ("3. Use nice/renice or ionice utilities to decrease the priority of the process sa-learn").

How, and from where, are the sa-learn/spamtrain processes started? How can just the sa-learn/spamtrain processes be disabled? How can "ionice -c2 -n7 nice -n19" be added to the command that starts the sa-learn/spamtrain processes?

Thanks for your time and help.

Regards,
Bob

 

[Bobbbb]

It appears the sa-learn/spamtrain processes are started by the "50plesk-daily" cron job which runs "/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php" which, according to KB article, runs the following subroutines...

# /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -l
CheckForUpdates
InstallUpdates
update-keys
plesk-usage
autoreport
sitebuilder
mail-usage
execute-statistics
optimize-statistics
analyse-domain-statistics
analyse-client-statistics
execute-spamtrain
execute-glcleaner
execute-apache-config-rotation
ppsmbe-filesharing
UpgradePanel

How can this single routine (execute-spamtrain) be disabled, or made to work with nice and ionice?

I tried prefixing the entire cron command with nice and ionice, but that has not appeared to effect the issue...

ionice -c2 -n7 nice -n19 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php >/dev/null 2>&1

When the high (CPU and Disk IO) load occur from the sa-learn/spamtrain processes, it frequently causes other services to fail, such as mysql, which then causes web site databases to be marked as crashed and require repair.

 

[Bobbbb]

Would clearing the Spamassassin database (perhaps on a monthly cron job) be enough to prevent the sa-learn/spamtrain processes from causing high load? See...

http://talk.plesk.com/threads/clear-spamassassin-database.318956/

 

[UFHH01]

Hi Bobbbb,

first of all, you could modify your unique command as:

nice -n 19 ionice -c2 -n7 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php >/dev/null 2>&1​

... to solve the issue, that it didn't seem to have any effect to your CPU/IO usage. Please try as well the full path, if you experience issues, as for example:

/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php >/dev/null 2>&1​

Please make sure, that "nice" and "ionice" are located at the given paths and don't just copy&paste the suggestion with the whole path - usage ( you could use "locate nice" / "locate ionice" to verify the paths ).



You asked as well for a different execution time, to avoid high usage during office hours. I would recommend to try several spamtrain - cronjobs, which you might setup with

For Plesk 11 you would use:
nice -n 19 ionice -c2 -n7 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f execute-spamtrain >/dev/null 2>&1
/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f execute-spamtrain >/dev/null 2>&1

For Plesk 12 you would use:
nice -n 19 ionice -c2 -n7 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f ExecuteSpamtrain >/dev/null 2>&1
/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f ExecuteSpamtrain >/dev/null 2>&1
​

or use the direct command with:
nice -n 19 ionice -c2 -n7 /usr/local/psa/admin/sbin/spamtrain >/dev/null 2>&1
/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /usr/local/psa/admin/sbin/spamtrain >/dev/null 2>&1​

If you still would like to exclude one of the daily DailyMaintenance - cronjobs from your "50plesk-daily" script, you have to modify the line

/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php >/dev/null 2>&1

to all different parts like:
/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f Plesk-COMMAND-name1
/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f Plesk-COMMAND-name2
/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -f Plesk-COMMAND-name3
...​

... where you have to leave out the commands, which should not be executed daily. ( Please see the current Plesk-commands for the DailyMaintenance - script with "/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -l" and be aware that Plesk 11 and Plesk 12 have different commands! )



From time to time, it is wise to clean up the spamassassin - database ( as you already found out with the linked forum-thread), but I would recommend a 6-month-cronjob and not a monthly one.
Please keep in mind, that spamtraining with spamassassin can be very time investing and you should try out addons, filters and rules to find the most efficient configuration for each, unique server - unfortunately there is no "best-for-all" solution.

 

[Bobbbb]

Thank you for your helpful and detailed reply.

I would be satisfied to do without the spamtraining and just use the typical Spamassassin ruleset. Can the spamtraining be turned off while still using the typical Spamassassin ruleset?

One followup question... If I break out the 50plesk-daily tasks separately, as you described, what order should I put them in? The same order as displayed by the "script.php -l" command (leaving out the "execute-spamtrain" command)?

 

[UFHH01]

Hi Bobbbb,

feel free to use an order of your choice, untill one of the Plesk-Team-Members writes something different.

Each maintenamce task is unique as far that I know and doesn't rely on a other one - maybe a Plesk-Team-Member is going to confirm my statement.


Please keep in mind, that Plesk updates/upgrades/patches might overwrite your unique settings without any notice - so please make a backup of the original and your unique "50plesk-daily" - file, to restore your settings in case of replacments.

 

[ehrenwert]

I got the same error... I will double-check your solution and give feedback.