Hey guys,
I am running Plesk Obsidian 18.0.41 Update #1 on Ubuntu 18.04.6 LTS on a vServer with 6 cpus, 12gb ram and 500gb ssd.
Usually the full backup (~ 32gb) takes 6 hours but in the last week it took 12 hours and the last backup I had to cancel.
The backup is executed over night and now it causes issues in the morning (e.g. long page loads for users).
I made the following observations:
Today I tested a manual backup with some sites:
Does anyone have a suggestion on how to solve the issue?
I thank you in advance.
I am running Plesk Obsidian 18.0.41 Update #1 on Ubuntu 18.04.6 LTS on a vServer with 6 cpus, 12gb ram and 500gb ssd.
Usually the full backup (~ 32gb) takes 6 hours but in the last week it took 12 hours and the last backup I had to cancel.
The backup is executed over night and now it causes issues in the morning (e.g. long page loads for users).
I made the following observations:
- CPU and memory are low during backup, Load average is high
- I/O is high, when a page get accessed iotop shows processes with IO between 100% and 800%
- memory usage (in grafana) shows 100% cached, 2gb used
- In the backup log there are some gaps (30 minutes up to an hour)
- I thought very small files (like sessions, cached data) causes the issues, I excluded them
- I turned off the compression
Today I tested a manual backup with some sites:
- The backup of the first page was done in 4 minutes and had a size of ~3gb - no issues with page loads during that time
- The backup of the second page took to long and caused issues with page loads during the backup - the real memory usage in grafana was 12gb again (only 2gb used) - I had to cancel the backup
Does anyone have a suggestion on how to solve the issue?
I thank you in advance.
Code:
...
[2022-02-17 01:10:02.471| 837] INFO: Repository '/var/lib/psa/dumps': Initialized
[2022-02-17 01:10:02.878| 837] INFO: Calculate list of files to pack
[2022-02-17 01:11:22.081| 837] INFO: Repository '/var/www/vhosts/': Initializing...
[2022-02-17 01:11:22.089| 837] INFO: Repository '/var/www/vhosts/': Initialized
[2022-02-17 01:11:22.093| 837] INFO: Repository '/var/lib/psa/dumps': Pack from /var/www/vhosts/domain.com to domains/domain.com/backup_user-data_2202170022.tar. Additional files count: 0. Split size: -1
[2022-02-17 01:11:22.333| 837] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 01:11:22.341| 837] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-kMAh8E
[2022-02-17 01:11:22.341| 837] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 01:47:56.496| 837] INFO: The utility succesfully executed.
[2022-02-17 01:47:56.753| 837] INFO: backup-archiver finished. Exit code: 0
[2022-02-17 01:47:56.891| 6107] INFO: backup-archiver started : /opt/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/www/vhosts/domain.com --destination=domains/domain.com/backup_apache-files_2202170022.tar --listing-file=/tmp/bafezT0wN --created-index-file=/tmp/bif5iwCdL --incremental-dependencies=/tmp/bdfJobC0s --user=www-data --no-recursion --session-path=/opt/psa/PMM/sessions/2022-02-17-002120.451 --warnings=/tmp/bwsWj6BNa --compression-method=zstd --compression-level=no-compression
[2022-02-17 01:47:56.897| 6107] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2022-02-17 01:47:56.901| 6107] INFO: Repository '/var/lib/psa/dumps': Initialized
...
[2022-02-17 02:28:33.618|28690] INFO: Export file domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:28:33.630|28690] INFO: Create directory /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com
[2022-02-17 02:28:33.658|28690] INFO: Link /var/lib/psa/dumps/domains/domain.com/backup_user-data_2202170022.tar to /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:28:33.670|28690] INFO: Add file '/tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar' to archive as 'domains/domain.com/backup_user-data_2202170022.tar'
[2022-02-17 02:28:33.698|28690] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 02:45:21.553|28690] INFO: Export file domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:45:21.553|28690] INFO: Create directory /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com
[2022-02-17 02:45:21.553|28690] INFO: Link /var/lib/psa/dumps/domains/domain.com/backup_user-data_2202170022.tar to /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:45:21.557|28690] INFO: Add file '/tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar' to archive as 'domains/domain.com/backup_user-data_2202170022.tar'
[2022-02-17 02:45:21.557|28690] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 02:45:21.557|28690] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --directory /tmp/pmm-de-tmp-repo-aB5JpO --dereference --blocking-factor 1 --no-recursion --files-from /tmp/pmm-taw-af-includes-SSzCDh
[2022-02-17 03:14:50.735|15384] INFO: The utility succesfully executed.
[2022-02-17 03:14:51.015|15384] INFO: backup-archiver finished. Exit code: 0
...
[2022-02-17 05:12:52.041|24711] INFO: backup-archiver started : /opt/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/qmail/mailnames/domain.com --destination=domains/domain.com/backup_domainmail_2202170022.tar --listing-file=/tmp/bmmuLi6sg --created-index-file=/tmp/bifgfR9nn --incremental-dependencies=/tmp/bdfQL8AOe --user=popuser --no-recursion --session-path=/opt/psa/PMM/sessions/2022-02-17-002120.451 --warnings=/tmp/bwsFeF2e6 --compression-method=zstd --compression-level=no-compression --id=33
[2022-02-17 05:12:52.042|24711] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2022-02-17 05:12:52.042|24711] INFO: Repository '/var/lib/psa/dumps': Initialized
[2022-02-17 05:12:52.043|24711] INFO: Calculate list of files to pack
[2022-02-17 05:12:58.783|24711] INFO: Repository '/var/qmail/mailnames/': Initializing...
[2022-02-17 05:12:58.787|24711] INFO: Repository '/var/qmail/mailnames/': Initialized
[2022-02-17 05:12:58.787|24711] INFO: Repository '/var/lib/psa/dumps': Pack from /var/qmail/mailnames/domain.com to domains/domain.com/backup_domainmail_2202170022.tar. Additional files count: 0. Split size: -1
[2022-02-17 05:12:58.870|24711] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 05:12:58.870|24711] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-cHxjMy
[2022-02-17 05:12:58.870|24711] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 05:48:39.589|24711] INFO: The utility succesfully executed.
[2022-02-17 05:48:39.797|24711] INFO: backup-archiver finished. Exit code: 0
...
[2022-02-17 05:55:39.132|26897] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 05:55:39.136|26897] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-B8A98d
[2022-02-17 05:55:39.140|26897] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 06:02:13.118|26897] INFO: The utility succesfully executed.
[2022-02-17 06:02:13.198|26897] INFO: backup-archiver finished. Exit code:
...
[2022-02-17 06:13:35.305|27896] INFO: backup-archiver started : /opt/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/qmail/mailnames/domain.com --destination=domains/domain.com/backup_domainmail_2202170022.tar --listing-file=/tmp/bmm2CnKKn --created-index-file=/tmp/bifl1vkXp --incremental-dependencies=/tmp/bdfPCFYhD --user=popuser --no-recursion --session-path=/opt/psa/PMM/sessions/2022-02-17-002120.451 --warnings=/tmp/bwsw1CiDQ --compression-method=zstd --compression-level=no-compression --id=46
[2022-02-17 06:13:35.314|27896] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2022-02-17 06:13:35.322|27896] INFO: Repository '/var/lib/psa/dumps': Initialized
[2022-02-17 06:13:35.346|27896] INFO: Calculate list of files to pack
[2022-02-17 06:13:38.687|27896] INFO: Repository '/var/qmail/mailnames/': Initializing...
[2022-02-17 06:13:38.695|27896] INFO: Repository '/var/qmail/mailnames/': Initialized
[2022-02-17 06:13:38.699|27896] INFO: Repository '/var/lib/psa/dumps': Pack from /var/qmail/mailnames/domain.com to domains/domain.com/backup_domainmail_2202170022.tar. Additional files count: 0. Split size: -1
[2022-02-17 06:13:38.771|27896] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 06:13:38.775|27896] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-7eCZ6O
[2022-02-17 06:13:38.779|27896] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 07:03:52.845|27896] INFO: RepositoryError Repository error: Unable to pack /var/qmail/mailnames/domain.com: Program has been terminated [common/plesk-utils/PMM/repository-transport/repository_utils.cpp:RepositoryError]
void plesk::CreateArchive(const string&, const plesk::PackSettings&, const plesk::UserInfo&, size_t (*)(const char*, size_t, void*), void*)
[2022-02-17 07:03:52.857|27896] INFO: backup-archiver finished. Exit code: 1
[2022-02-17 07:03:52.946|28650] ERR [panel] Backup has been stopped