• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Issue Full Plesk backup issues - very slow

S

sneppa

New Pleskian
Hey guys,

I am running Plesk Obsidian 18.0.41 Update #1 on Ubuntu 18.04.6 LTS on a vServer with 6 cpus, 12gb ram and 500gb ssd.
Usually the full backup (~ 32gb) takes 6 hours but in the last week it took 12 hours and the last backup I had to cancel.
The backup is executed over night and now it causes issues in the morning (e.g. long page loads for users).

I made the following observations:
  • CPU and memory are low during backup, Load average is high
  • I/O is high, when a page get accessed iotop shows processes with IO between 100% and 800%
  • memory usage (in grafana) shows 100% cached, 2gb used
  • In the backup log there are some gaps (30 minutes up to an hour)
What I tried:
  • I thought very small files (like sessions, cached data) causes the issues, I excluded them
  • I turned off the compression
... but he issue still occures.

Today I tested a manual backup with some sites:
  • The backup of the first page was done in 4 minutes and had a size of ~3gb - no issues with page loads during that time
  • The backup of the second page took to long and caused issues with page loads during the backup - the real memory usage in grafana was 12gb again (only 2gb used) - I had to cancel the backup
I added some screenshots of iotop/htop/grafana and a edited/shortened log.

Does anyone have a suggestion on how to solve the issue?
I thank you in advance.

Code: 
...
[2022-02-17 01:10:02.471|  837] INFO: Repository '/var/lib/psa/dumps': Initialized
[2022-02-17 01:10:02.878|  837] INFO: Calculate list of files to pack
[2022-02-17 01:11:22.081|  837] INFO: Repository '/var/www/vhosts/': Initializing...
[2022-02-17 01:11:22.089|  837] INFO: Repository '/var/www/vhosts/': Initialized
[2022-02-17 01:11:22.093|  837] INFO: Repository '/var/lib/psa/dumps': Pack  from /var/www/vhosts/domain.com to domains/domain.com/backup_user-data_2202170022.tar. Additional files count: 0. Split size: -1
[2022-02-17 01:11:22.333|  837] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 01:11:22.341|  837] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-kMAh8E
[2022-02-17 01:11:22.341|  837] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 01:47:56.496|  837] INFO: The utility succesfully executed.
[2022-02-17 01:47:56.753|  837] INFO: backup-archiver finished. Exit code: 0
[2022-02-17 01:47:56.891| 6107] INFO: backup-archiver started : /opt/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/www/vhosts/domain.com --destination=domains/domain.com/backup_apache-files_2202170022.tar --listing-file=/tmp/bafezT0wN --created-index-file=/tmp/bif5iwCdL --incremental-dependencies=/tmp/bdfJobC0s --user=www-data --no-recursion --session-path=/opt/psa/PMM/sessions/2022-02-17-002120.451 --warnings=/tmp/bwsWj6BNa --compression-method=zstd --compression-level=no-compression
[2022-02-17 01:47:56.897| 6107] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2022-02-17 01:47:56.901| 6107] INFO: Repository '/var/lib/psa/dumps': Initialized
...
[2022-02-17 02:28:33.618|28690] INFO: Export file domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:28:33.630|28690] INFO: Create directory /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com
[2022-02-17 02:28:33.658|28690] INFO: Link /var/lib/psa/dumps/domains/domain.com/backup_user-data_2202170022.tar to /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:28:33.670|28690] INFO: Add file '/tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar' to archive as 'domains/domain.com/backup_user-data_2202170022.tar'
[2022-02-17 02:28:33.698|28690] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 02:45:21.553|28690] INFO: Export file domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:45:21.553|28690] INFO: Create directory /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com
[2022-02-17 02:45:21.553|28690] INFO: Link /var/lib/psa/dumps/domains/domain.com/backup_user-data_2202170022.tar to /tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar
[2022-02-17 02:45:21.557|28690] INFO: Add file '/tmp/pmm-de-tmp-repo-aB5JpO/domains/domain.com/backup_user-data_2202170022.tar' to archive as 'domains/domain.com/backup_user-data_2202170022.tar'
[2022-02-17 02:45:21.557|28690] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 02:45:21.557|28690] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --directory /tmp/pmm-de-tmp-repo-aB5JpO --dereference --blocking-factor 1 --no-recursion --files-from /tmp/pmm-taw-af-includes-SSzCDh
[2022-02-17 03:14:50.735|15384] INFO: The utility succesfully executed.
[2022-02-17 03:14:51.015|15384] INFO: backup-archiver finished. Exit code: 0
...
[2022-02-17 05:12:52.041|24711] INFO: backup-archiver started : /opt/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/qmail/mailnames/domain.com --destination=domains/domain.com/backup_domainmail_2202170022.tar --listing-file=/tmp/bmmuLi6sg --created-index-file=/tmp/bifgfR9nn --incremental-dependencies=/tmp/bdfQL8AOe --user=popuser --no-recursion --session-path=/opt/psa/PMM/sessions/2022-02-17-002120.451 --warnings=/tmp/bwsFeF2e6 --compression-method=zstd --compression-level=no-compression --id=33
[2022-02-17 05:12:52.042|24711] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2022-02-17 05:12:52.042|24711] INFO: Repository '/var/lib/psa/dumps': Initialized
[2022-02-17 05:12:52.043|24711] INFO: Calculate list of files to pack
[2022-02-17 05:12:58.783|24711] INFO: Repository '/var/qmail/mailnames/': Initializing...
[2022-02-17 05:12:58.787|24711] INFO: Repository '/var/qmail/mailnames/': Initialized
[2022-02-17 05:12:58.787|24711] INFO: Repository '/var/lib/psa/dumps': Pack  from /var/qmail/mailnames/domain.com to domains/domain.com/backup_domainmail_2202170022.tar. Additional files count: 0. Split size: -1
[2022-02-17 05:12:58.870|24711] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 05:12:58.870|24711] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-cHxjMy
[2022-02-17 05:12:58.870|24711] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 05:48:39.589|24711] INFO: The utility succesfully executed.
[2022-02-17 05:48:39.797|24711] INFO: backup-archiver finished. Exit code: 0
...
[2022-02-17 05:55:39.132|26897] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 05:55:39.136|26897] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-B8A98d
[2022-02-17 05:55:39.140|26897] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 06:02:13.118|26897] INFO: The utility succesfully executed.
[2022-02-17 06:02:13.198|26897] INFO: backup-archiver finished. Exit code:
...
[2022-02-17 06:13:35.305|27896] INFO: backup-archiver started : /opt/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/qmail/mailnames/domain.com --destination=domains/domain.com/backup_domainmail_2202170022.tar --listing-file=/tmp/bmm2CnKKn --created-index-file=/tmp/bifl1vkXp --incremental-dependencies=/tmp/bdfPCFYhD --user=popuser --no-recursion --session-path=/opt/psa/PMM/sessions/2022-02-17-002120.451 --warnings=/tmp/bwsw1CiDQ --compression-method=zstd --compression-level=no-compression --id=46
[2022-02-17 06:13:35.314|27896] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2022-02-17 06:13:35.322|27896] INFO: Repository '/var/lib/psa/dumps': Initialized
[2022-02-17 06:13:35.346|27896] INFO: Calculate list of files to pack
[2022-02-17 06:13:38.687|27896] INFO: Repository '/var/qmail/mailnames/': Initializing...
[2022-02-17 06:13:38.695|27896] INFO: Repository '/var/qmail/mailnames/': Initialized
[2022-02-17 06:13:38.699|27896] INFO: Repository '/var/lib/psa/dumps': Pack  from /var/qmail/mailnames/domain.com to domains/domain.com/backup_domainmail_2202170022.tar. Additional files count: 0. Split size: -1
[2022-02-17 06:13:38.771|27896] INFO: ENV[LANG]=en_US.UTF-8
[2022-02-17 06:13:38.775|27896] INFO: Executing utility: /usr/lib/plesk-9.0/sw-tar --create --file - --ignore-failed-read --warning no-file-changed --no-recursion --verbatim-files-from --anchored --files-from /tmp/pmm-ru-ca-includes-7eCZ6O
[2022-02-17 06:13:38.779|27896] INFO: Load settings from /opt/psa/admin/conf/domain.com
[2022-02-17 07:03:52.845|27896] INFO: RepositoryError Repository error: Unable to pack /var/qmail/mailnames/domain.com: Program has been terminated [common/plesk-utils/PMM/repository-transport/repository_utils.cpp:RepositoryError]
void plesk::CreateArchive(const string&, const plesk::PackSettings&, const plesk::UserInfo&, size_t (*)(const char*, size_t, void*), void*)
[2022-02-17 07:03:52.857|27896] INFO: backup-archiver finished. Exit code: 1
[2022-02-17 07:03:52.946|28650] ERR [panel] Backup has been stopped
 

Attachments

  • iotop.PNG
    iotop.PNG
    140.8 KB · Views: 18
  • iotop-page-loading.PNG
    iotop-page-loading.PNG
    153.5 KB · Views: 14
  • htop.PNG
    htop.PNG
    171.1 KB · Views: 17
  • monitoring-memory.PNG
    monitoring-memory.PNG
    90.3 KB · Views: 16
  • monitoring-network.PNG
    monitoring-network.PNG
    35.5 KB · Views: 15
  • monitorung-cpu.PNG
    monitorung-cpu.PNG
    109 KB · Views: 17
Please install the free "Backup Telemetry" extension. Then you can descend into the backup transaction tree and easily see where the backup spends most of the time. Very often backups are slow due to a high number of files in some subdirectory.
 
Hey Peter,

I installed the Backup Telemetry extension yesterday morning et voilà here are some results:
  • Back up emails domain 1: 2h 26m 23s (8708.63 MB)
  • Back up webspace domain 2: 50m 37s (929.92 MB)
  • Back up webspace domain 1: 41m 11s (1504.79 MB)
  • Back up emails domain 3: 26m 24s (1528.03 MB)
  • Back up webspace domain 4: 16m 44s (302.02 MB)
  • Back up emails domain 4: 16m 29s (344.56 MB)
  • ...
I had to cancel the backup due high load average again and disabled the email backup so that at least the website data is backed up tomorrow.

I think the issue is caused by the buffered files in the memory (tar uses too much memory for its buffer - workaround?).
I tried to tar the emails of domain 1 manually.
When I tar the first mailbox (3.4gb) everything is fine.
When I tar the next mailbox (1.5gb) the memory buffer gets full and load average gets high.
Similar issue (not with plesk): running tar completely locks up mysql

Imo the possible solutions are:
  • upgrade the server memory
  • not use plesk backup and use a backup software which is using e.g. ionice (but I like the plesk backup...)
I think there is a better solution...
I'm a little bit suprised that a 30-40gb backup can cause such issues on a server with this configuration...
One of my other servers backups 15gb of data in 45min incl. ftp transfer and has only 2gb of memory... strange...
 
The niceness can be set on Tools & Settings > Backup Manager > Backup Settings.

A high load that causes the server to slow down is normally not caused by tar, but by the compression applied. Maybe you can test the backup with disabled compression. Compression level can be set in "Backup Settings", too.

The long runtime of the backup is due to a large number of individual files. I can imagine that many emails exist in your mailbox(es). Each mail is stored in a single file. Many files cause many tar transactions and hdd accesses.

A high I/O load on the disk is normal, because thousands, maybe - depending on the actual number - even millions of files need to be read.
 
Peter Debik said:
The niceness can be set on Tools & Settings > Backup Manager > Backup Settings.

A high load that causes the server to slow down is normally not caused by tar, but by the compression applied. Maybe you can test the backup with disabled compression. Compression level can be set in "Backup Settings", too.

The long runtime of the backup is due to a large number of individual files. I can imagine that many emails exist in your mailbox(es). Each mail is stored in a single file. Many files cause many tar transactions and hdd accesses.

A high I/O load on the disk is normal, because thousands, maybe - depending on the actual number - even millions of files need to be read.
Click to expand...
Hi Peter,
I have the same problem... after the upgrade to the last plesk version.
The last time it needed 55 hours.... without compression.
Services was very very slow and often unreachables.
Logs attached.

1657549352229.png


1657549312730.png

1657549515750.png
 
You can click on the Telemetry entries to descend into that tree to find out where the backup takes so long.
 
You must log in or register to reply here.

Similar threads

W
Issue backup task error | Unable to save statistics. Error: DB query failed: SQLSTATE[HY000] [2002] No such file or directory
Replies
2
Views
769
Wolfgang Reidlinger
W
HoracioS
Resolved Backup error
Replies
4
Views
834
HoracioS
HoracioS
S
Resolved Warning: Unable to back up files
Replies
4
Views
631
smilan
S
M
Issue File dump-header not found on first position in archive.
Replies
0
Views
743
massin
M
L
Question Manual Backup with pmm-ras won`t stop the process when finished.
Replies
2
Views
813
ldoerrdotcom
L
Back
Top