Question Very long backup time for one specific subscription despite low file count and low disk space usage

[Bitpalast]

On one single subscription of one single server backup telemetry measures a very long gap of 3 h 30 m and/or duration for packing files.



However, the subscription has only 28,000 files in total and occupies only 9 GB of disk space uncompressed. 3 GB of that are .tar.gz files.

Question:
Why does this one subscription take so long to backup up despite the comparatively small number of files and small disk space usage? If I run a full backup on another subscription with the same size, it normally completes within less than an hour on the same server and on other servers. Only this very one subscription repeatedly causes very long delays.

Excerpt from backup log that shows the "gap":

backup.log:[2022-04-01 02:30:35.753|26500] INFO: backup-archiver started : /usr/local/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/www/vhosts/<subscription> --destination=clients/<user (owner)>/domains/<subscription>/backup_user-data_2204010202.tzst --listing-file=/tmp/bufoU0yOE --created-index-file=/tmp/bif7Jamw2 --incremental-dependencies=/tmp/bdfBlvOp8 --user=<username> --no-recursion --session-path=/usr/local/psa/PMM/sessions/2022-04-01-020026.389 --warnings=/tmp/bwstTwhje --compression-method=zstd --compression-level=normal

backup.log:[2022-04-01 02:30:36.145|26500] INFO: Repository '/home/dumps': Pack  from /var/www/vhosts/<subscription> to clients/<user (owner)>/domains/<subscription>/backup_user-data_2204010202.tzst. Additional files count: 0. Split size: -1

backup.log:[2022-04-01 06:03:24.599| 5195] INFO: backup-archiver started : /usr/local/psa/admin/sbin/backup-archiver --pack-incrementally --source=/var/www/vhosts/<subscription> --destination=clients/<user (owner)>/domains/<subscription>/backup_apache-files_2204010202.tzst --listing-file=/tmp/bafdT4IzO --created-index-file=/tmp/bifWVlYth --incremental-dependencies=/tmp/bdfrnZFEk --user=apache --no-recursion --session-path=/usr/local/psa/PMM/sessions/2022-04-01-020026.389 --warnings=/tmp/bwsNJMdJm --compression-method=zstd --compression-level=normal

The server has >330 subscriptions and none of the others are causing any notable delays in backups.

I just cannot figure out what is wrong with this specific subscription that it takes so long to complete. Anyone with any ideas?

 

[Maarten]

I would narrow the issue down by excluding folders within the subscription and run another backup until you've found the one that causes the delay.

 

[Bitpalast]

A manual full backup of the same, unchanged subscription finishes within 1' 35". Then I created a clone of the subscription into a new user and ran a manual backup there. That also finished fast.

The only remaining cause can be that while one backup is running, other backups start and the "Maximum number of simultaneously running scheduled backup processes" prevents an already started backup to continue when other backups are active and not finished yet. It should not be that way though. It should be a "first come, first serve" queue. What's happening here is that several customers have scheduled their backups somewhere between 2 am and 7 am. And these seem to break into a backup pause of the server backup that is running between 1 am and normally 9 am, forcing the pause onto it so that the backup that actually started first, the full server backup, finishes last, because it continues only after all the shorter customer subscription backups are completed.

I'll try to formulate this as a bug.

 

[Bitpalast]

Reported in

Forwarded to devs - If the "Maximum number of simultaneously running scheduled backup processes" is set, shorter backups that start after a longer backup are prioritized

Username: TITLE If the "Maximum number of simultaneously running scheduled backup processes" is set, shorter backups that start after a longer backup are prioritized PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE CentOS 7.9, Plesk Obsidian with latest MU PROBLEM DESCRIPTION...

talk.plesk.com