@AndreyKashin
Now, after the preliminary introduction in my previous post, let's work towards to answers to your questions.
This statement
Problem:
- It is impractical to store such archives locally in /var/lib/psa/dumps: a full backup of 1-1.5 TB is collected for more than a day, it heavily loads the CPU and MySQL, as a result, the process freezes.
- On other servers with volumes of 250-700 GB, everything works fine.
is a clear indication that your server has not sufficient resources.
A tiny upgrade of the server would suffice, since any 32GB / 64 GB RAM server with 3TB to 4TB disk space could easily handle all of your backup needs.
That is,
local backups.
Having said the above, the best, most cheap and most effective method is a server upgrade.
If you want more than local backups, then you still will be better of by upgrading your server, given the fact that this server (the source) needs to be able to generate the backups that are stored locally, remotely or locally + remotely.
Stated differently,
upgrade the server anyway, no matter what.
To be honest, I find it a bit problematic to give you the aformentioned advice, since the Plesk backup process is rather resource hungry.
One could consider
workarounds to offload the resource hungry Plesk backup processes, but there are not any.
Incremental backups are not a workaround and certainly not a solution - the disk space used by backups will increase dramatically over time (and resource shortages are a current problem already, so this problem will only become worse when the backups become larger).
This part of the (first) question
Questions:
1. Is it possible to use external proprietary storage on a separate VM (for example,
BorgBackup +
Borgwarehouse) for admin backups so that:
- did customers see them in the Plesk interface,
can only be answered with : yes and no.
Yes, it is possible, as long as Plesk has direct or indirect support for that external backup solution.
No, do not spend money on an external backup solution
if you do not need it,
given the fact that you need to upgrade your (source) server anyway and that upgraded server probably is sufficient to let Plesk backup processes work like a charm.
No, do not spend money on an external backup solution
if you want to have duplicates of your backups on a remote server :
a) rent a small VPS or Dedicated Server for a small fee per month, set it up as FTP server,
OR
b) use a cloud based storage solution that allows FTPS connections,
and, in both cases, use Plesk native Backup Manager to
- store backups locally, (and)
- store backups remotely via FTP, (and)
- setup a configuration that
only allows for daily backups on the source server,
and,
obviously,
- when using option b) : use snapshot functionality in the cloud, to make multiple copies of the backups,
- when using option a) : create a script on the target server to duplicate each daily backup, to make snapshots by copying backups to local directories on the target server, with these local directories not being remotely accessible.
Both option a) and option b) is a fairly simple backup solution that fits well in the Plesk eco-environment.
Naturally, more complex situations and more complex backup processes require more complicated solutions.
However, in your situation, it is really not necessary to make use of a complicated solution.
This part of the (first) question
- and at the same time such backups were not taken into account in the subscription disk space quotas?
is something that I should not know the answer to.
To be honest, I do not have an answer that I am willing to share.
The essence is that
- you should always want to include backup space in disk space
usage (
not quotas) if you are offering those services to customers,
AND
- you should not be interested in disk space
quotas at all, unless you are billing disk space - if you are billing and have the need to look at "quotas", then you should be aware of the fact that you should get a bigger server in order to not worry about quotas and to be able to bill more.
The second question
2. Are there official or unofficial best practices for organizing big data backups (1+ TB) in Plesk?
can be answered and I can recall that I have written some kind of a lengthy story about backups on this Plesk forum in the far away past.
I will try to find it.
The essence of that article is that one should
never opt for incremental backup, certainly not when backups are made of domains / subscriptions with active hosting and
certainly not when hosting is WordPress based.
It is highly recommended to
1 - choose the option
full backup,
2 - choose the options
local + remote backup via FTP(S),
3 - setup remote backup with a
FTPS connection,
4 -
never use SFTP, since that will cause havoc due to the nature of the protocol,
5 - to install and use the Scheduled Backup List extension, since that will make life a bit more easy,
6 - make sure that the backups are run at intervals that are long enough to allow other backup processes to finish first,
7 - run Plesk backups in the nightly hours or whenever traffic is lowest
and updates are not executed on the Plesk instance,
and the above is just a simple 101 set of recommendations, to keep it simple.
In addition, it must be added that - for the sake of security and with the objective of fast remote storage - it is not really recommended (read: not at all) to use Dropbox, OneDrive or Google Drive based backups.
The best advice that can be given is to keep it as simple as possible.
This will avoid
common mistakes.
One of the common mistakes is to activate a "full server backup" - this is necessary, but does not really add a lot of value.
A "full server backup" will backup all domains, subscriptions, user data, a small part of server config data (that you really do not need) - this is a lot to handle!
The problem is not the "full server backup" as a concept, the problem here is the way Plesk backup processes work - resource hungry!!!!
A good solution is to split the backups into smaller parts - the best result will be obtained by creating backups
per subscription.
When using backups on a subsription basis, one can use the Scheduled Backup List extension to nicely spread the backup processes - in an ideal situation, a subscription will be backed up during the night, followed by the backup of another subscription ..... until all subscriptions are backed up.
By using backups on a subscription basis, one can schedule a full backup on a very low frequence with low retention - once per month, one copy retained.
Reducing the number of "full server backups" will save a lot of disk space - this will allow for a higher retention of the smaller backups of the subscriptions.
Increasing the retention of the smaller backups of the subscriptions should also have a positive effect on performance.
The smaller backups of subscriptions are associated with a reduced memory usage (and database usage) - this will increase the probability that backups are finished within a reasonable period of time, that other processes (other than backups) can still continue and so on.
The smaller backups of subscriptions also allow for the full backup method, as opposed to the incremental backup method.
In essence, using backups on a subscription basis,
with
- full backup method selected,
- local backup storage activated,
- FTPS file transfer protocol selected,
will allow for smaller backups
per subscription, with
- fast creation and compression of the local backup file,
- fast and asynchroneous file transfer to the remote storage,
and
without the issues that are regularly associated with full server backup and incremental backup methods.
In addition, when using simple extensions like the Scheduled Backup List extension, backup processes can be scheduled and hence optimized.
Stated differently,
no need to think difficult, since for most backup scenario's a fair solution can be easily created with out-of-the-box Plesk tools.
I hope the above helps a tiny bit!
Kind regards....
Facebook
Twitter
