Question Backup up the whole server

[Denis Gomes Franco]

So, I bought the REMOTE STORAGE extensions thinking it would be a breeze to set up backing up my accounts on Amazon S3. Looks like it's not the case.

On the first day of the scheduled backup everything crashed and the server disk was full. I just found out on another thread that Plesk has to create a local backup first, so that means the server has to have more than 50% free disk space (which is not my case) in order to complete it. Looks like my backup crashed because space ran out and Plesk didn't delete the partial backup files afterwards.

Well, I bought the remote storage extension *precisely* because I wanted to offload my backup, not store them locally. So I currently have no means to back up my clients' data. Also, it isn't feasible to have a server with double the disk space just to have backups being stored somewhere else.

So, any ideas on how to make this work?

I thought Plesk would do sequential backups, eg., generate a backup for the first subscription, upload to S3, delete the temporary files, generate a backup for the second subscription, upload, delete, generate, upload, delete....

By the way, never had any issues like that when using Updraft plugin for Wordpress when backing up to Amazon S3. Updraft ran perfectly even on 10 GB sites and didn't clog the server. I just thought that now I'm using Plesk it would be easier to just set up server-wide backups instead of installing and configuring a plugin on each and every site.

 

[garcue]

So, I bought the REMOTE STORAGE extensions thinking it would be a breeze to set up backing up my accounts on Amazon S3. Looks like it's not the case.

On the first day of the scheduled backup everything crashed and the server disk was full. I just found out on another thread that Plesk has to create a local backup first, so that means the server has to have more than 50% free disk space (which is not my case) in order to complete it. Looks like my backup crashed because space ran out and Plesk didn't delete the partial backup files afterwards.

Well, I bought the remote storage extension *precisely* because I wanted to offload my backup, not store them locally. So I currently have no means to back up my clients' data. Also, it isn't feasible to have a server with double the disk space just to have backups being stored somewhere else.

So, any ideas on how to make this work?

I thought Plesk would do sequential backups, eg., generate a backup for the first subscription, upload to S3, delete the temporary files, generate a backup for the second subscription, upload, delete, generate, upload, delete....

By the way, never had any issues like that when using Updraft plugin for Wordpress when backing up to Amazon S3. Updraft ran perfectly even on 10 GB sites and didn't clog the server. I just thought that now I'm using Plesk it would be easier to just set up server-wide backups instead of installing and configuring a plugin on each and every site.

Hello Denis Gomes Franco,

You have some option similar to "Save backups in server storage in case of an FTP error."?

This would prevent the backup files from being saved locally in case of error.

 

[trialotto]

@Denis Gomes Franco

I have read your post and I must first say that Updraft is not a feasible option at all: it is not a very secure solution AND it requires to use less secure configuration for Plesk.

It seems to be the case that you did a full server backup, or am I mistaken?

If I am not mistaken, you should be aware that it is always better to use a backup schedule that backups subscriptions, hence requiring less volume.

The proper configuration of scheduled backups would be that you

- would have at least 15 mins between the back of each subscription (and 30 mins if the subcription is relatively large, compared to the time to create and transfer the backup)
- only use full backups, since incremental backups can require more space than individual backups,
- retain the backups during a limited period of time (for instance, 3 to 5 full daily backups are sufficient for your purposes)

and that would really not burden your disk space.

Finally note that you can configure the backup manager to only make a backup in case that sufficient disk space is present: this will prevent any problems to occur.

Hope the above helps a bit in solving your challenge and/or issue.

Kind regards..........

 

[garcue]

Denis Gomes Franco,

Do you really find it more interesting to save your backups on an FTP server?

Plesk is ready for it. There are suppliers that provide very interesting infrastructures. Plesk has just released an extension that allows you to connect to SFTP servers so you will gain a layer of security management.

 

[Denis Gomes Franco]

You have some option similar to "Save backups in server storage in case of an FTP error."?

Now that you mentioned it, I saw that this option is enabled. But I am not backing up to FTP, I've set up my Amazon S3 account. If that option also applies to backups transferred to S3, then the wording should be changed. Should I disable it then?

Updraft is not a feasible option at all

Updraft ran very smoothly for the past several months, even for very large sites. I greatly recommend this plugin. But whenever possible I would like to use server-level functions instead of plugins.

It seems to be the case that you did a full server backup

Yes, I set it up via the Backup option under Tools & Settings, but not under each subscription. Is that correct? Setting it up under each subscription makes any difference?

Finally note that you can configure the backup manager to only make a backup in case that sufficient disk space is present

That option is already enabled but I was really expecting Plesk to just backup one subscription after another to avoid clogging up the server with backup files *during* the process.

to save your backups on an FTP server

AWS S3 seems like a better option for me. I'd like to have 14-day backups, my 80+ sites amount to 56 GB, that means I would need around 750 GB to do it (without using incremental backups). That would cost me a measly 17 dollars, even less if I use their "infrequent access" or "Glacier" options in exchange for a longer access time.

 

[Denis Gomes Franco]

Right now I noticed one thing: backup manager settings can be different per subscription. Just opened a subscription's settings and it didn't have my S3 credentials. So it seems that I will have to enable backups on a subscription basis, as it seems there is no way to set this up on a service plan or batch configure this.

 

[trialotto]

@Denis Gomes Franco

I will provide some quotes and respond to the quotes, for the sake of clarity.

Now that you mentioned it, I saw that this option is enabled. But I am not backing up to FTP, I've set up my Amazon S3 account. If that option also applies to backups transferred to S3, then the wording should be changed. Should I disable it then?

No, not really. Disabling that option would mean that FTP failure results in the loss of the already created backup, which is stored in server storage.

Nevertheless, even though a (temporary) FTP failure is quite common, your available disk space might require that you disable the option.

In general, the disablement of this option is not recommended, so let's first try to get your backup procedure running without disabling this option.

Yes, I set it up via the Backup option under Tools & Settings, but not under each subscription. Is that correct? Setting it up under each subscription makes any difference?

There are some differences and the most important ones are:

- the backup under "Tools & Settings" is a full server backup that will require lots of disk space, hence explaining your (current) issues with disk space
- the backup by means of "Subscriptions" is a partial backup that does not use much disk space during backup

It is important to note the advantages and disadvantages of backups at the various levels, so let me take the level of subscription based backup as an example:

- advantage: the amount of disk space used is very small, while all information regarding the subscription is still being backed up
- advantage: disk space used during backup time will be available after the backup has completed succesfully
- disadvantage: server wide settings are not included
- disadvantage: it can be rather tedious to setup subscription based backup if you have lots of subscriptions

The disadvantages of subscription based backups are minor though, given that (one the hand) the server wide settings included in full server backups are not extensive and certainly not complete (read: it has no value at all to backup these settings) and (on the other hand) an extension like Scheduled Backups can make tasks more easy.

In short, just cherish the advantages of subscription based backups, since the disadvantages are (very) minor.

That option is already enabled but I was really expecting Plesk to just backup one subscription after another to avoid clogging up the server with backup files *during* the process.

No, when having a full server backup, Plesk simply adds all data to one big backup file......... and that can be very problematic.

Not only is the backup file huge (with disk space related issues, as you have experienced), but also other (serious) problems can occur: for instance, Plesk backup manager is using pigz as the underlying tool for creating a backup file and pigz is using all available CPU power (read: that is remaining when running other processes) and, as a result, running a full server backup can make a huge hit on CPU performance and/or disk I/O (read: both of them are bad).

In general, a full server backup is not recommended on small servers and it is never recommended to run backup procedures during day time!

AWS S3 seems like a better option for me. I'd like to have 14-day backups, my 80+ sites amount to 56 GB, that means I would need around 750 GB to do it (without using incremental backups). That would cost me a measly 17 dollars, even less if I use their "infrequent access" or "Glacier" options in exchange for a longer access time.

This sounds like a proper idea, but the number of backups to retain is a bit on the high side: a number lower than 14 will certainly suffice when using full backups.

In the case of incremental backups, one would really need to retain backups for a longer time interval.

In the case of full backups........there is no need to do so, unless you want to be able to restore data from a previous period.

For the sake of backup alone, 2 to 3 backups are sufficient: after all, if just wanting to backup data, the data present in the previous backup will also be available in the next backup and so on......... so it is essentially "adding the new to the old" and that does not require a repetition of 14 times, does it?

For the sake of having a restore point, which is essentially different when compared to a backup, just create a restore point when needed: for instance, when launching a new version of an application and/or when updating an application, just create a manual backup for safety purposes.

I hope the above helps and explains a bit.

Kind regards.........

 

[Denis Gomes Franco]

Right trialotto, thanks for the explanations, understood everything. I'll go with subscription backups, it looks better.

a number lower than 14 will certainly suffice

I eventually want to go up to 30 days. I am not running a hosting business, I'm more of a web design agency who happens to provide hosting services and having so many backups is essential due to the possibility that some client might ask 'Hey, I changed something three weeks ago, can you revert that for me?' Believe me, it happens.

 

[trialotto]

I eventually want to go up to 30 days. I am not running a hosting business, I'm more of a web design agency who happens to provide hosting services and having so many backups is essential due to the possibility that some client might ask 'Hey, I changed something three weeks ago, can you revert that for me?' Believe me, it happens.

@Denis Gomes Franco

In that case, you probably are better off by realising the following combination of (types of backup):

a) regular, remote backups (via Plesk) with a retention rate of 2 to 3 days, (and)

b) local or FTP based remote backups, only storing a versioned history of changed files: this can be easily done with cronjob that runs a backup script (read: bash script) that finds changed files and saves them for a specific period of time, using a file and directory structure that has a naming convention that includes date/timestamps in names.

Note that second part, being point b, sounds like an "incremental backup", but there is a slight difference: all files are being kept and files with changes will not be overwritten, as can be (and often is) the case with incremental backup.

Also note that the second part can be based upon very fast tools like rsync and/or not that you can extend the second part to store a more fine-grained (versioned) history of changed files in the cloud, in a cloud based storage environment (such as S3).

Maybe this "combined setup" better suits your needs?

Kind regards......

 

[Denis Gomes Franco]

Thanks trialotto, I'll consider your points.

In the meantime, do you know if I can change Backup Manager settings via CLI?

 

[SteveITS]

It's a little more complex to set up than a simple rsync, but we us rdiff-backup to back up an entire server. It uses a combination of rsync and diff to maintain versioning.

You do have to be able to run it on the remote server so it doesn't work with say FTP. We set it up to run overnight on the backup server and connect to each VPS to be backed up, and the login runs rdiff-backup as the shell so they can connect. There are a variety of how-to docs online.

 

[Denis Gomes Franco]

@SteveITS: thanks for the tips. The subscription backup will suffice for now. It ran just fine this morning, I've set it up on 6 subscriptions just as a test.

But now there is one thing, which certainly is feedback for the Plesk team: when I open a subscription and then click on the Backup Manager link, it shows all backups from all subscriptions instead of backups from just the subscription I opened. I can only know which backup is which when hovering the mouse and reading the link.

Filtering seems to work so I can find a backup by typing the domain name, but I think it should already display only backups from that subscription, or at least show the account/domain name in the list alongside with the comments column.

 

[trialotto]

@Denis Gomes Franco

This part

Filtering seems to work so I can find a backup by typing the domain name, but I think it should already display only backups from that subscription, or at least show the account/domain name in the list alongside with the comments column.

I really do like ............... and it is an inconvenience for a long time now.

However, there is not really something that you can do about that, when using one and the same account: the account is used to list file belonging to that account.

In general, you can work around this issue by creating many accounts ........... but that is not really practical, unless one is using FTP.

I personal believe that there is a solution in caching file listing and filter them when reading that cache, but that is particularly hard to implement.

So, it is indeed a valid request to Plesk Team.......a request entailing that some filter of backup files exists, since it is not very secure that everyone sees all the backup files.

Regards.......

 

[Denis Gomes Franco]

since it is not very secure that everyone sees all the backup files.

I think I found what's going on: Plesk Backup Manager shows backups from different subscriptions

Looks like Backup Manager displays all files inside the backup storage location, whether they are from that user or not. Article says the solution is to set up backup using different folders, eg. I set it up to store it all under /plesk/ but will need to use /plesk/domainname.com/

 

[Denis Gomes Franco]

By the way, Updraft never had any issues with this, I set it up to backup everything under the same folder and still the plugin only displays backups originated from that website.

I think there is no reason for Plesk *not to* do the same, because Plesk is generating files named backup_domainname.com_<backupdate>.tar.

I just submitted an idea on Uservoice, let's see what happens. Also, I created those folders and moved the respective files there, and now it works as expected.

 

[trialotto]

@Denis Gomes Franco

I am aware of the KB article, but that article is essentially flawed.

It is based on the premise that one FTP account, with access to a specific directory, can travel through all subdirectories and/or create subdirectories.

The major problem with this dirty work-around is that the same FTP account can still be used to see and/or even retrieve backups of all domains in the directories associated with or created by that one particular account - not good at all!

In short, the KB article is a work-around and disables visibility of some part of the backups, but it is not a solution to a general security issue with FTP based backups.

For that reason, it is also never recommended to use the default Plesk account, associated with a domain/subscription: this default Plesk account is not only an account for Plesk and/or the Plesk Panel, it is also a FTP account, meaning that FTP hacking can make your Plesk Panel vulnerable.

In addition, when using a Plesk based domain for FTP backups, some of the directories or subdirectories used for FTP based backup might be openly accessible.

In conclusion, I would personally prefer to

- create separate FTP accounts (read: different from the default Plesk account) for each domain/subscription to be backed up, (and)
- create separate subdirectories on a per-subscription basis, (and)
- use a subdirectory created with the permissions 700 (or 740) and ownership of <default plesk account>:root (read: change the group ownership to root)

and that is not creating a 100% safety, but the above will create some additional security.

Hope the above helps a bit.

Kind regards.........

 

[Denis Gomes Franco]

@trialotto

I understand there might be security issues with FTP-based backups, but please note that I am storing them on Amazon S3. I have set up separate folders for backing up each domain, and now it works as intended. Each domain now only sees its own backups.

By the way, DigitalOcean Spaces also seem like a nice alternative, even price-wise.

 

[trialotto]

@Denis Gomes Franco

That is not what I intended to notify you about.

You are using one key or FTP account for Amazon S3 ...... and the directories might help you making certain backups invisible, but with that one key/account there is a (very likely) risk that all backups are accessible. The subtle nuance is in "visible" versus "accessible".

In essence, working with (key based or FTP) accounts, each with their own (limited) access to one directory, often works best and is (to some extent more) secure.

Regards.......

 

[Denis Gomes Franco]

Got it. Thanks!