Resolved SFTP backup manager

[fabieng]

Hello
I noticed that when the scheduled backup manager is on local system, each subscription / website has access to the backups.
but if the scheduled backup manager is configurated with SFTP, then, only the plesk admin can restore some subsciption with files. there is no backup available from the subscription / web.

how can we solve that, did I miss some configuration somewhere ?

thx !

 

[trialotto]

Hello
I noticed that when the scheduled backup manager is on local system, each subscription / website has access to the backups.
but if the scheduled backup manager is configurated with SFTP, then, only the plesk admin can restore some subsciption with files. there is no backup available from the subscription / web.

how can we solve that, did I miss some configuration somewhere ?

thx !

@fabieng

I do not think that you missed something.

The whole problem with SFTP is that you need root privileges - and that is something that you should not want to share with and/or make accessible to customers.

To a large degree, what you are observing is (on the one hand) expected or intended behavior and (on the other hand) good practice.

In short, no need to solve this particular "challenge".

Kind regards.........

 

[fabieng]

why root if SFTP is configurated with distant cedentials ? this is not related to local user in plesk as it is configurated at global level.

then, why is there difference between local / distant storage ?

using local storage, we can do restore from domain directly
using distant storage, we cannot do anymore ?

there is no reason to do that, even in admin mode ...

 

[trialotto]

why root if SFTP is configurated with distant cedentials ? this is not related to local user in plesk as it is configurated at global level.

then, why is there difference between local / distant storage ?

using local storage, we can do restore from domain directly
using distant storage, we cannot do anymore ?

there is no reason to do that, even in admin mode ...

@fabieng

To do what? To what refers your "that"? To using remote storage or to using local storage?

Anyway, it is good practice to have a local and remote backup :

- the local backup is primarily easy in terms of recovering backups fast : however, when your disk fails, local recovery is very likely to fail too, (and)
- the remote backup is primarily intended to remove single points of failure : backup integrity is not dependent on local disk failure anymore.

Hope the above explains a bit.

Regards......

 

[fabieng]

yes, you're right, it is a good practice to have both storage, BUT for server capacity reason, this is not always possible to use local storage.

I will try to explain better :

So, first, I used local storage as backup. then, when needed, I would able to recover files directly from domain tab. and customer as well from their subscription access.

But, then, for 1st reason, because of storage, I was not able anymore to have backup on local server.

Then, backups are now via SFTP. no other change (incremental / full / daily etc)

and now, customer are not able to restore themself their backup as the SFTP backups are not available from customer panel control.

"that" is refereing to the method of storing the backup is preventing the capacity for customer to restore their own backup. this is an issue as it's limiting customer autonomy.

 

[trialotto]

@fabieng

I understand the explanation and the challenges - but I think your challenges will be solved by using FTPS based backup : just choose FTP as backup method and select the "Use Passive Mode" and "Use FTPS" checkboxes.

The above mentioned backup method is working perfectly - that is not something I can say about SFTP (read: security considerations) or cloud (read: not working properly, at least in most cases there are many disadvantages) based backup methods.

A major advantage of FTP(S) based backups is that you (as a sysadmin of the external FTP server) can have full control over ftp users, ftp domains and ftp access rights, whereas your customers still have full access to and control over their backups.

A minor disadvantage of FTP(S) based backups is that you have or will encounter issues with transfer speeds, certainly when having a small backbone (less than 1Gbps).

In short, it just changing the letter "S" - from SFTP to FTPS!

Kind regards.........

PS As a somewhat more advanced alternative, you can always store backups locally and rsync the backup files to a remote storage/backup server with a script that runs on a daily basis. This will allow you to have full benefits of speed, as associated with local backups, whilst still maintaining the safety of remote storage of backups. Nevertheless, I would strongly recommend the FTP(S) based backups, for specific reasons that are not mentioned here - for the sake of convenience.

 

[ChristophRo]

Why not mount that SFTP folder (via sshfs utility) on your server and then use local backup method again?

 

[fabieng]

@trialotto : I use SFTP because my distant server does not have ftp server installed. This is why I use FTP over SSH as it is already secured. So, I cannot use FTPS for now. the means and position of the S letter does matter a bit

I cannot implement your alternative solution because rsync backup files would means having space on local server, capacity that I don't have anymore

@ChristophRo , yes, you're right, this is an alternative, maybe using nfs system.

But again, all those solutions are workaround. I really do not understand why storing the backup with a SFTP protocol will have different effect than FTP one... really strange behaviour.

 

[trialotto]

Why not mount that SFTP folder (via sshfs utility) on your server and then use local backup method again?

You are kidding?

Mount a SFTP folder??? That is contradictory : you mount a (parts of a) remote filesystem on a local filesystem - one does not mount an SFTP folder.

Mount a SFTP folder??? That is odd : you are suggesting to create (some kind of) a distributed filesystem to ..... - yes, to do what? Makes no sense at all.

Really, if one is getting something advanced for something simple as creating backups ....... mounting remote filesystems on the local filesystem is not recommended.

You can try, but you will not the problems when your network is down, if your traffic has a hackup, when you try to restore compromised backups and so on.

And, please, sshfs is not an utility or alternative, if one can use rsync or even SFTP - sshfs is not even an alternative for any proper distributed filesystem out there.

All of the above is just some information, intended as feedback that will allow you to think critical of the solutions you suggest and you probably want to apply somewhere.

Regards........

 

[trialotto]

@fabieng

This statement

I cannot implement your alternative solution because rsync backup files would means having space on local server, capacity that I don't have anymore

is not correct - rsync just simply is the copy command for file transfer between servers (simply summarized) : on the local server, no extra space is required.

A small note : any backup will make use of additional disk space, remote backups will use local disk space temporarily and local backups will use local disk space permanently.

In contrast, rsync based backup methods are essentially not using additional local storage space at all - you miss out on the backup of databases though, but that can be easily solved by creating a database dump (like Plesk backup does) and move that dump file with rsync to a remote server.

This statement

This is why I use FTP over SSH as it is already secured.

is also not correct : SFTP or file transfer over SSH is not secure by itself - in general, it is not safe at all.

In a twist of pure irony, both rsync and FTPS are considered to be more safe with a standard configuration - SFTP has to be configured properly and even then, it is not safe.

It is comparable to driving fast in a safe car, because "the car is safe" - that is a wrong method of thinking.

If you would ask me, I would rather have all ports blocked except for FTP than having all ports blocked except for SSH related ports - any form of SSH based transfer of files would require SSH related ports to be open........... the ports that are relevant for root access and hence the ports that you have to secure properly.

Without any proper security configuration, SFTP is just as unsecure as any other method of file transfer - with this exception : all other methods of file transfer will not make your command and control SSH ports open and vulnerable.

Just a line of reasoning............maybe interesting to take that into account.

Kind regards........

 

[fabieng]

I appreciate your security advices, but as said, ssh is already secured and, again, this is why I use it, do not worry about. but this is not really the purpose of the question.

The matter is that using this protocol, end customer are losing some advantages, as they cannot recover by themselves their own backup. and honestly, this should not appear, whatever the protocol or the backup method.

 

[trialotto]

@fabieng

This statement

The matter is that using this protocol, end customer are losing some advantages, as they cannot recover by themselves their own backup. and honestly, this should not appear, whatever the protocol or the backup method.

is fairly odd : the rsync based solution is the only alternative that you have, if and as long you have no FTP server on the remote server.

That is, an alternative that ticks all the boxes of your requirements.

To be honest, you would be better off to install a FTP server at the remote server - if you have root acces, you can also install a FTP server.

Kind regards......

PS You are free to choose whatever solution suits you best, but whatever backup method you wil choose - you will end up exhausting disk space on the local server, which is the basic reason that you started with this entire thread : you are running out of disk space, giving you the impression that you need some fancy solution to create a remote backup - which ironically also uses (a lot of) disk space, before the backup file is send to the remote server. That is why I called it a "fancy solution" - during every backup or even a file transfer, some additional disk space will be used - disk space that you are in shortage of. I really think that creating additional problems -with (on the one hand) a SFTP client that is not very stable or mature and (on the other hand) a backup method that consumes disk space that you do not have- is not a good solution at all.

 

[fabieng]

ok I will try to understand better your rsync solution....

here is the configuration :
- daily backup
- incremental backup
- full backup weekly
- retention : 4 weeks

>> Issue is that I do not have anough space to store 1 month of backup, this is why I use SFTP now.

So let's say I will rollback on local storage and set up some rsync script between my plesk serveur and my distant server. In my local storage, I can keep backup only for 1 week. in my distant server, I can keep 1 month of backup, no pb with that.

then, if my capacity is only one week on local storage, in backup manager, whatever the rsync copy and retention in my distant server, how can I recover files from 2 weeks ago, if those backups are not in plesk backup manager ?! or I missed some part of your explanation.

 

[trialotto]

@fabieng

In response to your last questions, I have to say that I can understand your questions - but I also have to place them into a different context.

The proper context for local versus remote backups is in your case :

- local storage : actual backup
- remote storage : static backups of the local backup files, as present on the local server

In essence, the context is that the remote storage is intended only to keep remote copies of local backup files, in case the local disks (or the local server) are failing.

The before mentioned context implies that you

- can just run a simple cron script on the local server to transfer existing local backups to a remote server on a daily basis,
- do not have any additional disk space usage on the local server - the local backups to be transferred elsewhere are already present and are rotated every x days,
- can just rsync local backup files to a remote server - and optionally run a daily cron script to delete files on the remote server that are older than y days.

Note the difference between x and y days, the values can be chosen to your own liking - in accordance to the retention policy that you want to achieve.

Restoring is as simple as rsyncing the remote backup files (on the remote server) to the local server - just put them in /var/lib/psa/dumps - Plesk will recognize them.

Nevertheless, restoring would also require that you

1 - move the existing backup files on the local server to a separate directory : just protect them!

OR

2 - that you make some manual changes to specific files related to the local Plesk backups.

Method 1 is the safe and easy option, whereas method 2 is the complex and thorough option - anyway, Plesk requires that the Plesk Backup Manager can read from one or more specific files, containing a sort of index of the various compressed files that compose the backup.

Method 1 will prevent a complex restore method and you can move the backup files back after the restore process.

Method 1 and 2 both require some careful processing of backup files........ so there is always a more easy solution.

For instance, when using the rsync method, one can also simply copy all files from the directories related to mail, the domains and/or the databases - this is not equal to rsyncing static and compressed backup files, this is more equivalent to rsyncing changed files only - which is quite convenient!

Nevertheless, when choosing any rsync based method with compressed backup files, one has to be aware that one mimicks the default migration process.

In short, the most easy solution would be installing a second Plesk instance and just running Plesk Migration Manager on a daily basis.

The latter solution also has the advantage that you would have a live domain whenever your main server fails to work - you are creating some minimal HA setup.

In conclusion, there are many solutions to solve your current challenge related to (lack of) disk space and the desire to backup domains or subscriptions remotely.

To be honest, I would simply say -entering again in the same circle of reasoning as before : just install a FTP server on the remote server! Easy, quick and painless.

Hope the above helps a (tiny) bit.

Kind regards........

 

[fabieng]

ok, then basically, you confirm that the different method are always ending in same point. the customer won't be able to restore a backup without admin help.

the only solution is using ftps. I will try to install on my distant server today to do more tests.

 

[trialotto]

@fabieng

The answer to

ok, then basically, you confirm that the different method are always ending in same point. the customer won't be able to restore a backup without admin help.

is "yes, almost every time" AND "no, not really in the case of the rsync based backup method" - after all, when putting back specific backup files (from the remote server to the local server), it will (on the one hand) be visible for and hence restoreable by customers and (on the other hand) your already existing local backup files remain visible.

Nevertheless, FTPS based backup is the most easy and reliable solution - it is a very easy and proven concept in the current Plesk eco-environment.

For that reason I added the the line "just install a FTP server on the remote server".

Regards.......

 

[fabieng]

ok I installed FTP server in my distant server. I changed configuration in plesk in order to do the backup in SFTP (passive mode). I ran a manuall backup from backup manager.
then, I went into domains >> backup manager >> and again, same issue, I only see the backup done at local level. I do not see the latest backup done via SFTP.
using SFTP does not change anything. Same issue than FTP over SSh.

moreover, I see in backup manager from domain tab that ftp is not configurated while it is at global server configuration level.

This means that again, using FTP method, if I do not configure manually for all domain, the FTP credentials, customer won't be able to restore its own backup without admin help.

 

[fabieng]

Let me add some picture, maybe this will show better the issue :

from backup manager (admin view only)

>>> you can see here the 2 latest backup, one done via FTP (latest) and the other at local level

and if I go from domain tab (using customer access) :

the backup via FTP is not visible

>> Let's wait for scheduled backup tonight, but I'm afraid that FTP is not soling the issue.

If I setup distant backup storage, customer is not able anymore to see it.

 

[fabieng]

ok I can confirm now that issue is same with SFTP or FTPS. whatever the method, if backup is on distant server, customer will not see it.

 

[ChristophRo]

You are kidding?
Mount a SFTP folder??? That is contradictory : you mount a (parts of a) remote filesystem on a local filesystem - one does not mount an SFTP folder.
Mount a SFTP folder??? That is odd : you are suggesting to create (some kind of) a distributed filesystem to ..... - yes, to do what? Makes no sense at all..

Ahem, why not? What is NFS if not exactly the abomination you just wrote about? A remote Filesystem mounted locally....with lots of magic in between to make it look like a local one. (root_squash galore anyone? id mapping?)
Same goes for SMB and mounting a share.....it's common practice in the computer world to do just that, simplify file handling operations on remote storage.
And it really does not have anything to do with a distributed filesystem....I don't know why you brought that up, as this is something completely different.

I don't think that mounting an SFTP oder FTP/FTPS share as a local filesystem is a particular nice solutions, but sometimes it's a nifty workaround if the application you're using, lacks some serious features or sense.....like the integrated backup solution of Plesk we're talking about.

And while using FTPS does not help with the issue this thread is all about, mounting a SFTP volume might just do that.