Issue Problems with daily backup (subscription)

[omexlu]

Hello @IgorG,

Because the option was removed to disable a website before the daily backup a subscription I get often this error (translated from German to English from my head-memory):

• The backup can be restored and only minor errors occurred.

On the logs of that I only see that some files on the time of the backup arent there anymore.

Can this errors be ignored and the backup are fully restorable?

And how can we/I prevent this in the future? This is why I missed the option to disable the domain before update.

Thank you

 

[IgorG]

Could you please file a detailed report here Reports ?

 

[omexlu]

Could you please file a detailed report here Reports ?

What information do you need?

The backup can be restored and has only minor errors, these are conditioned that some files on the site no longer exist (image attachments from a forum) at the time when the backup runs.

Not all the data was backed up into /var/lib/psa/dumps/domains/domain.tld successfully. Total space: 1734.12 GB; Available space: 1379.85 GB; Mounted on: /var. /bin/tar: httpdocs/wcf/attachments/77/127175-tiny-77f7d0cb3924696ec9afc68306d45868d08a112a: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/77/127175-thumbnail-77f7d0cb3924696ec9afc68306d45868d08a112a: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/77/127175-77f7d0cb3924696ec9afc68306d45868d08a112a: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/78/127171-thumbnail-78dd580d52c8e51cddfbbf1605049d4fbd300ceb: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/78/127171-78dd580d52c8e51cddfbbf1605049d4fbd300ceb: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/78/127171-tiny-78dd580d52c8e51cddfbbf1605049d4fbd300ceb: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/98/127173-98810a1148b04fcf42afddcde657b93e949ca03f: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/98/127173-thumbnail-98810a1148b04fcf42afddcde657b93e949ca03f: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/98/127173-tiny-98810a1148b04fcf42afddcde657b93e949ca03f: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/b0/127172-tiny-b05afd92527911d32ca35fa0e6b725e5b53b97b7: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/b0/127172-b05afd92527911d32ca35fa0e6b725e5b53b97b7: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/b0/127172-thumbnail-b05afd92527911d32ca35fa0e6b725e5b53b97b7: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/bc/127174-bc8b41db1e478772095dbcba009f31015141a174: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/bc/127174-tiny-bc8b41db1e478772095dbcba009f31015141a174: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/bc/127174-thumbnail-bc8b41db1e478772095dbcba009f31015141a174: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/c2/127179-c23bef441513de1809d84e701ec63a799b2f749c: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/c2/127179-thumbnail-c23bef441513de1809d84e701ec63a799b2f749c: Cannot stat: No such file or directory /bin/tar: httpdocs/wcf/attachments/c2/127179-tiny-c23bef441513de1809d84e701ec63a799b2f749c: Cannot stat: No such file or directory /bin/tar: Exiting with failure status due to previous errors

 

[Ales]

I guess these are an unfortunate but somewhat expected consequences when doing backups of a live site.

As to whether the backups are fully restorable... I don't think there can be a definite answer to this, as it depends on a particular case. Backups might be fully restorable technically, but at the same time, the site's internal data consistency could be compromised. E.g. there might be inconsistencies between site's databases and files, etc., depending on what functionality the site has and how it was used during a backup.

@IgorG, are there any plans to re-implement the functionality to suspend domains during the backup?

 

[IgorG]

are there any plans to re-implemet the functionality to suspend domains during the backup?

AFAIK at the moment, we have no such plans. Perhaps @dash can give more information.

 

[omexlu]

@IgorG & @Ales

Ales has thought to the point that is exactly what I had already mentioned in this thread:
Question - Daily-Backup

This feature to disable a domain during backup is very important for live system such as forums where the data is constantly changing (Files & Database).

Because finally what is the use of an backup if this cannot be used if necessary because they are not consistent?

 

[omexlu]

Hello,

This night the same problem/error on full backup of the subscription.

The plesk-team really needs to look into this and find a solution to keep backups consistency.

That's because the option to disable a domain before backuping was so important / useful.

Thanks in advance.

 

[Bitpalast]

The notifications are indeed only warnings that certain files that were present when the backup started are missing when they are actually subject to be backed up. This happens frequently on live sites where many changes occur. It also happens frequently on email boxes. There will be nothing wrong with the backup, it can normally be restored without any issues (and of course without the files that were missing at the time of the backup). I personally don't see how more backup consistency could be created. There is always some time passing between a first log of the files that should be backed up and the actual backup process. The only way I see how better consistency could be achieved is to first create a full shadow copy of everything and then backup the shadow copy. But this will lead to a considerable expansion of backup duration and of course a considerable amount of disk additional disk space required.

 

[omexlu]

The notifications are indeed only warnings that certain files that were present when the backup started are missing when they are actually subject to be backed up. This happens frequently on live sites where many changes occur. It also happens frequently on email boxes. There will be nothing wrong with the backup, it can normally be restored without any issues (and of course without the files that were missing at the time of the backup). I personally don't see how more backup consistency could be created. There is always some time passing between a first log of the files that should be backed up and the actual backup process. The only way I see how better consistency could be achieved is to first create a full shadow copy of everything and then backup the shadow copy. But this will lead to a considerable expansion of backup duration and of course a considerable amount of disk additional disk space required.

Thank you for your answer, the easyst way is to reinsert the option to disable the domain before backup, so no more errors
It can be restored but database and files can be damaged because the files are missed in filesystem but are maybe in the database.

 

[seqoi]

Thank you for your answer, the easyst way is to reinsert the option to disable the domain before backup,

If you disable domain during backup and backup takes some time (for whatever reason) what happens with emails and mailboxes then? I don't think disabling anything is viable option (for me). I would rather sacrifice few minor errors which are completely irrelevant then to have my clients and their mailboxes completely not working and disabled.

As Peter said there will be nothing wrong with the backup, it can normally be restored without any issues. Since there is always some time passing between a first log of the files that should be backed up and the actual backup process you'll always have minor errors which in practice are completely irrelevant right??

 

[omexlu]

But what if e.g. the entry in the database (e.g. in a forum) is still present but not in the filesystem?

If you play in a backup again it could come here in the forum to error messages, because database and file system are not on the same level.

That's why I think it makes sense to deactivate the domain, maybe plesk can only block the website access during the backup and leave everything else activated, that would be an option.

 

[Ales]

If you disable domain during backup and backup takes some time (for whatever reason) what happens with emails and mailboxes then?

As long as DNS records exist, pending emails would wait in the queue and be delivered later. That's how SMTP works.

EDIT: come to think of it, I'm not sure if this is applicable in case of Plesk backups. I'd have to check if mail accounts were even disabled and if so, how.

As Peter said there will be nothing wrong with the backup, it can normally be restored without any issues.

There might be nothing wrong with the backup itself, but there might be something seriously wrong with the site that was backed up even if it seemingly restores without issues, up to a point of having a non functioning site or having hidden security issues.

The worst thing is, many users won't even notice, at least not until trying to restore and perhaps not even then.

Striving to have consistent backups is one of the most important goals of every backup system. It's sometimes hard to achieve, but I see this regression in Plesk backup functionality as a noticeable step in the wrong direction.

As it is now, I'd advise anyone with a complex site to disable it (or at least disable some functionality, it depends...) before using Plesk's backup.

 

[omexlu]

I can't say often enough, that's why the function is so important that the website is deactivated BEFORE the backup is run and only deactivated again when the backup is FINISHED, so this problems don't exists

Plesk should think about this a little bit because with plesk obsidian this should not happen and backups should be one of the most important functions in Plesk.

@IgorG should have a look at it again and maybe forward it to the team.

Thank you.

 

[seqoi]

Igor that makes sense but be honest and practical. If you are already in position to recover something from backup in complex websites then that means (in most of the sites and real world scenarios) that something will be lost during restore because of situation you settled in. I am not saying users should not have exact backup without errors far from it - but i don't see how this could possibly happen.

You can not freeze whole world while you are making backups. On big sites or forums, downtime or domain shut down is not acceptable. And when you run backup, someone could post something, place order or whatever.

All i am saying is that snapshot/backup can be taken and one has to understand what one is provided with after backup is done.

Small niggling errors in plesk log files are laughable, even few days of missing postings are laughable in case you run big profile forum, you get hacked because of freshly released exploit and then you need to revert and patch accordingly. When compared to not having your website available every day due backup domain being shut down for a brief moment of time. That's just my view of course. No disrespect intended.

 

[omexlu]

Every Plesk user must have the possibility to enable/disable this according to his wishes.

In my case this would be more than important that I could count on my backups for my very large forum, even if there might be a small downtime for this domain.

There would be a much longer downtime if the whole forum would be 'broken' by a possibly damaged backup.

 

[seqoi]

Every Plesk user must have the possibility to enable/disable this according to his wishes.

In my case this would be more than important that I could count on my backups for my very large forum, even if there might be a small downtime for this domain.

There would be a much longer downtime if the whole forum would be 'broken' by a possibly damaged backup.

I agree about option, they should return it back!

But how do you fail to realize there is no "broken" forum scenario by possibly damaged backup? You can try to restore these how you call them "damaged" backups (which are not damaged) somewhere else and see for yourself. To me this whole thing seems as blown over every proportion.

All what they (Plesk) are saying is you will have a fully working and restore-able backup even though there is always some time passing between a first log of the files that should be backed up and the actual backup process (hence your little errors in log files which are irrelevant). In your forum scenario that means (in practical form) that it can happen that you
a: you do not shut down your domain during backup
b: Plesk initiate backup
c: backup is started around 10:00
d: backup is finished around 10:05
e: john doe posted a picture in your forum around 10:03
f: this means if you ever needed to restore your backup - your john doe forum picture probably isn't backed up (and this is why you can get error in your log file) - but everything else up to point 10:00 IS COMPLETELY BACKED UP - how do you fail to understand this? Yes log file can report error in file discrepancy because john doe uploaded file but this is not "damaged" backup.

Has it really never occurred to you that we other Plesk users are doing backups of large websites and forums without any domain shut down and we never had any problem in restoring??

In all honesty if Plesk team chosed to simply NOT record this in log files and not to report these small errors you would never even realize them - hence my statement - this is blown out of proportion.

But you are right. They should return it back and be done with it. If people choose to have such option enabled so be it. I personally don't want any forum or website downtime. Completely unacceptable in business environment.

 

[Ales]

But how do you fail to realize there is no "broken" forum scenario by possibly damaged backup?

With all due respect, you're wrong.

What you are failing to see is that a backup of a site, as performed by Plesk, can not be viewed as a single atomic event. It's a series of events.

The backups not being atomic is actually one of the biggest issues any backup software is trying to solve in some way or another. Plesk's solution so far was to suspend sites in an attempt to keep them immutable during a backup. For some reason, they've given up on this approach. I'm sure they have a valid reason, but I suspect it's more to do with the fragility of the related suspend/unsuspend tasks that, in Plesk's eyes, outweigh the benefits. I'm guessing here.

Anyway, instead of being atomic, during a backup of a single site several tasks are performed at different times, e.g. databases are backed up at one point in time, files are backed up at a different point in time, emails at yet another, etc.. What happens with a live site in between these points in time can have measurable consequences.

In other words, you're wrong to believe that a backup simply happens e.g. from 10:00 to 10:05 and that all the data during that time will be backed up exactly as it was when the backup started or when it finished. In case of Plesk backups, the reality is that databases will be recorded e.g. at 10:00-10:01 and the files will be separately recorded at 10:03-10:10.

So let's take a forum as an example: if a forum is functional during a backup, one might end up with a backed up database that has a record of a file upload while the backed up files do not contain this file at all.

And yet both the backed up database and the backed up files will be a part of the same single site backup. One would expect them to relate to a single point in time, but alas, they do not.

Sadly, the removal of the functionality to suspend sites during a backup opens a larger window for any number issues to happen. Please note, Plesk is completely upfront with this, they are not denying the risk or claiming that it can't happen. And all this being said, Plesk backup still has value.

Everybody just needs to be aware of the risks and adjust their backup procedures and strategies as they see fit.

BTW, for those who really did back up busy, highly visited active sites such as forums, chat rooms, e-commerce, etc., without suspending them or without performing some other kind of snapshotting, replication, etc. techniques, please rethink your approach.

Do not believe that your backups are uniform and without potentially serious errors just because they can be restored or because such errors aren't visible. The busiest the site was during the backup, the greater the possibility that your backups contain discrepancies. If you haven't restored the site in the past, consider changing your backup approach ASAP. If you did... well, assess the damage and move on.

Plesk's backup might not have been an ideal tool for such sites even before the functionality to suspend sites was removed, now it certainly isn't. Please consider alternatives.

 

[omexlu]

You bring it on the point

PLEASE PLESK-TEAM BRING BACK THE OPTION OR KEEP A EYE ON IT TO IMPROVE THAT

 

[seqoi]

Ales - Jesus Christ!! Man i was trying to describe process in simplistic way. I know it's not "atomic" event which last 5 minutes. I haven't realized i need to describe every possible process in order to not be corrected.

It was merely an illustration!!!

Ok i am going to play/bite now and nitpick. Your "disable domain" during backup is equally wrong! Even if one disable domain i can see potential problems in server/Plesk background processes which can lead to backup errors unless whole damn server is shut down!! Powering down your server and running backup is only way to ensure data consistency on the disk. Oh look. How i am smart.

However, anyone can also create them from a running system. With problems. I have yet to see ANY serious server company offering server backups with running systems and 100% data consistency without powering down servers (and running backup ABOVE server processes).

So i don't see how that can be done with consistency. If you are here talking current way is wrong what are you proposing? Because shutting down domain may also lead to problems if we are going to nitpick.

I don't see how that can be done. Perhaps Plesk can develop intelligent flush algorithm before snapshotting/backup procedure i don't know.

However even flush cannot guarantee data consistency unless server is powered down.

You said yourself there is no perfect backup. So what are you proposing then?? What?

I am asking you also - how many times you run in to restore problems with current Plesk backup mechanism ? Zero problems here. I want to hear your problematic cases (i am genuinely interested).

You all are acting like a bunch of spoiled kids demanding 100% perfect backup which is impossible to have.

I will repeat - if Plesk chosed to not reveal those tiny discrepancies in their error reports 99.99.99% of their users wouldn't even be aware of "broken" backups - hence you wouldn't whine here about utopian backup procedure which does not exist.

 

[seqoi]

You bring it on the point

No he is not. He is nitpicking. If you knew how operating system works you would knew 100% backup is impossible to have in running operating system and busy server - unless server is powered down and backup is done above server processes. Like i said if Plesk didn't decided to include little discrepancies in their errors you wouldn't be aware of them possibly ever and you wouldn't create thread about it.

That's my stand. I may be wrong but imho it's best that they bring it back and be done with this drama.

I surrender and you guys keep living in your 100% perfect backup bubble wrap which does not exist in current Plesk backup mechanism (even if they bring old option back again)