Plesk 11.09 makes nightly full server backup but will not upload backup to FTP server

[Matt Grant]

I have 2 servers hosted with 1&1. One is running Plesk 10.4.4 Update #57 and is setup a web server and the other is running Plesk 11.0.9 Update #60 setup as a mail server (both servers have been automatically updated to the latest versions in the last few days).

For well over a year, I have had both servers doing nightly backups of the entire server(s) and uploading the backups to a 1&1 provided FTP server (solely for server backups). About 3 or 4 months ago the Plesk 11 mail server stopped uploading backups to the FTP server with no errors in any logs that I can find. It still makes a local backup every night, which if I do not check every week or so, the backups fill up the hard drive and make the server unresponsive. The older Plesk 10 server does its nightly backups and then uploads it to the FTP server (only keeping the last 3 days as I have both servers setup for). I have spent many days researching how to fix it as well as posting on this forum for help and I cannot figure out what is causing it. I have also re-installed the backup module and reset the backup settings to default and set it up from scratch and it still will not FTP the backup to the FTP server.

Today I was perusing the Plesk Updates web page (http://kb.parallels.com/en/products/latest.html?id=52) and saw this article (http://kb.parallels.com/en/118644) and thought finally a fix!!! I followed what it said and I do not see an error about the /usr/local/psa/PMM/tmp/ directory being full, so it must not be the same issue I have been dealing with. The web server (Plesk 10.4.4) says its /usr/local/psa/PMM/tmp/ directory is using 46% and the mail server (Plesk 11.09) says its /usr/local/psa/PMM/tmp/ directory is using 38%. So the amount of space is not the issue. Both directories show as empty when I ls or ls-al also the /usr/local/psa/PMM/sessions/ directory is empty on both servers which is where I guess I should expect the errors to be if there were any.

[root@web logs]# df -h /usr/local/psa/PMM/tmp/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-usr 3.9G 1.7G 2.1G 46% /usr

[root@mail tmp]# df -h /usr/local/psa/PMM/tmp/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-usr 4.0G 1.4G 2.4G 38% /usr


I decided to compare the pmmcli.log file form both servers and notice they are exactly the same as far as the commands it is running except for one part (well two parts).

I have bolded in red the different parts...

MAIL SERVER pmmcli.log file (all personal info has been x'd out)

8535: 2014-02-17 08:15:52,652 DEBUG --> <pmmcli.GetDumpsListAction object at 0x255ca50>
8535: 2014-02-17 08:15:52,653 INFO stdin: <dump-list-query><dumps-storage-credentials storage-type="foreign-ftp" use-passive-ftp-mode="true"><login>xxxxxxxxxx</login><password>xxxxxxxxxx</password><hostname>backupxxx.onlinehome-server.com</hostname><root-dir>/mail/</root-dir><file-name/></dumps-storage-credentials><object-specification type="server"/></dump-list-query>
8535: 2014-02-17 08:15:52,660 INFO Packet succesfully validated.
8535: 2014-02-17 08:15:52,661 DEBUG <pmmcli.ActionRunner object at 0x255ca10>: doActivity
8535: 2014-02-17 08:15:52,667 INFO Executing <subprocess[8536] '/usr/local/psa/admin/bin/pmm-ras --get-ftp-dump-list --use-ftp-passive-mode --no-default-mask --dump-storage=ftp://[email protected]//mail/ --session-path=/usr/local/psa/PMM/logs'>
8535: 2014-02-17 08:15:52,701 INFO Execution of <subprocess[8536] '/usr/local/psa/admin/bin/pmm-ras --get-ftp-dump-list --use-ftp-passive-mode --no-default-mask --dump-storage=ftp:/[email protected]//mail/ --session-path=/usr/local/psa/PMM/logs'> finished successfully.
8535: 2014-02-17 08:15:52,703 DEBUG <pmmcli.GetDumpsListAction object at 0x255ca50>: response
8535: 2014-02-17 08:15:52,705 INFO Outgoing packet:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<errcode>0</errcode>
<data>
<dump-list>
</dump-list>
</data>
</response>


WEB SERVER pmmcli.log (all personal info has been x'd out)

16974: 2014-02-17 07:59:41,488 DEBUG --> <pmmcli.GetDumpsListAction object at 0x1204390>
16974: 2014-02-17 07:59:41,488 INFO stdin: <dump-list-query><dumps-storage-credentials storage-type="foreign-ftp" use-passive-ftp-mode="true"><login>xxxxxxxxxx</login><password>xxxxxxxxxx</password><hostname>backupxxx.onlinehome-server.com</hostname><root-dir>/web/</root-dir></dumps-storage-credentials><object-specification type="server"/></dump-list-query>
16974: 2014-02-17 07:59:41,494 INFO Packet succesfully validated.
16974: 2014-02-17 07:59:41,495 DEBUG <pmmcli.ActionRunner object at 0x1204350>: doActivity
16974: 2014-02-17 07:59:41,500 INFO Executing <subprocess[16975] '/usr/local/psa/admin/bin/pmm-ras --get-ftp-dump-list --use-ftp-passive-mode --no-default-mask --dump-storage=ftp://[email protected]//web/ --session-path=/usr/local/psa/PMM/logs'>
16974: 2014-02-17 07:59:41,551 INFO Execution of <subprocess[16975] '/usr/local/psa/admin/bin/pmm-ras --get-ftp-dump-list --use-ftp-passive-mode --no-default-mask --dump-storage=ftp://[email protected]//web/ --session-path=/usr/local/psa/PMM/logs'> finished successfully.
16974: 2014-02-17 07:59:41,553 DEBUG <pmmcli.GetDumpsListAction object at 0x1204390>: response
16974: 2014-02-17 07:59:41,555 INFO Outgoing packet:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<errcode>0</errcode>
<data>
<dump-list>
<dump description="" owner-guid="" name="backup_1402150242.tar" creation-date="1402150242" fullname="backup_1402150242.tar" size="17326234796">
<dump-status dump-status="OK">
</dump-status>
</dump>
<dump description="" owner-guid="" name="backup_1402160242.tar" creation-date="1402160242" fullname="backup_1402160242.tar" size="17326695596">
<dump-status dump-status="OK">
</dump-status>
</dump>
<dump description="" owner-guid="" name="backup_1402170242.tar" creation-date="1402170242" fullname="backup_1402170242.tar" size="17327156396">
<dump-status dump-status="OK">
</dump-status>
</dump>
</dump-list>
</data>
</response>


I am assuming the text that is in between the <dump-list> and <dump-list/> on the older web server (Plesk 10.4.4) is the list of the 3 latest backups on the FTP server (which I have the server setup to only keep the 3 latest backups on the FTP server). I know they are different version Plesk on each server and backups might be handled differently from version to version, but they are almost identical. I am about to just get a new server in hopes to fix this issue, but what if something in the migration borks the new server's backup system too? Is there anywhere else I can look for errors pertaining to backups?

I also found that the cron.d file plesk-backup-manager are different (assuming because of different versions of Plesk. Does the spacing in the asterisks matter? I think I copied and pasted the contents of the file from the web server to the mail server to see if it fixed it. For some reason when I paste the cron.d text into this page it does not show any difference between the two. The web server the asterisks are 1 carriage return apart and on the mail server the asterisks are 3 carriage returns apart.

WEB
12,27,42,57 * * * * root [ -x /usr/local/psa/admin/sbin/backupmng ] && /usr/local/psa/admin/sbin/backupmng >/dev/null 2>&1

MAIL
12,27,42,57 * * * * root [ -x /usr/local/psa/admin/sbin/backupmng ] && /usr/local/psa/admin/sbin/backupmng >/dev/null 2>&1


I am sorry this is such a long post, I am just trying to be as thorough providing information to whoever can help me as I can. This is driving me nuts!


My undying gratitude for whoever can help!

 

[Matt Grant]

Wow, 58 views and not one reply...

 

[Matt Grant]

87 views and no replies???

 

[cciecka]

I think I found a fix for this, it is currently being tested as I write this but all signs point to success. Make /usr/local/psa/PMM/tmp a symbolic link to a drive that has adequate space for your backup and it appears to have fixed the failure to upload.

In our case the /usr partition was like yours with 4gb allocated to it but the backup was 8gb. The backup would eat up all the space on the partition and fail. This caused the backup to fail as well as some other nastiness. Long story short aiming that dir to /var/tmp via symbolic link stopped /usr from getting crushed.

Until this latest update we were working fine for about 2 years. Not sure what version we had before this update as we did not track it and this one appears to have been forced on us by our host.

 

[Matt Grant]

Did it work?

I am not the most Linux savvy person out there and I do not want to mess up my server. What was the command you used to make the symbolic link to another directory?

Also if it was a space issue, why wouldn't there be an error somewhere about it? And why would my Plesk 10 server not have the same issue when the partitions are setup the exact same way?

I think I found a fix for this, it is currently being tested as I write this but all signs point to success. Make /usr/local/psa/PMM/tmp a symbolic link to a drive that has adequate space for your backup and it appears to have fixed the failure to upload.

In our case the /usr partition was like yours with 4gb allocated to it but the backup was 8gb. The backup would eat up all the space on the partition and fail. This caused the backup to fail as well as some other nastiness. Long story short aiming that dir to /var/tmp via symbolic link stopped /usr from getting crushed.

Until this latest update we were working fine for about 2 years. Not sure what version we had before this update as we did not track it and this one appears to have been forced on us by our host.

 

[cciecka]

yes...

rmdir /usr/local/psa/PMM/tmp
ln -s /var/tmp /usr/local/psa/PMM/tmp

Test by running the backup again. In my case /usr had 4gb and /var had 65gb. If you have no space on var you need to update the /var/tmp above with wherever you have the space to fit the backup.

To undo these changes,
rm /usr/local/psa/PMM/tmp
mkdir /usr/local/psa/PMM/tmp

 

[Matt Grant]

Ok, I am trying it now, but I have to wait for the backup to run tonight to test it.

Also if it was a space issue, why wouldn't there be an error somewhere about it? And why would my Plesk 10 server not have the same issue when the partitions are setup the exact same way?

 

[cciecka]

You could user the backup manager ui to run a test backup. That's what I did all yesterday to determine the fix.

As for why is there no error and variations for Plesk versions - ya got me... I stumbled on the fix due to the fact that I could not d/l a backup and that lead me to determine where it was getting made and that while it was being made my /usr partition filled up. A lot of df -h while the backups and download requests were being processed.

 

[Matt Grant]

This server is a production server hosting a few hundred email accounts actively being hosted on it. When I run the backups it slows everything down and I get complaints. Is there some sort of quick backup I can try that will not slow the server down?

You could user the backup manager ui to run a test backup. That's what I did all yesterday to determine the fix.

As for why is there no error and variations for Plesk versions - ya got me... I stumbled on the fix due to the fact that I could not d/l a backup and that lead me to determine where it was getting made and that while it was being made my /usr partition filled up. A lot of df -h while the backups and download requests were being processed.

 

[cciecka]

You can do just the config, that is real fast but it does not do everything... If it works than it probably confirms your issue is space related...

 

[Matt Grant]

Eureka!!!!

I think you fixed it!!! The test backup ran and uploaded to the FTP server. But I need to do a full server backup to be absolutely sure. The server config backup is only 3MB, so it may have worked without the symlink. I will report back tomorrow if it works or not.

Thanks again!! I would buy you a beer if you lived in Southern California!

No thanks to Igor or any of the other Parallels support team for not even responding to my cries for help!

You can do just the config, that is real fast but it does not do everything... If it works than it probably confirms your issue is space related...

 

[Matt Grant]

Screw it, I am running the whole backup now. I will report back later.

 

[Matt Grant]

We have success!!!!

Thank you again, again and again!!!

You can do just the config, that is real fast but it does not do everything... If it works than it probably confirms your issue is space related...

 

[cciecka]

Glad to hear it worked for you.