Question "Unable to upload dump to FTP, it remains in server repository", Oh no it doesn't!

[Richard18]

Ubuntu 14.04.5 LTS‬
Plesk version 12.5.30 Update #57

Over the last few days we've had a few backups fail on the upload to our external FTP server with timeout issues.

The backup log says: Unable to upload dump to FTP, it remains in server repository and you can manually download and upload it to FTP.

But, I can't find the backup on the server to manually upload to the FTP server, it looks like the backup process fails and then it deletes the backup tars. I can see from the file system usage graphs that after the backup fails, the fs usage goes back to what it was before the backup.

/var/lib/psa/dumps just has a domains folder with some compressed sql files and some mysql.daily.dump files.

There are none of my backup_XXX.tarX files, how do I get the server to keep the .tar files if the backup fails so I can manually transfer them if need be?

PS, maybe the backup logic could be changed so that if it fails to upload a tar and can't resume the file, it actually tries to upload the whole file again and again (within reason), until it completes the backup operation and not just fail?

INFO: Ftp init url ftps://xxx/backup_1701240206.tar9
INFO: Error 28 Timeout was reached
INFO: Trying to resume upload...
INFO: Unable to resume interrupted upload. It seems like FTP server deleted the file, when the connection was lost. Try to repeat or perform upload manually

 

[IgorG]

Make sure that it is not issues on the FTP server's side:

1. Not enough free disk space on the FTP server.
2. No permissions for the FTP user to write to the target directory.
   

 

[Richard18]

Over 100GB free on the server, backup size is ~28GB, split into 512MB chunks and the FTP user can definitely write to the target directory.

I'm sure it's just a transient issue with the destination FTP server, but I'd still like to be able to manually upload the tar files to it when a backup fails, like the log file says I can do?

INFO: Ftp init url ftps://xxx/backup_1701240206.tar
INFO: Ftp init url ftps://xxx/backup_1701240206.tar1
INFO: Ftp init url ftps://xxx/backup_1701240206.tar2
INFO: Ftp init url ftps://xxx/backup_1701240206.tar3
INFO: Ftp init url ftps://xxx/backup_1701240206.tar4
INFO: Ftp init url ftps://xxx/backup_1701240206.tar5
INFO: Ftp init url ftps://xxx/backup_1701240206.tar6
INFO: Ftp init url ftps://xxx/backup_1701240206.tar7
INFO: Ftp init url ftps://xxx/backup_1701240206.tar8
INFO: Ftp init url ftps://xxx/backup_1701240206.tar9
INFO: Error 28 Timeout was reached
INFO: Trying to resume upload...
INFO: Unable to resume interrupted upload. It seems like FTP server deleted the file, when the connection was lost. Try to repeat or perform upload manually

 

[IgorG]

Any additional details from /usr/local/psa/PMM/logs/backup-<datetime> log file?

 

[Richard18]

This is the end of the file where it starts to go wrong, slightly edited to remove server/user names.

[2017-01-24 03:10:39.316| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar
[2017-01-24 03:20:36.639| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar1
[2017-01-24 03:30:38.371| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar2
[2017-01-24 03:40:26.813| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar3
[2017-01-24 03:47:51.882| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar4
[2017-01-24 03:55:30.049| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar5
[2017-01-24 04:03:26.409| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar6
[2017-01-24 04:10:13.127| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar7
[2017-01-24 04:19:07.253| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar8
[2017-01-24 04:26:39.144| 3876] INFO: Ftp init url ftps://xxx/backup_1701240206.tar9
[2017-01-24 04:26:54.275| 3876] INFO: Error 28 Timeout was reached
[2017-01-24 04:26:59.276| 3876] INFO: Trying to resume upload...
[2017-01-24 04:27:02.697| 3876] INFO: Unable to resume interrupted upload. It seems like FTP server deleted the file, when the connection was lost. Try to repeat or perform upload manually
[2017-01-24 04:27:02.759| 3876] INFO: The utility executed with the return code: -1
[2017-01-24 04:27:03.127| 3876] INFO: Unable to resume interrupted upload[930ee38a-c7a9-4d23-89c9-b026a481e49d]: Unable to resume interrupted upload. It seems like FTP server deleted the file, when the connection was lost. Try to repeat$

[2017-01-24 04:27:03.128| 3876] INFO: pmm-ras finished. Exit code: 1
[3044]: 2017-01-24 04:27:03.235 ERROR 9d089b75-667f-4d43-832d-9f22a9dc4ad2 Unable to upload dump to FTP, it remains in server repository and you can manually download and upload it to FTP. Upload diagnostic message: Unable to resume int$

[2017-01-24 04:27:03.425| 9139] INFO: pmm-ras started : /opt/psa/admin/sbin/pmm-ras --delete-dump --dump-specification=backup_info_1701240206.xml --session-path=/opt/psa/PMM/sessions/2017-01-24-020627.898
[2017-01-24 04:27:03.426| 9139] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2017-01-24 04:27:03.426| 9139] INFO: Repository '/var/lib/psa/dumps': Initialized
[2017-01-24 04:27:03.428| 9139] INFO: Repository '/var/lib/psa/dumps': Get backup info from backup_info_1701240206.xml
[2017-01-24 04:27:03.428| 9139] INFO: Repository '/var/lib/psa/dumps': Get initial backup info for backup_info_1701240206.xml
[2017-01-24 04:27:03.428| 9139] INFO: Repository '/var/lib/psa/dumps': Get items for discovered 'backup_info_1701240206' at path
[2017-01-24 04:27:03.428| 9139] INFO: Repository '/var/lib/psa/dumps': Get files for discovered 'backup_info_1701240206' at path
[2017-01-24 04:27:03.453| 9139] INFO: Repository '/var/lib/psa/dumps': Get info from .discovered/backup_info_1701240206/props
[2017-01-24 04:27:03.453| 9139] INFO: Read properties from xml backup_info_1701240206.xml
[2017-01-24 04:27:03.469| 9139] INFO: Repository '/var/lib/psa/dumps': Validate backup backup_info_1701240206.xml
[2017-01-24 04:27:03.470| 9139] INFO: Repository '/var/lib/psa/dumps': Get items for discovered 'backup_info_1701240206' at path
[2017-01-24 04:27:03.470| 9139] INFO: Repository '/var/lib/psa/dumps': Get files for discovered 'backup_info_1701240206' at path
[2017-01-24 04:27:03.470| 9139] INFO: Repository '/var/lib/psa/dumps': Get info from .discovered/backup_info_1701240206/props
[2017-01-24 04:27:03.470| 9139] INFO: Repository '/var/lib/psa/dumps': Get info from .discovered/backup_info_1701240206/status_OK
[2017-01-24 04:27:03.470| 9139] INFO: Repository '/var/lib/psa/dumps': Get info from .discovered/backup_info_1701240206/dumpresult_SUCCESS
[2017-01-24 04:27:03.485| 9139] INFO: Find incremental backups in '' for prefix 'backup' and base version 1701240206
[2017-01-24 04:27:03.486| 9139] INFO: Repository '/var/lib/psa/dumps': Delete backup /backup_info_1701240206.xml
[2017-01-24 04:27:03.548| 9139] INFO: Repository '/var/lib/psa/dumps': Found 189 files for deleting
[2017-01-24 04:27:05.128| 9139] INFO: Repository '/var/lib/psa/dumps': Initializing...
[2017-01-24 04:27:05.128| 9139] INFO: Repository '/var/lib/psa/dumps': Initialized
[2017-01-24 04:27:05.137| 9139] INFO: Only remote repositories can be cached
[2017-01-24 04:27:05.137| 9139] INFO: pmm-ras finished. Exit code: 0
[2017-01-24 04:27:05.160| 9140] INFO: pmm-ras started : /opt/psa/admin/sbin/pmm-ras --rotate-dump --dump-rotation=2 --guid=b1c98d32-0189-464f-afc1-a8c47cea58b5 --dump-specification=backup_1701240206.tar --dump-storage=ftps://xxx$
[2017-01-24 04:27:05.166| 9140] INFO: Repository 'ftps://xxx': Initializing...
[2017-01-24 04:27:05.167| 9140] INFO: Curl version: 0x72300
[2017-01-24 04:27:05.168| 9140] INFO: Repository 'ftps://xxx': Initialized
[2017-01-24 04:27:05.168| 9140] INFO: Repository 'ftps://xxx': Rotate packed backups is started for owner b1c98d32-0189-464f-afc1-a8c47cea58b5
[2017-01-24 04:27:05.168| 9140] INFO: Ftp init url ftps://xxx/
[2017-01-24 04:27:09.908| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701240206.tar
[2017-01-24 04:27:09.925| 9140] INFO: Ftp init url ftps://xxx/backup_1701240206.tar
[2017-01-24 04:27:14.434| 9140] INFO: Repository 'ftps://xxx': List packed backups at
[2017-01-24 04:27:14.435| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206.tar
[2017-01-24 04:27:14.435| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206.tar
[2017-01-24 04:27:18.264| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206_1701160206.tar
[2017-01-24 04:27:18.264| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206_1701160206.tar
[2017-01-24 04:27:22.379| 9140] INFO: Repository 'ftps://xxx': Validate incremental backup backup_1701150206_1701160206.tar
[2017-01-24 04:27:22.380| 9140] INFO: Find packed incremental backups in '' for prefix 'backup' and base version 1701150206
[2017-01-24 04:27:22.380| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206_1701170206.tar
[2017-01-24 04:27:22.380| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206_1701170206.tar
[2017-01-24 04:27:26.489| 9140] INFO: Repository 'ftps://xxx': Validate incremental backup backup_1701150206_1701170206.tar
[2017-01-24 04:27:26.489| 9140] INFO: Find packed incremental backups in '' for prefix 'backup' and base version 1701150206
[2017-01-24 04:27:26.490| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206_1701180206.tar
[2017-01-24 04:27:26.490| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206_1701180206.tar
[2017-01-24 04:27:30.499| 9140] INFO: Repository 'ftps://xxx': Validate incremental backup backup_1701150206_1701180206.tar
[2017-01-24 04:27:30.499| 9140] INFO: Find packed incremental backups in '' for prefix 'backup' and base version 1701150206
[2017-01-24 04:27:30.501| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206_1701190206.tar
[2017-01-24 04:27:30.501| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206_1701190206.tar
[2017-01-24 04:27:34.487| 9140] INFO: Repository 'ftps://xxx': Validate incremental backup backup_1701150206_1701190206.tar
[2017-01-24 04:27:34.487| 9140] INFO: Find packed incremental backups in '' for prefix 'backup' and base version 1701150206
[2017-01-24 04:27:34.489| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206_1701200206.tar
[2017-01-24 04:27:34.489| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206_1701200206.tar
[2017-01-24 04:27:38.681| 9140] INFO: Repository 'ftps://xxx': Validate incremental backup backup_1701150206_1701200206.tar
[2017-01-24 04:27:38.681| 9140] INFO: Find packed incremental backups in '' for prefix 'backup' and base version 1701150206
[2017-01-24 04:27:38.683| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701150206_1701210206.tar
[2017-01-24 04:27:38.684| 9140] INFO: Ftp init url ftps://xxx/backup_1701150206_1701210206.tar
[2017-01-24 04:27:42.443| 9140] INFO: Repository 'ftps://xxx': Validate incremental backup backup_1701150206_1701210206.tar
[2017-01-24 04:27:42.443| 9140] INFO: Find packed incremental backups in '' for prefix 'backup' and base version 1701150206
[2017-01-24 04:27:42.444| 9140] INFO: Repository 'ftps://xxx': Get packed backup info from file backup_1701240206.tar
[2017-01-24 04:27:42.444| 9140] INFO: Ftp init url ftps://xxx/backup_1701240206.tar
[2017-01-24 04:27:46.749| 9140] INFO: Repository 'ftps://xxx': Rotate structured backups is finished for owner b1c98d32-0189-464f-afc1-a8c47cea58b5
[2017-01-24 04:27:46.896| 9140] INFO: pmm-ras finished. Exit code: 0

 

[IgorG]

Try to debug it with command like

# /usr/local/psa/admin/sbin/pmm-ras --check-repository --debug --verbose --dump-storage=ftp://login@<ftp.server.name>/<dumpstoragepath>

 

[Richard18]

I can't get that to work, it doesn't ask me for a password when I run the command and if I try and put the password in the line it still fails to log in. I can log in fine with the FTP details I'm using!

ie: /usr/local/psa/admin/sbin/pmm-ras --check-repository --debug --verbose --dump-storage=ftp://USERNAMEASSWORD@FTPHOST