Resolved Backup Manager times out and unusable after failed remote backup

[ArmReu]

Server operating system version

Plesk Obsidian v18.0.50_build1800230207.15 os_CentOS 7

Plesk version and microupdate number

CentOS Linux 7.9.2009 (Core)

Hi everyone,

I am facing a somehow strange problem:

We are running local and remote backups (via FTP) on a regular base and everything worked fine.

Last night the provider where the FTP-space is located had a problem and so the backup could not be completed. I received a notification mail about this saying
" FTP network error: Unable to access the storage: Transport error: unable to list directory: Curl error:"

So far so good, the problem of the FTP accessibilty was solved and the space is reachable again- verified by trying to reach it from the server via CLI.

BUT now I can't access backup manager anymore to for example restart the backup or even just change/remove the remote settings.

When trying to access backup manager though Plesk backend it keeps running into a timeout.

My wild guessing is that the backup manager or its related services somehow still keeps trying to access the remote connection or what ever.

I tried to find some way to remove the FTP setting manually or via CLI but could not find anything.

I found a way to change FTP timeout settings and such, but nothing to stop the service, remove the remote connection or what ever without having access to the backup manager through plesk tools site.

Also: Does restarting Plesk also include restarting any services or deamons related to the backup manager and might fix the problem?

Kindest regards from Germany!

Armin

 

[ArmReu]

A little update on the issue:

Using the changes suggested in the link below I was able to access the backup manager settings after waiting for the time set in the parameters:

https://support.plesk.com/hc/en-us/articles/360006515153-Plesk-Backup-Manager-takes-too-long-to-respond-or-does-not-load-at-all-when-FTP-storage-is-configured

I would have expected there was a timeout set by default but it appears there is none, thats why the backup manager keeps running into timeout itself trying to reach the remote connection over and over and over ...

 

[Peter Debik]

Hi @ArmReu, thank you for your asking this question here. Before you kill running PMM processes, please make sure that they are indeed hung. Here is how:

https://support.plesk.com/hc/en-us/articles/115000912854-How-to-check-if-a-Plesk-backup-restoration-task-is-still-running

It is thinkable that a backup that previously stalled continues after an issue at the FTP storage is resolved. It at least makes some attempts. So it can happen that it is still going, creating a lot of load which then causes the Backup Manager screen to time out, because the web server that shall display it does not get a feed from the process quick enough.

But I see you've already found the article that describes that.

 

[ArmReu]

For those running into the same problem, here the way I solved it:

1. Reduce Timeout for FTP in /usr/local/psa/admin/conf/panel.ini to something like
   [pmm]
   ftpResponseTimeout = 30
   ftpConnectionTimeout = 60
   ftpRequestTimeout = 60
2. In Plesk to into Backup Manager and wait the seconds you have set above.
   Then you should see you local and remote backups, some marked with an (!) .
3. There might be one backup where the checkbox is greyed out.
   Thats the one keeping your backup manager busy.
4. Go to Remote Storage Settings and disable FTP access.
5. Back to the backups list all your remote backups will have disappeared.
6. Back to Remote Storage Settings again, enable it with the proper credentials.
7. Back in backups list you should now again see all local and remote backups.
8. The one that used to be "greyed out" will still have the (1) but can now be selected, deleted etc.
9. Create a manual local and remote backup to check if everything runs smoothly. For me it did.
10. If your remote server is kinda slow in general, raise the timeout value again to something around 120 for Connection and Request

Cheers from Germany!

Armin

 

[Necroman]

I'm having the same issues and they started after the update to Plesk Version 18.0.50 Update #1. I didn't had this problem on the previous version.
I can't fix the issue, because backup manager timeouts. I tried this https://support.plesk.com/hc/en-us/...ot-load-at-all-when-FTP-storage-is-configured and this https://support.plesk.com/hc/en-us/...-file-in-Plesk-interface-504-Gateway-Time-out but didn't help.
I even tried to change store in both Both server storage and FTP(S) storage to Server storage in /var/lib/psa/dumps only on the Scheduled Backup Settings (using the Scheduled Backups List extension), but it also timeouts!

Where can i find Remote Storage Settings? All actions related to plesk backup are getting timeouts!

Thanks!

 

[Necroman]

One more thing, i'm using Plesk temporary domain name on my two servers to save FTP backups from each other - does this has anything to do with it?
Thanks!

 

[Necroman]

Ok i found the Remote Storage Settings but after disabling i'm no longer able to enable it, i save the settings and i see the Please Wait info, but the screen never changes.

I used curl -v -P - --ssl -k -u <user> '<ftp_address>' as stated on the Plesk error i got when i got to Backup Manager and it connects successfully with no errors!

 

[Necroman]

Sorry for the multiple messages...
After saving the Remote Storage Settings i got the following error:
Error: Unable to access the storage: Transport error: unable to list directory: Curl error: (28) Timeout was reached: Last FTP request: LIST Last FTP response: 150 Opening ASCII mode data connection for file list
Make sure you have entered the correct storage settings. You can check them independently with the command:
curl -v -P - --ssl -k -u <user> '<ftp_address>'

As stated before, if i run the command i get:
* About to connect() to <host> port 21 (#0)
* Trying <ipv6_address>...
* Connection refused
* Trying <ipv4_address>...
* Connected to <host> (<ipv4_address>) port 21 (#0)
< 220 ProFTPD Server (ProFTPD) [<ipv4_address>]
> AUTH SSL
< 234 AUTH SSL successful
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=<host>
* start date: Jan 22 08:31:10 2023 GMT
* expire date: Apr 22 08:31:09 2023 GMT
* common name: <host>
* issuer: CN=R3,O=Let's Encrypt,C=US
> USER <user>
< 331 Password required for <user>
> PASS <password>
< 230 User <user> logged in
> PBSZ 0
< 200 PBSZ 0 successful
> PROT P
< 200 Protection set to Private
> PWD
< 257 "/" is the current directory
* Entry path is '/'
> CWD /<path>
* ftp_perform ends with SECONDARY: 0
< 250 CWD command successful
> EPRT |1|<ip_server>|57721|
< 200 EPRT command successful
* Connect data stream actively
> TYPE A
< 200 Type set to A
> LIST
< 150 Opening ASCII mode data connection for file list
* Maxdownload = -1
* Preparing for accepting server on data port
* Checking for server connect
* Ready to accept data connection from server
* Connection accepted from server
* Doing the SSL/TLS handshake on the data stream
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=<host>
* start date: Jan 22 08:31:10 2023 GMT
* expire date: Apr 22 08:31:09 2023 GMT
* common name: <host>
* issuer: CN=R3,O=Let's Encrypt,C=US
* Remembering we are in dir "/<path>/"
< 226 Transfer complete
* Connection #0 to host <host> left intact

One last note, i'm using a hostname instead of IP Address and although i'm using two servers, i can only access the backup manager on one of the servers, the other one, even after disabling FTP backup i still get "504 Gateway Time-out nginx" when accessing backup manager.

Thanks in advance!

 

[Necroman]

So, enabled passive FTP and i could save the settings! Very strange this behavior... I will wait for the scheduled backup to run and report back, althoung i still can access backup manager on my other server after making the changes i made on this one.

 

[Peter Debik]

@Necroman Please see Issue - Backup broken since update to 18.0.50 (when using FTPS+"active mode") . It is a known product issue that will be fixed in one of the nearest updates.