• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Resolved SFTP to Azure issue - EXTPLESK-3418 issue again?

trialotto

Golden Pleskian
Plesk Guru
Username:

TITLE

SFTP to Azure issue - EXTPLESK-3418 issue again?

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE

All versions!

PROBLEM DESCRIPTION

SFTP to Azure (Blob storage) results in

Warning: Failed to resume upload of a test file to the storage: Failed to transfer the remote file: sftp: "FeatureNotSupported: This feature is not supported." (SSH_FX_OP_UNSUPPORTED)
. Upload resuming may not be supported by the remote storage. Configure the storage or use another one.

STEPS TO REPRODUCE

STR : not really needed

COMMENT : Azure Storage does not support certain operations, see

Limitations & known issues with SFTP in Azure Blob Storage - Azure Storage

ACTUAL RESULT

Warning: Failed to resume upload of a test file to the storage: Failed to transfer the remote file: sftp: "FeatureNotSupported: This feature is not supported." (SSH_FX_OP_UNSUPPORTED)
. Upload resuming may not be supported by the remote storage. Configure the storage or use another one.

EXPECTED RESULT

Should have been resolved already, see Plesk Obsidian release notes, in particular

Configuring or attempting to back up to a remote FTP repository hosted on the Azure Blob service no longer fails with the “(SSH_FX_OP_UNSUPPORTED)” error. (EXTPLESK-3418)

ANY ADDITIONAL INFORMATION

none

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM

Confirm bug
 
Thank you for reporting, I have forwarded it to developers as an attachment to the previously reported case.
 
Developers explained that Azure Blob storage via SFTP does not support uploads resuming. So any time a network issue occurs, upload breaks. It is something that cannot be fixed on our side, because resuming is simply not supported by the target. "Resuming uploads" is also explicitely listed as an unsupported feature on Limitations & known issues with SFTP in Azure Blob Storage - Azure Storage .

Peter,

The limitations of Azure are well-known, but this is a bug that is not related to Azure limitations.

It is all related to a well-known bug in SSH ....... and it is an issue on the client-side (Plesk) and not on the server-side (Azure, AWS etc).

In most cases, this bug can be worked around by messing a bit with the read/write permissions, but that also requires the client-side code to be rewritten.

As far as I know, the bug has not been patched (not yet), but most "clients" have been rewritten ...... since there is alternative way to connect, native to SSH.

Please note that it has been a while that I investigated the issue, so all of the above is from the back of my head.

If necessary, I will dive into the topic ......... but if Plesk Team is not willing to change code or investigate the bug, then any research does not matter.

Nevertheless, it would be a very bad thing that a bug in SSH would become persistent if developers only point to alleged issues "on the other side".

In fact, this is one of the primary reasons why the SSH bug has not been getting priority ...... and this is a bad thing, for many reasons.

As a final remark, Azure (and other cloud platforms) do not allow for resuming uploads, because it is not necessary : it is "old skool" behavior of clients and inherent to most client-side code (that has not really changed), even though the cloud platforms do not need to support "resuming uploads".

Stated differently, cloud platforms maintain an open connection and only the client-side is attempting to resume uploads without necessity.

This is also why the odd work around (to the bug) of messing with read/write operations is working : the client-side is simply uploading without having an option to fall back in the "bad behavior" associated with the "resuming uploads" functionality.

I am particularly using the words "old skool" since we are not living in a period anymore where connections were consistently unreliable and interrupted : any file of any size can be pushed to the cloud without any effort.

I am not sure yet whether the SFTP extension issue is related to the extension or the SSH related libraries ... but by intuition, it is related to the extension.

Kind regards....
 
What about the "Unsupported operations" section in the Azure article?

Is not really relevant.

I can recall that I have investigated the topic in a recent past and that the SSH bug was signalled, but also outshadowed by a plethora of articles.

It is essentially a "one-cause-many-consequences" issue that has been dealt with / discussed on the "symptom" level : consequences got most attention.

Why are the Azure limitations not relevant?

Well, by simply removing the "read" (if I can recall it correctly) permission on the cloud-side, there was no issue on the client-side (at the Plesk instance).


The thing here is that there are two issues that should be dealt with separately in order to come to a proper analysis.


First, when configuring a SFTP connection to Azure (or other specific cloud platform), one often gets an error notification.

The error notification can be ignored, but gives an indication of the issue : a read related issue.

Despite of the error notification - indicative of the SSH bug - there is an option to upload backups to the cloud : it works to some extent, at least it did.


Second, when having configured the SFTP connection and using the SFTP connection to backup specific files, then there might be an issue that is dependent on the size of the backup : large files will not backup, small files will (at least, they did in the past).

This seems to be related to - for instance - resuming uploads related limitations, but they actually are not.

The bad behavior is less present or even absent when removing the "read" permission on the cloud-side.


In summary, the whole issue seems to be related to the strange (and native) behavior on the client-side, caused by the "SSH bug".

Nevertheless, even though it actually is a bug, there is also the fact that alternative method of connection can be created on the client-side.

Most of the cloud based backup providers are confronted with this particular issue and they use the alternative method as a work-around.

So, one can read a lot of articles about the work-around, even though only 1 or 2 articles exist on the actual cause : the SSH bug.


Again, all of the above is just from the back of my head, I simply stopped reading and researching ........


Kind regards....
 
Hi. I am an average newly registered user but even me can see that there is a problem using SFTP Azure in Plesk under Backup Manager > Remote Storage Settings. I have tried so many combinations on "FTP(S) Storage Settings" but nothing works so I searched and found this thread. The killer argument for me is the SFTP client psftp.exe from the PuTTY website: my credentials are fine and yes I can transfer files like on the command line. So the problem is really with Plesk.

I don't buy any argument. Please create an SFTP storage blob on Azure (costs so little and you can even have a free trial) yourself then get the SFTP client psftp.exe from the PuTTY website and try. Then use those exact same credentials including directory for backup files storage. That's 100% reproductible error. As simple as that.

I cannot comment on the rest, I am not advanced enough for that but happy to confirm a bug with off-the-self (free) tools.
Thank you.
 
@max.plesk, FTPS and SFTP are two completely different things. To use SFTP you need Backup to Cloud Pro extension.

@Peter Debik and @Kaspar

It has to be admitted that @max.plesk is correct - he is right in stating that SFTP to Azure does not work.

At least, he is right unless there has been an update of the extension after my last moment of testing.

It is not relevant to state that SFTP and FTPS are different and it is also not relevant to state that Backup to Cloud Pro has to be active.

I did some testing in the past - with Backup to Cloud Pro active - and the SFTP to Azure does not work.

I did multiple test at various moments in time in the past, with the following results :

- at time t = 0, the SFTP to Azure did not work,
- at time t + 1, SFTP to Azure started working with issues (but it worked to some extent) due to an update of Backup to Cloud Pro,
- at time t + 2, SFTP to Azure stopped working (did not work at all), as a result of protocol updates on the Azure side,
- at time t + 3, SFTP to Azure worked to a very minor degree (read: randomly), as a result of issues in / bugs with Backup to Cloud Pro,
- at time t + 4, SFTP to Azure worked again with issues (see t + 1) due to an update of Backup to Cloud Pro,
- at time t + 5, SFTP to Azure stopped working again as a result of protocol updates on the Azure side,
- at the last moment of testing, SFTP to Azure is not working as a result of issues in / bugs with Backup to Cloud Pro.

I honestly do not know what the actually status and nature of errors and error notifications is, but if the Backup to Cloud Pro extension has not been updated, then @max.plesk is fully correct and SFTP to Azure does not work.

In essence, there are two types of reoccurring issues with SFTP to Azure :

1 - Permission related issues

Plesk SFTP to Azure tries to write a file to Azure in order to establish that all required permissions are present.

In some cases, Plesk is not able to write the test file : this implies a wrong SFTP setting on the Azure side.

If all SFTP settings are correct on the Azure side, Plesk is able to write the test file, but fails to establish correct read/write permissions.

This is a well-known bug in the SFTP protocol packages that are used at compilation of the packages provided by Plesk.

There are two workarounds for this well-known bug : (1) allowing full permissions on the Azure side and (2) changing code of the SFTP client.

Allowing full permissions on the Azure side has been tested thoroughly by me : it does not solve the issue.

Stated differently, the issue is not on the Azure side.

In conclusion, Plesk Team should solve this bug by altering the code of the SFTP client, otherwise SFTP to Azure will never work.

2 - File transfer related issues

Azure works with file chunks and file upload resumption.

The SFTP client from Plesk does not deal with file chunks and/or file upload resumption properly.

As a result, files of 2Gb or more are often not properly uploaded when using SFTP to Azure.

I did not test whether this issue is specific to Azure alone, but I can safely assume that it is a general issue related to the SFTP client in Plesk.

In conclusion, Plesk Team should also investigate this issue and adjust the code for the SFTP client, if necessary.


In summary, @max.plesk is correct in his statements and I have provided some other topics that deserve some attention.

After all, it cannot be the case that a paid-for extension like Backup to Cloud Pro is not working properly.


Kind regards....
 
Again, please look at the "Unsupported" section of Limitations & known issues with SFTP in Azure Blob Storage - Azure Storage. It clearly states that resuming uploads are unsupported.

@Peter Debik

Exactly my point.

Nevertheless, we apparently both speak English, but we do not understand each other.

So, I suggest that you create an Azure development account, create a storage space and activate SFTP (with all permissions allowed for the SFTP user).

Then try to fill in the SFTP credentials in Plesk, in the pages dedicated to SFTP Backp - part of the Backup to Cloud Pro extension.

Before pressing enter, make sure that you have the Azure Portal open on the storage account that you have created and that you can refresh instantly, to see what happens in the storage account.

You will see that - when pressing enter - the SFTP client on the Plesk side will create a file ..... and delete that file.

At least, that is what should happen and what did happen in the past.


Nevertheless, the SFTP client will moan about something and give you some error notifications.

Yes, error notifications, even though the test file has been created properly and deleted properly.

So, when not talking about file transfer related issues, there is the aforementioned issue that is clearly related to the SFTP client from Plesk.


Now, let's talk about the transfer related issues that I mentioned.

Microsoft states that file upload resumption is not allowed.

Microsoft actually means to say that "IF the transfer is interrupted FOR WHICHEVER REASON, THEN the file upload will not resume".

So, connectivity issues on the SFTP client side, bad code on the SFTP client side ....... that is mostly the reason why file uploads fail ......... since on the Azure side there is no real problem with connections (read: it is almost impossible to break the connection on the Azure side!!!!)

In fact, Microsoft does not allow for file resumption IN ORDER TO PREVENT THAT BAD SFTP CLIENTS make a mess of Azure and endanger SECURITY!


Now, back to the file transfer related issues CAUSED by the SFTP CLIENT.

There is this remarkably simple test that one can do in order to investigate the topic "how bad is the SFTP client?" : just use SFTP directly from the command line on a server and upload any (small or large) file to Azure ........ it goes without any problem.

IF direct SFTP to Azure is not an issue at all, then the SFTP client provided with the Backup to Cloud Pro extension must be an issue.


Sure, we do not want to alarm anybody, but there certainly is something about the SFTP client provided with the Backup to Cloud Pro extension.

One can try 100 times to create an upload via SFTP to Azure and most of the attempts will fail, but some of the attempts are succesful to some degree.

Nevertheless, none of the succesful attempts are coherent or consistent in terms of results.

Again, this proofs that there is nothing wrong on the Azure side of SFTP transfers - only the SFTP client should be investigated.


Having said all of the above, it is possible to provide a simple analysis of the transfers that were tested and that were succesful in the past.

Yes, 100MB, 1GB, 100GB transfers are/were NOT an issue on the Azure side.

That is, when SFTP to Azure was still working properly!

So, if these small and large files cannot be transferred at all, then the SFTP client is not working at all.

In fact, the last time I checked, the SFTP client stated that the client was not able to connect, even though the test file was written - how can a SFTP client not connect and still write a test file??????


So, having concluded that the SFTP client does not work properly AT THIS MOMENT, now BACK TO THE DAYS WHEN IT WORKED.

At the time the SFTP to Azure functionality worked properly, it was not an issue to send a file of any size to Azure.

Stated differently, we need to talk about what actually happens under the hood of the SFTP protocol machine.


SFTP is inherently ineffective for large transfers and is associated with a huge disadvantage : a considerable performance overhead.

SFTP works with a single TCP connection, mostly with limits set to throughput : the chunk size.

SFTP works with SSH to decrypt and decrypt data, so file resumption is barely possible if connectivity issues occur during the tranfer of a chunk or a file.

In brief, SFTP inherently works with chunks and the SFTP client should be coded in such a way that chunks are properly transferred AND coded in such a way that chunks (or files) can be resumed if a temporary failure during transfer occurs.


The last paragraph gives simple hints were Plesk Team can or should improve the Cloud to Backup Pro extension, at least for Azure support.

Nevertheless, that is not the full issue here.

As stated before, SFTP via SSH to Azure works fine and the SFTP to Azure via the Cloud to Backup Pro extension does not .......

............... so, if SFTP works over SSH and SFTP via SSH to Azure works fine, then there is a big issue with the SFTP client provided by Plesk.

For security reasons, we do not allow SFTP to Azure via the Cloud to Backup Pro extension, given the issues with this particular SFTP client.


Sure, we can talk about what a manual of Microsoft seems to suggest or what you seem to deduct from my posts ......

........ but should it not be more IMPORTANT to get a PAID-FOR extension - like the Cloud to Backup Pro extension - working PROPERLY AND SECURE???


As for my English, my sincere apologies if it cannot be comprehended in full.


Kind regards....
 
Back
Top