Internal backup over CIFS / SSHFS mount

[enerspace]

Username:

TITLE

Internal backup over CIFS / SSHFS mount

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE

AlmaLinux 8.10, Plesk Obsidian 18.0.65

PROBLEM DESCRIPTION

Scenario:

Our server currently has only 5% free disk space, which makes it impossible to create a complete backup directly on the server. To address this issue, we have configured the following setup:

• DUMP_D: /var/lib/psa/dumps
  This directory is mounted via CIFS and serves as the final destination for backups.
• DUMP_TMP_D: /var/lib/psa/backup_tmp
  This directory is used as a temporary working directory to prepare backup files before transferring them to the final destination. It is also mounted via CIFS.

Problem:

When we select the option in Plesk to save the backup directly to the server storage (/var/lib/psa/dumps), the process completes without any errors, and the backup is created successfully.

However, when we choose to store the backup on an external FTP account in the Plesk Panel, the following error occurs: "Cannot take ownership of /var/lib/psa/backup_tmp/...: Permission denied", and the backup process fails.

The full error message is:

Cannot take ownership of /var/lib/psa/backup_tmp/pmm-xp-stfa-pmm-de-pffe-files-avt7bO-IRzXiX: Permission denied

[2024-11-16 01:17:53.555|108999] ERR [panel] Plesk\backup\Agent\Exception\ExportFailed: Unable to create the remote backup: Cannot take ownership of /var/lib/psa/backup_tmp/pmm-xp-stfa-pmm-de-pffe-files-avt7bO-IRzXiX: Permission denied
file: /usr/local/psa/admin/plib/backup/Agent/BackupExporter.php
line: 352
code: 1
trace: #0 /usr/local/psa/admin/plib/backup/Agent/BackupExporter.php(195): Plesk\backup\Agent\BackupExporter->finishPmmRas()
#1 /usr/local/psa/admin/plib/backup/Agent/Agent.php(221): Plesk\backup\Agent\BackupExporter->finish()
#2 /usr/local/psa/admin/plib/backup/Agent/Agent.php(116): Plesk\backup\Agent\Agent->finishExporter()
#3 /usr/local/psa/admin/plib/api-common/cuBackupAgent.php(307): Plesk\backup\Agent\Agent->dump()
#4 /usr/local/psa/admin/plib/api-common/cuApp.php(79): cuBackupAgent->_dumpCommand(string '/usr/local/psa/PMM/sessions/2024-11-16-011442.879', string 'a9da3107-50cc-11eb-896d-3a58546ca051', string 'server', string '', NULL null, string '', string 'fast', NULL null, NULL null, string '', string '', string 'backup', string 'script', NULL null, NULL null, string 'ftp://[email protected]//backup/', NULL null, NULL null, NULL null, NULL null, string 'manual', string '/usr/local/psa/PMM/sessions/2024-11-16-011442.879/dump_description', string '', string '', string '')
#5 /usr/local/psa/admin/plib/api-common/cuApp.php(73): cuApp->_callCommandMethod(array)
#6 /usr/local/psa/admin/plib/api-common/cuApp.php(53): cuApp->_executeCommand()
#7 /usr/local/psa/admin/plib/api-common/CuExecutor.php(69): cuApp->runFromCliGate()
#8 /usr/local/psa/admin/plib/api-common/CuExecutor.php(131): CuExecutor->execUtil(string 'cuBackupAgent', string 'runFromCliGate')
#9 /usr/local/psa/admin/plib/api-cli/CliUtilityRunner.php(28): CuExecutor->run()
#10 /usr/local/psa/admin/plib/api-cli/CliUtilityRunner.php(37): CliUtilityRunner->run()
#11 /usr/local/psa/admin/sbin/backup_agent(3): require_once(string '/usr/local/psa/admin/plib/api-cli/CliUtilityRunner.php')

Cause:

The error occurs because the Plesk backup tool attempts to take ownership of temporary files in the /var/lib/psa/backup_tmp directory but fails due to permission issues.

Impact:

Due to this permission error, the backup cannot be transferred to the external FTP resource. As a result, we are currently unable to store backups on the external target, which poses a significant risk to our data security.

STEPS TO REPRODUCE

1. Configure the following settings in Plesk:
   • DUMP_D: /var/lib/psa/dumps (CIFS mounted directory as the final backup destination).
   • DUMP_TMP_D: /var/lib/psa/backup_tmp (CIFS mounted directory used as a temporary working folder).
2. In Plesk, attempt to create a backup:
   • Select the option to save the backup directly to the server storage (/var/lib/psa/dumps).
   • Alternatively, select the option to save the backup to an external FTP account.
3. Monitor the backup process in both scenarios.

ACTUAL RESULT

• When saving the backup to the server storage (/var/lib/psa/dumps), the process completes successfully without errors.
• When saving the backup to an external FTP account, the following error occurs in the temporary directory /var/lib/psa/backup_tmp:
  

  Cannot take ownership of /var/lib/psa/backup_tmp/pmm-xp-stfa-pmm-de-pffe-files-avt7bO-IRzXiX: Permission denied

  This error prevents the backup from being uploaded to the external FTP account.

EXPECTED RESULT

Backups should complete successfully in both cases:

• When saving directly to the server storage (/var/lib/psa/dumps).
• When transferring the backup to an external FTP account, the process should finish without errors, and the backup should be uploaded successfully.

ANY ADDITIONAL INFORMATION

(DID NOT ANSWER QUESTION)

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM

Confirm bug

 

[Sebahat.hadzhi]

Hello, @enerspace. Thank you for the report. That doesn't sound like a Plesk bug, but a permission issue with the remote storage. Please check the permissions and ownership. Ideally, if you have the option, please submit a support ticket and provide our team with access to the remote storage, so they can review the setup and attempt to determine what the issue might be.

 

[enerspace]

This is not a permissions issue. The process actively attempts to change the user, which is not possible with a CIFS/SSHFS mount and results in an error. Even if the user and group of all files are set to psaadm and the permissions of the directory and files under /var/lib/psa/backup_tmp are set to 777, the error persists. This error can be reproduced in any installation and should not require an individual support request, as it is a general issue.

 

[enerspace]

The remote storage is a storage box from Hetzner. These generally do not support the customization of user permissions, regardless of whether via CIFS or SSHFS. Nevertheless, it remains unclear to me why Plesk completely aborts the backup at this point. There seems to be no obvious reason for this.

 

[Maarten]

Have you seen this tutorial?

https://community.hetzner.com/tutorials/automount-filesystems-with-systemd

 

[enerspace]

No, but the same error will occur there as well. It is essentially what we have already implemented.

 

[Sebahat.hadzhi]

Thank you for the update and the additional details. I forwarded the report to our engineers for further review so they can confirm if I am mistaken here. When you say it is no permission issue, have you verified that the psaadm user can create files in the mounted directory? Could you please try:

sudo -u psaadm touch /var/lib/psa/backup_tmp/test.txt

Thank you in advance for your cooperation.

 

[enerspace]

Hi,

The files can be created even if they later have the fixed assigned user:



Mount Option:
sshfs -o IdentityFile=~/.ssh/$_ssh_key_name -o allow_other,idmap=user ${_sshSUBUSER}@${_host}:/ ${_backuptmpFolder}

Alternative for testing:
sshfs -o IdentityFile=~/.ssh/$_ssh_key_name -o allow_other,idmap=user,uid=$(id -u psaadm),gid=$(id -g psaadm),umask=000 ${_sshSUBUSER}@${_host}:/ ${_backuptmpFolder}

The error occurs in both versions. Basically, only root should have access to the folder to minimize the security risk.

 

[Sebahat.hadzhi]

Thank you for the update. The permissions are correct. Our team informed me that CIFS DUMP_TMP_D is not supported for the time being. Apologies for the initial misunderstanding caused by my end. Unfortunately, we cannot provide any ETA for when support will be introduced.

 

[enerspace]

Thank you for your response.

However, there is also a significant difference between CIFS and SSHFS. With SSHFS, only the process that attempts to set the file permissions needs to be adjusted. I believe the adjustments for SSHFS are less extensive than for CIFS. This post was primarily about SSHFS, not CIFS.

Best regards

 

[enerspace]

Why is this feature important?

Many of our customers order vServers with 100GB to 300GB of storage space. In most cases, customers switch to vServers due to a lack of space. It is therefore not uncommon for these machines to be filled up to 90%-95% by the customer. In such cases, no backups can be performed by the customer if the backup is supposed to run via an external FTP connection. This small adjustment would allow us to integrate temporary storage space, which could be used to create the backups.

 

[Sebahat.hadzhi]

Mount Option:
sshfs -o IdentityFile=~/.ssh/$_ssh_key_name -o allow_other,idmap=user ${_sshSUBUSER}@${_host}:/ ${_backuptmpFolder}

Alternative for testing:
sshfs -o IdentityFile=~/.ssh/$_ssh_key_name -o allow_other,idmap=user,uid=$(id -u psaadm),gid=$(id -g psaadm),umask=000 ${_sshSUBUSER}@${_host}:/ ${_backuptmpFolder}

The error occurs in both versions. Basically, only root should have access to the folder to minimize the security risk.

I forwarded the additional details you provided for the SSHFS configuration as well. As soon there is any feedback I will let you know. Thank you in advance for your patience in the meantime.

 

[Sebahat.hadzhi]

According to our engineers it is safe enough to use -o allow_other,default_permissions since it only allows users accessing the folder that are either way allowed to do so locally.

Please let us know if there's a further issue.