Amazon Glacier Integration.

[Slavik]

Amazon glacier is the new backup solution, costing pennies a month opposed to hundreds.

http://aws.amazon.com/glacier/

Do summarise it quickly, it is the new, ultra cheap storage solution available.

Perfect for Plesk backups. However it requires API integration, not just an FTP connection.

Would be awesome if you guys could make this work.

We are excited to announce the immediate availability of Amazon Glacier – a secure, reliable and extremely low cost storage service designed for data archiving and backup. Amazon Glacier is designed for data that is infrequently accessed, yet still important to retain for future reference. Examples include digital media archives, financial and healthcare records, raw genomic sequence data, long-term database backups, and data that must be retained for regulatory compliance. With Amazon Glacier, customers can reliably and durably store large or small amounts of data for as little as $0.01/GB/month. As with all Amazon Web Services, you pay only for what you use, and there are no up-front expenses or long-term commitments.

Amazon Glacier is:

•Low cost- Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month, irrespective of how much data you store.
•Secure - Amazon Glacier supports secure transfer of your data over Secure Sockets Layer (SSL) and automatically stores data encrypted at rest using Advanced Encryption Standard (AES) 256, a secure symmetric-key encryption standard using 256-bit encryption keys.
•Durable- Amazon Glacier is designed to provide average annual durability of 99.999999999% for each item stored.
•Flexible –Amazon Glacier scales to meet your growing and often unpredictable storage requirements. There is no limit to the amount of data you can store in the service.
•Simple– Amazon Glacier allows you to offload the administrative burdens of operating and scaling archival storage to AWS, and makes long term data archiving especially simple. You no longer need to worry about capacity planning, hardware provisioning, data replication, hardware failure detection and repair, or time-consuming hardware migrations.
•Designed for use with other Amazon Web Services – You can use AWS Import/Export to accelerate moving large amounts of data into Amazon Glacier using portable storage devices for transport. In the coming months, Amazon Simple Storage Service (Amazon S3) plans to introduce an option that will allow you to seamlessly move data between Amazon S3 and Amazon Glacier using data lifecycle policies.
Amazon Glacier is currently available in the US-East (N. Virginia), US-West (N. California), US-West (Oregon), EU-West (Ireland), and Asia Pacific (Japan) Regions.

A few clicks in the AWS Management Console are all it takes to setup Amazon Glacier. You can learn more by visiting the Amazon Glacier detail page, reading Jeff Barr’s blog post, or joining our September 19th webinar.

 

[IgorG]

Thank you. I have submitted corresponding feature request to developers.

 

[Faris Raouf]

Personally I don't think Glacier is the right place to put backups. Glacier is for archiving, not backing up.
Yes, I know, there is only a small difference, mainly related to restoring, but nevertheless, Glacier isn't the ideal solution in my opinion.

Instead, if you are going to want to integrate a cloud-based storage service then Amazon's S3 is a better option.

Of course you might want to move some of your S3 backups to Glacier, and this will be possible via the AWS console at some point in the not too distant future.

This is not to say your suggestion isn't an excellent one. I'm sure many would prefer to go direct to Glaceir as it is cheaper than S3. So maybe we could have a choice - S3 or Glacier (or Azure equivalent etc etc).

 

[jojo]

Anyone knows if this has ever been implemented? S3 Glacier is way cheaper and recommended for backups. Thanks.

 

[IgorG]

Plesk Amazon S3 extension does not configure your Amazon storage, it just configures entry point of your storage. So, you just define URL, credentials of your Amazon storage. It may be S3 or Glacier or what else.
Try to define Glacier there, if I'm not mistaken it should work.

 

[jojo]

Plesk Amazon S3 extension does not configure your Amazon storage, it just configures entry point of your storage. So, you just define URL, credentials of your Amazon storage. It may be S3 or Glacier or what else.
Try to define Glacier there, if I'm not mistaken it should work.

Currently it is only creating an s3 bucket by default, no option to set Glacier. It is fine for low storage volume, but once you start bulking up, the cost ramps up pretty quickly.

 

[Faris Raouf]

Hi Jojo.

To be honest, I have discovered that using Glacier DIRECTLY for Plesk backup storage is not ideal.
Have you looked at Wasabi? (search wasabi s3). Wasabi uses the same API as S3 and is pretty much interchangeable. They charge $5.99 per TB which is much less than S3 Standard (note, however, that there is a $5.99 minimum charge per month for new customers and a 90 day minimum retention policy). Using their own figures, if you store data on Wasabi for more than 23 days, it will be cheaper than S3, but if you store it for less than 23 days, S3 is cheaper. For backups, I would imagine you would be keeping data for between 30 to 90 days. Check out their FAQs for more details.

This 90 day minimum retention policy can be a problem (same as Glacier of course) if you are doing short term backups. For example, if you are backing up a WordPress site once every month and only want to keep one copy, then the 90 day retention policy will be a problem in that you will effectively be forced to keep the data for 90 days rather than one month, and your costs triple compared to what you really wanted. Again the same applies to Glacier, which has a 90 day (or is it 60?) minimum retention policy.

I have not yet checked to see if the Plesk cloud storage extension supports wasabi or not. It should - there is no reason for it not to - and I may have a look today.

Other companies also offer S3-compatible storage which may be cheaper. The data centre we use has done so, for example, which I found surprising. I may give it a go soon. I believe DigitalOcean's Spaces is also S3 compatible, so that's another option.

But if you want to stick to S3, then you have a couple of options:
1) As Igor mentioned, you can create a Glacier bucket and backup to that. True, the bucket created by the Extension is a normal bucket. Don't use the extension to create a bucket. Create the bucket in S3 yourself first, then enter the bucket name in the Extension. But I don't recommend this because restoring from Glacier is not immediate. I don't know if the Extension can handle a situation where the data is not immediately available to recover.

2) Create rules in S3 to automatically move your stored backups from the Standard bucket to Glacier after X days. This is, in fact, the best way to do it. The logic is basically you store stuff in Standard for 30 days then move to Glacier and store for 60 or 90 days or something. Recovering the data from Standard is straightforward. Immediate. High speed. Trying to recover data stored in Glacier is not immediate. I don't know if the Extension can handle it as mentioned previously. I think you may have to move the data from Glacier to Standard (via S3 console) then THEN restore.

**** IMPORTANT: I do not recommend using the Extension to create a bucket. You really should create the bucket yourself in S3. You can then set the encryption, storage type and access list properly. A minor error in the extension could potentially make a created bucket public. That would be bad. Creating the bucket manually is far more secure. AND you really should encrypt your buckets, ideally with your own key.

There is a minimum number of days that data must be stored in various types of bucket before it can be moved to another type. But there is also a minimum number of days that data must be stored in some types of bucket otherwise you get charged for it even if you delete it (especially Glacier). What I'm really saying is that sometimes something may appear cheaper but long term it is not, or it may force you to store data for longer than you want, potentially increasing the storage cost by two or three times more than you really want or need.

 

[ControlFish]

Hi Jojo.

To be honest, I have discovered that using Glacier DIRECTLY for Plesk backup storage is not ideal.
Have you looked at Wasabi? (search wasabi s3). Wasabi uses the same API as S3 and is pretty much interchangeable. They charge $5.99 per TB which is much less than S3 Standard (note, however, that there is a $5.99 minimum charge per month for new customers and a 90 day minimum retention policy). Using their own figures, if you store data on Wasabi for more than 23 days, it will be cheaper than S3, but if you store it for less than 23 days, S3 is cheaper. For backups, I would imagine you would be keeping data for between 30 to 90 days. Check out their FAQs for more details.

This 90 day minimum retention policy can be a problem (same as Glacier of course) if you are doing short term backups. For example, if you are backing up a WordPress site once every month and only want to keep one copy, then the 90 day retention policy will be a problem in that you will effectively be forced to keep the data for 90 days rather than one month, and your costs triple compared to what you really wanted. Again the same applies to Glacier, which has a 90 day (or is it 60?) minimum retention policy.

I have not yet checked to see if the Plesk cloud storage extension supports wasabi or not. It should - there is no reason for it not to - and I may have a look today.

Other companies also offer S3-compatible storage which may be cheaper. The data centre we use has done so, for example, which I found surprising. I may give it a go soon. I believe DigitalOcean's Spaces is also S3 compatible, so that's another option.

But if you want to stick to S3, then you have a couple of options:
1) As Igor mentioned, you can create a Glacier bucket and backup to that. True, the bucket created by the Extension is a normal bucket. Don't use the extension to create a bucket. Create the bucket in S3 yourself first, then enter the bucket name in the Extension. But I don't recommend this because restoring from Glacier is not immediate. I don't know if the Extension can handle a situation where the data is not immediately available to recover.

2) Create rules in S3 to automatically move your stored backups from the Standard bucket to Glacier after X days. This is, in fact, the best way to do it. The logic is basically you store stuff in Standard for 30 days then move to Glacier and store for 60 or 90 days or something. Recovering the data from Standard is straightforward. Immediate. High speed. Trying to recover data stored in Glacier is not immediate. I don't know if the Extension can handle it as mentioned previously. I think you may have to move the data from Glacier to Standard (via S3 console) then THEN restore.

**** IMPORTANT: I do not recommend using the Extension to create a bucket. You really should create the bucket yourself in S3. You can then set the encryption, storage type and access list properly. A minor error in the extension could potentially make a created bucket public. That would be bad. Creating the bucket manually is far more secure. AND you really should encrypt your buckets, ideally with your own key.

There is a minimum number of days that data must be stored in various types of bucket before it can be moved to another type. But there is also a minimum number of days that data must be stored in some types of bucket otherwise you get charged for it even if you delete it (especially Glacier). What I'm really saying is that sometimes something may appear cheaper but long term it is not, MyCFAVisit or it may force you to store data for longer than you want, potentially increasing the storage cost by two or three times more than you really want or need.

Plesk Amazon S3 extension does not configure your Amazon storage, it just configures entry point of your storage. So, you just define URL, credentials of your Amazon storage. It may be S3 or Glacier or what else.

 

[João Raposo]

After analysing several options for backup I found out that amazon glacier it's the best solution for my infrastructure.
We don't need to restore the DB immediately, and we can restore the DB by command line interface.

I have read in several posts that this could be possible using the glacier end point URL, but we are not managing to configure it correctly.

Can anybody see what we are configuring incorrectly?

In the IAM account there no information of a attempt to access with the selected user :/


Does anybody already tried to integrate the Plesk backups directly to S3 Glacier?

 

[IgorG]

After analysing several options for backup I found out that amazon glacier it's the best solution for my infrastructure.
We don't need to restore the DB immediately, and we can restore the DB by command line interface.

I have read in several posts that this could be possible using the glacier end point URL, but we are not managing to configure it correctly.

Can anybody see what we are configuring incorrectly?

In the IAM account there no information of a attempt to access with the selected user :/
View attachment 18261

Does anybody already tried to integrate the Plesk backups directly to S3 Glacier?

Try to apply solution from this KB article Adding OVHcloud OpenStack-based S3-compatible storage as backup storage in Plesk fails: Unable to check bucket and directory existence

 

[João Raposo]

Try to apply solution from this KB article Adding OVHcloud OpenStack-based S3-compatible storage as backup storage in Plesk fails: Unable to check bucket and directory existence

Hi Igor!

Thank you for the help!

Unfortunately that solution didn't work out

Did anybody ever successfully connected the Plesk backup system directly to Glacier?

Maybe the API is different and would not work

Probably the best solution it's to use S3 and LifeCycle to transfer the files to Glacier.

Is anybody manages to connect Plesk Backs directly to Glacier, please let me know!

Thank you!

 

[João Raposo]

Hello!

I am now a Licenced Backup to Cloud Pro user, and I am using that extension to upload my Plesk Backups to the BlackBlaze service.

I am just having one problem that I think that the Plesk Team could help me with.

It seems that all the deleted backup files are archived in the BlackBlaze services instead of being deleted, as the image below shows.

View attachment 18951

Is there any way to configure Plesk to delete the files in the BlackBlaze services instead of archive them?

Thank you!

 

[Faris Raouf]

I'm currently investigating this. Older backups are definitely not being deleted.

Part of it is related to the default B2 bucket policy of keeping all file versions.

Logically this should not matter because we are uploading a new file with a new name each time. However, it appears the S3 extension is either:

1) uploading a 0 byte version of the file rather than deleting it, causing two versions, one 0 byte, one original size OR
2) not using the API in the way B2 expects, causing this problem.

I do know that I had to jump through big hoops to delete files in B2 using the API when using my own code, mainly because deleting didn't work as expected (deleting just caused files to be hidden, I think, or at any rate not deleted. I don't honestly remember).

I'm performing some experiments with different bucket policies at present to see what's really going on.

Note that the test** are never deleted either, and you end up with one pair for each backup ever made, plus one, even if they are only 64 bytes in size.

 

[Faris Raouf]

OK. Update: We need to wait at least 24 hours now.

I set a bucket to "keep only last version", waiting 10 minutes (B2 days it can take 10 minutes for a new policy to apply).

This is the equivalent of a manual lifecycle rule of DeleteAfterHiding set to 1 day (you cannot set 0 days).

I then uploaded a new backup and deleted an older backup.

The older backup, as before, was not deleted and instead was one full size version with the original date, and a 0 byte version with the current date marked Hidden. This is actually backwards in my view. Shouldn't the original file be Hidden? Oh well. The whole situation is crazy if you ask me, so this should come as no surprise.

It is unfortunate that it is the 0 byte version of the file that is marked as Hidden, so in theory only the 0 byte version will be deleted with this setting.

Lifecycle rules are applied once per day, so we have to wait 24 hours to see what will really happen.

 

[Faris Raouf]

OK...the mystery is solved.

It turns out that Backblaze B2 is not fully supported after all.

We therefore have to wait for better support before older backups are actually deleted.

In the meantime we will have to manually delete older backups via the B2 control panel.