BunnyCDN Data Loss – An Important Lesson

This last week, we learned an important lesson about using BunnyCDN without another cloud provider, open source packages, and the importance of full incremental backups.

Why We Migrated To BunnyCDN

Migrating to BunnyCDN was entirely about storage costs. Amazon S3 is expensive and was dwarfing all the other costs of running our service. Likewise, Cloudflare Images was also lackluster because the pricing model was based on Volume, which we have a lot of. We had to find a solution that was going to not only keep our images small, but store them at a lower rate and not price gouge us on bandwidth. And, indeed, for several months we cut the total price of Amazon S3 from about $80 a month to about $20 – a 4x decrease with most of the cost coming from the fact that having image optimization on is about $10.

Additionally, the server had no need to run a serverless compression on images or for us to compress them on rack. We didn’t need to have any compression, any thumbnail generation, etc. It was all handled by bunny, which was amazing and easy to implement.

Previously, we had used or own compression, tried serverless and then settled on ShortPixel to handle our compression.

No S3 API

Despite the wins from BunnyCDN, they do not have an API that is compatible with an S3 API, which is basically the standard nowadays. Digital Ocean, R2, S3, and others all support the S3 object API.

In 2023, they relayed it was a work in progress – but we’re well into 2024 and it has not arrived.

To handle the implementation of this within MyWaifuList, we then decided to use an open source package.

Our Messy Data

Since we had moved from operating our own compression, then moved to a third party one, then abandoned it altogether, we had a few issues.

First off, the database layer contained four keys for images:

• s3_image_key
• s3_thumbnail_key
• image
• thumbnail

With Shortpixel being the last compression choice before migrating to BunnyCDN, we would actually perform the following for images:

• Create initial image key in a pending folder
• The thumbnail URL would be automatically set to the main pending url while waiting for it to be processed by ShortPixel
• Save a final version into the proper keyed location
• Update the keys

Some problems can arise, though, where certain images would fail to be compressed and therefor the s3_thumbnail_key would be empty as no thumbnail was generated.

The Disaster

With the s3_thumbnail_key being non-generated for some failed images, it was only a matter of time before things blew up. Specifically, the case where the open source library we used was not handling cases where the passed in key could be empty.

Typically, sending in an empty key would mean that the deletion would just fail as there is nothing to delete. But for this issue in particular, sending in an empty key would run a deletion at the root of the CDN folder on BunnyCDN.

While the issue was patched in a later release, the library version we were using was not up to date.

Restoration Efforts

Multi-region replication was on, but despite that, apparently because the changes are sync’d so fast it means they were destroyed on both regions. Secondly, Bunny has no object versioning and no grace period for deletions.

Restoration efforts were ran by scraping the CDN cache and attempting to find images that were still floating around and accessible. This restored a lot of images, but not nearly enough.

Realistically, any gallery images uploaded around Q2 2023 to 2024-04-02 are gone into the void. Anything prior to 2023 was on S3 still, and, thus, was able to be restored back.

Moving Forward

Moving forward, we’ve decided to shift to a new model where data is sent to Cloudflare R2 then automatically copied to Amazon S3. BunnyCDN will sit in front of R2 and offer CDN only.

4 Days ago, Cloudflare rolled out R2 notifications which is now used to carry out this automatic sync for us.

For deletions, items will be deleted from s3 which has object versioning on, so it’s able to be restored within 30 days. No more fuck ups, onii-chan!

Restoring Galleries To Their Former Glory

In the coming days, we’ll be sending an email notification to users to upload back their images to the Waifu Galleries.

Image Upload Permissions have been relaxed, and you can find the options to enable it by accepting the TOS under your Account Settings.

Will this suck? Yes.

Can we rebuild? Yes.

Expect more soon!