What Happened in A Real Ransomware Recovery
We’ve now seen this occur in many ransomware incidents: attackers compromised the customer’s Veeam Backup and Replication (VBR) server and obtained the secret key pair credentials with read/write privileges used to manage the object storage repository.
They could not delete the immutable backup objects. Instead, they deleted the non-immutable metadata objects required for Veeam to access those objects.
The backup data was still present in the object-storage bucket. But from the Veeam’s perspective, it was inaccessible and could not be used for recovery.
The first time we saw this, Veeam support provided a PowerShell script to restore the deleted metadata by removing each object’s “delete marker”, available because versioning is always enabled in an immutable bucket. The script worked, but repairing their 59 TB bucket took more than 60 days. If those object-storage backups had been the customer’s only usable copies, the organization could have faced a choice between waiting for the script to complete to begin recovery – which they didn’t know at the time would take more than two months – or paying the ransom.
Fortunately, the customer had additional backups stored on security hardened Linux repositories. Those copies were available for immediate local recovery. A separate protected offsite backup using our Secure Cloud Connect also enabled Cloud IBR to recover their entire environment in hours, while they performed forensics on their infrastructure.
The incident did not prove that immutability failed. The immutable objects did exactly what they were designed to do.
It proved that immutability alone was not enough – the entire reason we architected Secure Cloud Cloud connect which prevents you from storing the secret key pair on your VBR.
Where The Risk Remains
Direct-to-object backup can simplify offsite storage, but the path to that storage often begins inside the same environment an attacker is trying to compromise.
If the backup server stores or can use the credentials for the object-storage repository, compromising that server may give the attacker a path to damage the recovery process. Even when immutable objects cannot be erased, attackers may target:
- Backup metadata and delete markers
- Credentials used to access the repository
- Backup application configuration
- Repository permissions
- The servers and systems needed to perform recovery
This is why the security conversation cannot stop at, “Is the storage immutable?”
The more important question is:
What remains recoverable if the backup server itself is compromised?
Stronger Recovery Starts with Separation
A ransomware-resilient design should not depend on a single administrative system, one set of credentials, or one type of storage.
The goal is to create independent recovery paths by separating backup copies from the production environment and from one another.
That can include:
1. Protect: Store a local backup copy on a security-hardened Linux repository so the organization has a fast, protected recovery option.
2. Isolate: Send an additional backup copy through Secure Cloud Connect into an environment with a separate security boundary and separately controlled object-storage credentials.
3. Recover: Regularly prove that the protected backups can be used to bring systems back online – not merely that the backup jobs completed.
Additional providers, media types, or geographic locations can add even more redundancy for organizations with higher availability or compliance requirements.
The principle is simple: more independent copies and more isolation reduce the chance that one compromise can eliminate every recovery path.
Immutable object storage is an important part of ransomware protection. It prevents protected backup objects from being changed or deleted during the retention period.
But immutability protects the backup objects – not every component required to recover from them.
Technical Summary
- Immutable does not automatically mean recoverable. Backup objects can remain intact while the organization loses immediate access to them.
- The backup server remains part of the attack surface. If it is compromised, attackers may gain access to repository credentials, backup configuration, metadata, or other components used during recovery.
- Metadata matters. Even when ransomware cannot delete immutable backup objects, damaged or deleted non-immutable metadata can prevent the backup application from locating and using them.
- Independent recovery paths reduce the risk. Hardened local backups, isolated offsite copies, separate credentials, and distinct security boundaries help prevent one compromise from affecting every available copy.
- Recovery testing provides the proof. A successful backup job and an immutable storage status do not confirm that systems, networks, applications, and data can actually be restored.
“Immutability protects backup objects. Backup architecture and recovery testing determine whether the business can recover.”
Your backup data can still exist - and still be unusable.
Immutable
Backup objects remain intact
Recoverable
Data + metadata + credentials + access + tested recovery
- Data (backup objects)
- Metadata
- Credentials (keys)
- Network access
- Tested recovery
Build independent recovery paths
Production workloads
Local hardened backup copy
Tested recovery with Cloud IBR
A Successful Backup Job is Not Proof of Recovery
Backup monitoring tells you whether a job completed. Immutability tells you whether protected objects can be changed or deleted during a defined retention period.
Neither, by itself, proves that the business can recover.
Recovery testing should confirm that:
- The required backup data is present and accessible
- Critical servers can be recovered
- Systems can communicate on the recovery network
- Authorized users can connect
- Applications and data operate as expected
- The organization can document the test result
Cloud IBR provides the recovery layer that validates whether the architecture works. Automated monthly recovery tests can identify problems that ordinary backup-job monitoring may not reveal, while producing a PDF audit report that helps document recovery readiness.
The objective is not another green status light. It is evidence that the protected backup can restore business operations.
Five questions to ask about your backup architecture
Use these questions to determine whether immutable object storage is part of a complete recovery strategy or has become the only recovery strategy:
- If the backup server is compromised, can the attacker use its stored credentials to reach the offsite repository?
- Are backup metadata and the systems required for recovery protected as carefully as the backup objects?
- Do you have another independent copy outside the same administrative and credential boundary?
- Could you recover critical systems from that copy without first repairing the primary backup environment?
- When was the last time you proved the entire recovery process with a real test?
The compromised backup server test
If your backup server were compromised today, which backup copy would still be usable?
Backup server
- Independent copy
- Separate credentials
- Isolation boundary
- Tested recovery
Immutability is a layer, not the architecture
Immutable object storage remains an important part of a modern backup strategy. The mistake is assuming that one storage capability protects every component needed for recovery. A stronger design combines hardened local storage, isolated offsite copies, limited access, separate security boundaries, and repeatable recovery testing. Each layer reduces the chance that one compromised system or credential set can take every recovery option with it. The right question is no longer simply, “Are our backups immutable?” It is:“If ransomware compromises our backup environment, which copy can we recover from, and have we proved it?”
Review the architecture behind your backups
Cloud IBR helps organizations evaluate whether their current backup design has the copies, isolation, access controls, and recovery testing needed to withstand a ransomware event. Schedule a Backup Architecture Review to identify potential recovery gaps before an incident exposes them.Backup architecture review
Would your backups still be usable after a ransomware attack?
Review your backup architecture and identify anything that could prevent recovery.
Schedule a Backup Architecture Review