An untested immutable backup is not a backup: a full Veeam + Wasabi Object Lock disaster drill

In the ransomware era, Object Lock immutable backups are the last line of defense. But between “configured” and “will actually save you” stands one full rehearsal. We played out the worst case: assume the Veeam server itself is destroyed — can data come back from a bare cloud bucket?

The full drill timeline — the red node is the trap you won’t know about until you rehearse

Drill design

1. Create a test bucket on Wasabi (Singapore region) with Object Locking enabled;
2. Add it to Veeam as an object storage repository, immutability set to 4 days;
3. Manually back up a domain controller to it;
4. Verify immutability: attempt to delete the backup → denied ✅ (the lock genuinely works);
5. Simulate disaster: remove the repository from Veeam entirely — the “VBR server destroyed, all configuration lost” scenario.

The restore phase (where the trap lives)

Re-attach the same bucket in the “new” environment:

1. Repository connects; an encrypted backup file is visible inside;
2. Enter the backup encryption password to decrypt — and then the weirdness: once decryption completed, the backup disappeared;
3. For a while it looked like the drill had failed and the data was gone. Calmly re-ran import backup, then restarted the Veeam console — the backup appeared;
4. From there, smooth sailing: browse restore points → mount the backup image → restore files → compare against the source — identical. Drill passed.

That “vanished after decryption” trap gets no prominent warning in the docs: a decrypted backup needs a re-import / console refresh before it shows up in the UI. In a real disaster, if your first rehearsal is the real thing, this moment will break your spirit — you’ll believe the last backup is gone when it’s sitting there fine, one stale UI away.

Bonus experiment: just how immutable is immutable

Cleaning up the test bucket delivered its own lesson:

• A bucket holding immutable objects cannot be deleted;
• The objects can’t be deleted (that’s the point);
• Suspending the bucket’s versioning — also not allowed;
• The only exit: wait out the 4-day immutability, then clean up.

That’s what Object Lock really means: even you — admin, root, the boss — cannot delete it. An attacker with your full credential set gets nothing. The flip side: oversized immutability windows on oversized test data burn storage money just as immutably. Rehearse with small data and short windows.

Lessons

1. The acceptance test for backups is the restore, not the backup. A backup scheme that’s never been restore-drilled has unknown reliability;
2. Drill the worst case: not “click restore in a healthy Veeam”, but “Veeam is gone; re-attach the bucket from zero”;
3. Write the UI traps you find (decrypt → re-import) into the DR runbook — at disaster time there’s no spare composure for debugging; one line in the runbook saves a breakdown;
4. Object Lock’s undeletability cuts both ways: it stops the attacker and it stops your regret. Walk the expiry-cleanup path on a test bucket before production.