By Seth Crosby, Structured Enterprise Services Sr. Consulting Engineer —

Data Protection is frustrating. You’ve seen the statistics, you’ve labored over the vendor decisions, you’ve implemented your data protection solutions, and you’ve done the table-top exercises.

You can do all the right things and still hold onto hope that someone else is in the hot-seat and your backups and restore plans are never needed.

Can we be done?

And you’re not alone. A growing number of our peers have been through a cybersecurity disaster. Yet even if you’ve had a real fire and it hasn’t ended in disaster, data protection can leave anyone with a sense of dread. The threats are getting more complex (thank you, AI) and vendors are changing their products to adapt. Then there’s the bias to secretly believe that the worst attack we’ve survived is the worst we’ll ever see.

So no, we can’t be done. But we can take things in bite-size chunks. Let’s look at 5 issues that undercut the whole goal of data protection. If you take the time and attention to think these through, they will reduce your future regret.

1: Untested Backups

Testing Backups is table stakes, and we all know it. But many IT departments think they can’t afford the overhead to properly test. I’ve personally led several cybersecurity incident data recovery efforts, and I’ll tell you now: no one ever says, “I sure am glad we didn’t test the backups earlier, that would’ve been painful.”

Well, to be clear, it IS painful. Leaders must set up a sandbox environment to restore into, consider the dependency systems, and coordinate the application specialists or owners who will need to validate the data is usable. But none of that is as painful as doing the same thing in a crisis.

Put yourself in those shoes for a moment — the whole team is trying to find even one copy of viable data, while also fielding leadership communication needs, coaching the helpdesk to politely handle end user requests, and attempting to keep up with the barrage of requests from your Incident Response team. That’s an unpleasant 3-way tug of war.

Let’s keep this simple: Test your backups before the crisis. Advocate for the required time and attention even if other projects must be delayed. If it’s executive sponsorship you need, tell your management or peers “I’m accountable for something I haven’t tested, and that’s not going to go well for any of us. Would you like to take the accountability for viable backups, or can we allocate time and resources to do this properly?”

2: Backup your Backup Infrastructure

You read that right. Your backup system has its own core database that points to where your data backups reside. That database is something you’re going to need to restore from good backups. That’s exactly why threat actors stay stealth for so long — while they’re after your key application data, they’re also working to locate your backup system and compromise your backup data. (No recoverability leads to higher ransoms, after all.)

Simply put, make sure that everything you need to rebuild your backup system is safely backed up.

Here’s an example: Veeam Data Platform. At its core, Veeam stores all the metadata and links for backups in a “configuration” database, and it’s encrypted. If Veeam’s server itself is compromised, at minimum you will need the configuration data file and encryption key and password to rebuild Veeam’s servers and initiate a restore of your actual business data. Secure all three of those out of band from your regular backups.

3: No Real Immutability

Immutability simply means “can’t be changed” — that there are some copies of your data that cannot be overwritten before a certain time runs out. It requires that the controls of the filesystem are entirely inaccessible from the network. Think “cable unplugged.”

Real immutability means that you must do something very special to remove or overwrite data that should be locked. “Immutability” that can be bypassed with a special account, or Linux terminal know-how, or by hitting the management interface of a storage array to bypass those trusty XFS flags by clearing the volume, that is not real immutability. That’s misplaced confidence.

In many cases you don’t want all your backup data to be immutable. Immutability everywhere is probably more expensive than you realize when you’re dealing with low RPO’s (restore point objectives).

The deltas of data that are captured with a short RPO need to be written back to the full backup at some point to avoid a huge catalog of fragments, and eventually the full backup gets replaced with newer data. Generally, it’s those consolidated backups that you want to flag as immutable on a daily-to-weekly basis.

4: Backup Permissions Tied to Active Directory

Every data protection vendor emphasizes that integrating Active Directory as an authentication method for backup administrators is a very bad idea —maybe a “worst practice” belongs in their documentation.

Here’s the thing you already know: a threat actor’s primary targets include Active Directory itself because it’s the keys (and certificates) to the kingdom. Mishandling authentication will devastate any organizations’ ability to recover.

Takeaway: Use local accounts, enforce strong passwords with scheduled rotation and monitoring and alerting, and require a robust out-of-band form of MFA.

5: Inadequate Restore Capacity

In an incident of any magnitude, it is often required that all encrypted data be retained for a period of time. This usually is measured in months, and the mandate comes from your insurance company, a government agency, or both. But ask if you have adequate storage capacity for two things:

• To maintain storage of inaccessible encrypted data (which will do something very unpleasant to your deduplication ratios)
• To spin up restored data and workloads to resume business

What comes next isn’t hard to sort out, but it’s a bitter pill to have to reduce velocity to resume business because of inadequate storage capacity.

Suddenly you may feel like the insurance and government agencies that are supposed to have your back are inhibiting your ability to recover. Three ways this can be avoided:

1. Be prepared to invest in additional storage immediately in the event of an incident. Call it ambulance-chasing, but OEMs are usually very happy to move heaven and earth to accelerate delivery if you’re in a crisis.
2. Have adequate secondary low-cost storage to offload encrypted data. Consider expansion of an archival tier for your backup storage.
3. Keep a DR site warm with capacity available to recover critical workloads.

Final Thoughts

These five things are lessons from experience. I’ve noticed the pattern that they are commonly mishandled because they’re simply misunderstood. We like to believe we’re ready — that the threats we face won’t be worse than the worst we’ve heard about, but we’d be like Lucretius’ fool if we don’t recognize that the trends point another direction.

If you need help considering how to make your data estate more resilient to the growing threats, at Structured we have the expertise to help you step toward resilience, not regrets. Contact your Structured account manager today or email info@structured.com.

About the Author

Seth Crosby is an Enterprise Services Senior Consulting Engineer with more than 20 years’ experience spanning the full technology endgame, from infrastructure to data systems to business outcomes.

An early adopter of virtualization technologies and a practitioner in the Web 2.0 and early container application space, Seth brings a rare perspective that pairs deep application development and data systems experience with enterprise infrastructure expertise. He holds multiple iterations of VMware’s core certifications across the SDDC, Network Virtualization and Security, and Desktop Virtualization tracks, most recently in VCF, and has led several cybersecurity incident recovery efforts that restored critical operations under pressure.

Seth’s full-stack fluency — from platform architecture through data systems to the business questions they serve — allows him to help clients connect technology decisions to measurable outcomes. Whether the workload lives in a private datacenter, the public cloud, or both, his end-to-end perspective puts clients on stronger footing to secure and deliver the data that drives their business.