Many IT departments view the issue of compliance versus storage costs as a battle where neither party will win. The legal department will always promote the retention of all information, while finance will always push for decreased expenses. However, a tight compliance regulatory framework is one of the most efficient weapons for minimizing your total storage.
Why Over-Retention Is Its Own Risk
Many people believe that saving more data is the safe and conservative option. That’s not true. Every byte you retain after it should have been deleted is a liability that you have to secure on your systems. The more data you have, the more you lose during a breach. Those records you were supposed to wipe after three years but left there “just in case” will be declared part of the breach. The regulators will notice that. And, when legal action for e-discovery calls for electronic evidence, an obnoxious amount of unsorted storage perturbs a relatively easy legal process by turning it into an extremely expensive manual triage situation. Having lawyers sift through terabytes of redundant, obsolete, or trivial files is not what you want to present to the auditors saying you are compliant.
Generally speaking, a sizeable portion of all business content has no business, legal, or regulatory value once it has been dealt with, which means it should be possible to deflect the deletion debate in that general direction. It’s not a small inefficiency. It is most of what most organizations are storing and protecting.
The Data You’re Actually Required to Keep
Various regulations mean different retention periods. HIPAA demands a patient’s health records be retained at least six years from creation date, or from when they last were in effect. But a hold request can require those records be kept even longer. SEC Rule 17a-4 mandates that certain business documents and communications be stored in a way that is impervious to tampering – meaning WORM (Write-Once-Read-Many) technology isn’t optional. Privacy requirements based on personal affiliation often demand that such records be destroyed once the primary purpose that justified their creation no longer exists.
Building the Framework Before Touching the Infrastructure
Before any storage tiering or automated archiving make sense, you must understand what is data retention policy and how it gets defined for your data types and obligations. A retention policy isn’t one thing. It’s a classification system that answers three questions for every category of data you have: why do we hold it, what requires us to keep it, and when does it expire.
Without that classification work done first, any automation you build is operating blind. You’ll tier data you should have deleted and delete data you needed to keep. Data lifecycle management – the overused term for governing data from creation through active use, archiving, and destruction – only works when the policy layer tells the technical layer what to do with what. The policy comes first. The infrastructure follows.
Automated Storage Tiering as a Cost Lever
Once you’ve categorized your data, the concept of tiering is quite simple. High-performance local storage is where active, frequently accessed data goes. Everything else that is categorized as needing to be kept, but isn’t regularly used, gets automatically sent to one of the low-cost cloud storage archive or “cold” tiers. This is pretty common for outdated contracts, archived email, or historical anything. On-demand “cold” storage from large cloud providers is really cheap compared to either on-prem storage or “hot” storage, which is how they differentiate their pricing. The cost difference between low and high ends is enough to significantly reduce the entire line item in your storage budget.
Tiering at scale only happens if it’s left entirely to computers. They read the metadata about how old the file is, what type of data it is, the last time someone looked at it, and whether you decided it was regulatory data. The computer moves the data between the storage tiers, based on the categories it also read from a spreadsheet when you “classified” your data. Ideal tiering requires your employees to have shared their opinions about what data you have in a machine-readable format. If the last two decades of software sales in this area are any indication, that machine-readable format is rusty Excel alone in a folder over on marketing’s SharePoint instance.
The biggest roadblock to a tiering approach is data being stuck in application-specific silos – marketing keeps everything they’ve ever produced, HR keeps every employee record, finance won’t get rid of any transaction data, etc. None of that data is labeled according to your company’s record management categories, because none of those departments have probably even seen the SharePoint list of categories. The unsexy reality is that someone has to aggregate all that information and map it out and get a few people to agree on how it should be labeled and then everyone has to use the same labels when they save anything ever again.
Defensible Deletion Closes the Loop
A retention policy has no teeth if it is not enforced through systematic, routine disposal. If you are in a position to prove that something was kept past its destruction date, you’re also in a position to prove that something was removed precisely when it should have been removed. The former is a fine; the latter is a strong defense. Disposal processes should log what was deleted, when, by what rule, and who authorized it. That record doesn’t need to be elaborate, but it needs to exist.
Compliance as a Cleanup Tool
The organizations that are most successful in this treat compliance and storage, to begin with, as identical concerns. After all, the regulations already prescribe the retention rules. Knowing both what to store and for how long removes confusion about obligations. Cling to the specifics of those rules – no more, no less – and you’re effectively authorized to scrap the rest. That doesn’t sound like a compliance cost; it sounds like a data-mapping opportunity with a free license to delete.
