The Trend Toward Long-Term Data Retention
But why are we producing so much data? True, technology advances have better enabled our ability to capture and store data. But technology alone is not sufficient to account for the current rate of data growth.
Data may need to be retained for both internal and external reasons. Internal reasons are driven by company needs. If an organization business requires the data to conduct business and make money then that data will be retained. Today’s modern organizations are storing more data for longer periods of time for many internal reasons. Typically, data is stored longer than it used to be to enable analytical processes to be conducted on the data. Data warehousing, data mining, OLAP, and similar technologies have delivered more and better techniques for extracting information out of data. So businesses are inclined to keep it around longer.
Obviously, however, the number one driver of data management initiatives is likely to be government regulations. Indeed, the growing number of regulations and the need for organizations to be in compliance is driving data retention. Regulations and laws such as the Sarbanes-Oxley, HIPAA and BASEL II are laws governing how long data must be retained. Industry analysts have estimated that there are over 150 federal and state laws that dictate how long data must be retained.
Furthermore, many of these laws are greatly expanding the duration over which data must be retained. Until recently most organizations dealt with mandatory retention periods of only a few years for important business data. And this data was kept around longer because of business reasons, not legal requirements. But the situation has changed due to an onslaught of regulations at the federal, state, and local levels. Depending on the industry, what was once five or seven year retention periods is now expanding to 20, 30, or even 70 years. Indeed, retention periods are determined almost exclusively by government regulations and not from business needs.
So, the current situation is that businesses are storing more data longer than ever before. I'll talk more about this issue in upcoming blog posts...
