The Perils of Long-Term Storage

Wednesday Sep 22nd 2004 by Henry Newman

The Perils of Long-Term Storage

Over the last few months, the issue of long-term tape archiving has come up in my work several times.

Each time the issue is brought up, I hear from supposedly smart people, "We will put it on tape and migrate it in 15 years, one-half of the shelf life of the tape." The people saying this are in technology smart positions (VP of Engineering and Technology for Blah Blah, etc.). Often I am hearing this from organizations that have large digital archives or large analog archives that they plan to digitize. The problem is that you can't always take a tape drive that is 20 years old, with 20-year-old data on it, read the tape and migrate it to a new type of media.

Twenty years ago, the IBM 3480 cartridge was introduced, and if you had written the data from an IBM mainframe running MVS, you would likely still be able to read it today. But what if you had waited two to five years and then written it on a Sun Workstation running Solaris 4.0 or a Cray-XMP, or something else? Do you think you'd be able to read it today? Perhaps, but not very likely, and even if you could read it, how much would it cost to read the Cray-XMP data?

In one sense, I am making a case for mainframes. It might be because mainframe technology, rightly or wrongly, does not change quickly. Change in the mainframe market is evolutionary, not revolutionary. Remember back 20 years ago: you had multiple mainframe vendors besides IBM, including Univac, CDC and others, and the Japanese and Amdahl clones. These vendors have for the most part disappeared, but mainframe technology is backwards-compatible for much, much longer than open systems, be they Unix, Windows or whatever.

There are multiple issues surrounding long-term archival storage that must be considered if you hope to navigate these treacherous waters successfully. Let's take a closer look at them and outline the issues you need to think about carefully. With more and more regulations requiring that data be retained long-term, the topic is an important one.

Issue 1: Getting the data to the media

I have repeatedly discussed tape bit error rates recovered and un-recovered and various disk bit error rates for SCSI and SATA drives, but what about the network to the disk and/or tape device? What are the bit error rates for the Fibre Channel? The recovered bit error rates at the hardware level are about 10E-12. This is 100,000 times greater than the bit error rate on enterprise tape media. This might be why some new emerging standard for end-to-end data reliability are emerging using an 8-byte checksum from the HBA to the storage device. Look at and end-to-end data protection. We are still a few years away, but this will encompass data checking from the host to the device.

Issue 2: The media

As has been discussed ad infinitum, media has a 30-year shelf life. What does that mean? It means if you keep it under the manufacturer's perfect conditions for temperature, humidity and cleanliness, that you should be able to read most of your data, if not all.

What if you have a few days that you do not meet those requirements? Over 30 years, a lot can happen: hurricanes, ice storms, tornados, earthquakes, floods and power outages are just some of the possibilities. You name the disaster and someone has faced it, and even if you planned carefully, you still might not meet all the conditions.

That said, tape is still far cheaper thank disk storage, based on per-byte costs, including compression and power issues, and enterprise tape media is more reliable than the equivalent byte of SATA (3x) or SCSI disk (2x). Calculation of reliability using RAID is difficult at best and is usually proprietary. However, tape density is not increasing as fast as disk density.

In a perfect world, we would mount and re-tension (but not too often), it would not be handled by humans that could drop it, and the environment would always be perfect. If all of this was true, my tape media sources tell me there would be no drop in bit error rate. Since there is little likelihood of all these conditions being met, we will never know for sure.

Page 2: Steps You Can Take

Continued From Page 1

Issue 3: The format

Tapes have often been written in a strange format. My first job as a CO-OP Student working for the U.S. Department of Housing and Urban Development (HUD) in 1979 was to figure out how to read data written on a tape about seven years earlier. HUD no longer used that tape format and no longer had the tape drive. After a few weeks, I located a drive in the bowels of the Pentagon, but they did not have the software to connect the drive to their current system. After a few more weeks, I found the software from the original hardware vendor, the people in the Pentagon hooked the tape drive up, read the data, and then wrote another tape on another machine and I left with tape that we could read. This was my first foray into the compatibility of formats. I have long since forgotten what the drive type was, but I haven't forgotten the lesson. The same issues apply today.

Tapes and drives are not backward-compatible for very long. Five years seems to be about it. Add to this the problem of the hardware interface: in a short time in history, we have seen IPI, SCSI-WD, SCSI-FW, Ultra-SCSI, Ultra160-SCSI, FC-AL, FC-Fabric 1 Gb, FC-Fabric 2 Gb, and soon, FC-Fabric 4 Gb. This does not bode well for future tape drives, interfaces and backward compatibility. What OS vendor is going to maintain drivers for tape drives that are no longer used?

Issue 4: Using the data

Formats for most applications change over time. Try using a document created in Word 1.0 or a very old version of WordPerfect or Excel or any other application. If you digitize movies, will you find the correct MPEG decoder in 20 years? The safest format for movies, from what I have been told by people in the business, is copying to another film. Adobe PDF is guaranteed to be a compatible format for 30 years, but other formats are not. Migrating data from one tape format to another gives you the opportunity to update the application format, which is likely a good idea given how often formats change.


Tapes have some significant advantages over disk for long-term storage, but with those advantages comes the requirement for migration. Long-term, I am not sure that current disk interfaces won't have the same migration problem. Just 10 years ago, we were running RAID devices using SCSI-FW. You can probably find a device driver in most vendors' operating systems for that RAID, but will you have a file system that is compatible that you can mount? Whether it is tape or disk, you are going to have to migrate something.

Here are some steps you can take to prepare yourself for eventual migration:

  • Find out how long the technology you are using will be supported.
  • Understand the costs — including time and personnel needs — of what you will need to migrate.
  • Develop a good understanding of the technologies that are available and their planned migration paths and upgrade paths.

You need to determine what technologies meet your needs for cost, longevity and reliability, because without this information you cannot make the best choices for your organization. Vendors will try to sell you what they have tomorrow for these types of problems; make sure you get the vendor's roadmaps from previous years to make sure that they have met their promises.

You need to view migration as a fact of life, for tape or for disk. With the coming T10 Object Storage Device (OSD) standard, file system migration is going to be a must. I still think that tape is cheaper than disk for long-term storage, even given the migration issues. But whatever your choice for long-term storage, migration must be part of the plan.

See all stories by Henry Newman

Mobile Site | Full Site
Copyright 2017 © QuinStreet Inc. All Rights Reserved