Why LTO storage is so good for archiving video

HP LTO-6 Ultrium

Phil Rhodes gives us the technical run-down on LTO storage and why you should reconsider storing your valuable footage on hard drives.

The question of what to do with original digital camera material once it's been shot has been vexing for some time. From back-bedroom shorts to blockbusters, it's easy to end up with data that represents far more work than can reasonably be risked on the reliability of a single hard disk.

Achieving reliable storage is at least as much about technique and procedure as it is about technology. Even so, the requirement for a bit bucket that's cheap enough to put on a shelf for long periods, that will remain reliable over those long periods, with high capacity and acceptably fast, seems likely to grow. Camera systems, after all, are continuously being developed to producer greater dynamic range, resolution and frame rate. Welcome as it is, that can only mean more data and, very possibly, an exponentially increasing amount of data (even just assuming linear increases in resolution).

To put this in context, the only magnetic video tape format for which new machinery can be easily found is HDCAM SR and it's already out of manufacture. Along with one or two obscure data tape formats still in use in upscale corporate IT infrastructure, LTO is among the last magnetic tape storage formats made. Tape, it seems, is not nearly as lamented a technology as film was.

The technology itself has been fast and capable for some time; particularly, the associated standardisation process is a lovely thing to behold. The late 90s seemed, in the moment, to be a great time for data tape, with performance and capacity leaping away from the sluggish and unreliable formats of old. Compared to the cooperation and standardisation that's happening now, though, the mixed bag of physical tape formats that we used to deal with, along with the wide variety of data formatting options when actually writing files to tape, was something of a disaster.

The Hardware

The actual LTO tape drive itself, as well as the media it uses, is an advanced expression of modern mechatronics. A lot of historic data tape formats used spinning heads, similarly to a video recorder. The idea here is to increase the speed with which the head and tape move past each other, without having to move the tape very rapidly. The higher the tape speed, broadly speaking, the higher the maximum frequency of information that can be recorded on it and, therefore, the greater the data rate. This works OK, but as DAT users might recall, spinning heads are power hungry, costly, make it far harder to calibrate mechanisms (such that everything will read and write tapes compatibly) and present the possibility of some really serious tape chewing if things go wrong.

Instead, the L in LTO stands for linear; the heads are static, as in audio tape recorders of old, and the tape moves past them. The analogue audiotape metaphor falls apart quickly after this, though, because LTO tape drives have not two (as for stereo) but sixteen heads, each of which writes a portion of a track each time the tape moves past, then moving to cover a different portion of the track. As such, the tape moves back and forth along its length several times in order to write its entire capacity, meaning the tape can move quickly, maximising bandwidth and, with both that and the sixteen heads working in concert, there is no need for tricky spinning heads.

One particularly key aspect of linear tape formats is that it is possible to include what would, on an old audio tape format, be called a confidence replay head. LTO drives read the data they've just written to confirm that it's been written correctly, which isn't something that's really possible with a helical-scan tape format. This adds a lot to the reliability of the LTO format. There are arguments that this feature, which will detect hard write errors, doesn't completely obviate software verification, which can also detect problems in shipping the data to the drive, but it's another welcome feature nonetheless.

The sixth and current generation of LTO offers 2500GB of native capacity, which we need to be cautious about because many statistics for LTO quote a capacity using the system's inbuilt compression and using base-10 numbers. Headline capacities are often given, assuming the compression is switched on and working, which is why many LTO6 tapes are marked with a capacity of more than six terabytes. The real-world formatted capacity of an LTO6 cartridge without compression is actually about 2.29 terabytes. The compression won't work, or at least won't work well, with video data that's already compressed. The compression can be disabled at the point of formatting a tape, although the drives should automatically disable it for data that can't be compressed. LTO6 writes this data at a theoretical maximum of 160 megabytes per second, although real-world rates, given the usual small inefficiencies, are about 140 on large files.

File systems

The other historical problem with tape was the manner in which files were stored on it. A tape drive, much like an unformatted hard disk, provides nothing more than a huge list of stored numbers. Usually, on a hard disk, each of these sectors is 512 bytes long (about 512 text characters worth), and has a unique address which can be used to store or retrieve its contents. To take this and turn it into the normal structure of files and folders that we see on modern computers requires a file system, the system that allows groups of sectors to be stored together and considered as a file, which can then be organized into a folder, among many other features. There are various ways to do this; Apple Macs, Linux machines and Windows PCs do it differently, which is why hard disks created on one aren't always readable on the other. Until LTO5, however, there was no equivalent system for tape drives and various, often incompatible approaches were used. Now, however, we have the linear tape file system (LTFS) which provides the familiar configuration of files and folders and allows files to be stored and retrieved at will.

There are necessarily a few limitations to this. Ordinarily, when a file is deleted, the space it used is freed up for future use. This may mean that gaps can appear in the list of sectors that actually contain data and those gaps won't be the same size as files to be written in future. Because of this, files can be fragmented over several clusters of sectors, and reading the file requires the device to skip around and reassemble them. This is not ideal on hard disks, but it's manageable. On tape, with the need to physically shuttle up and down the tape, it would create an unreasonable performance problem, so deleting files doesn't actually free up space. Given that LTO and LTFS is intended for backup work, this doesn't seem like too much of a problem. Reformatting the tape does, of course, make all the space available again. The only other issue with putting a file system like this on a tape is that maximum performance is only achieved if the files are read in the same order they're written. This is reasonable; it's tape; that's how it works, and it's both a convenient way to do it given the chaotic historical alternatives and a very welcome piece of standardisation.

Practicalities

Current LTO drives are SAS devices, meaning that most computers will need a plug-in PCIe card to control them. SAS is to SCSI what SATA is to the old parallel IDE, and in simple situations, SAS is about as fast as SATA (though it is much more flexible in complex scenarios). There are what purport to be USB3 or Thunderbolt LTO drives, although this usually means a SAS drive in a box with the electronics to control it and communicate via a USB3 or Thunderbolt port.

HP generously supplied me with a kit including an LSI SAS card (which was recognised without fuss by Windows 7) and their LTO6 drive. The LTFS driver software is a free download. Beyond that, it's very straightforward; tapes appear in the same way as any storage device when inserted and require formatting via a supplied utility. Formatting is quick (there is no slow sector-by-sector going over of the tape media) and once that's done, things operate in more or less the same way as a hard disk. The LTFS driver doesn't consume unreasonable amounts of CPU time. Ejecting a tape is as simple as ejecting a flash storage card. In the interests of complete disclosure, there were a few minor wrinkles getting the drive to recognise fresh, unformatted tapes after ejecting a formatted tape, but these issues were quickly solved with a power cycle and didn't happen again.

One thing we noticed when trial-testing the previous version of LTFS on an LTO5 drive is that small file performance could be poor, with the drive audibly ramping the tape transport speed up and down for each file. This is important because it is directly relevant to file-per-frame video storage, where the individual files may be small (eight megabytes for some HD video frames), but the numbers of them are huge. Cameras such as Blackmagic's recording DNG frames, or the raw recording feature of the Magic Lantern firmware for Canon EOS series DSLRs, would create this situation. The performance problem here appears to be more or less fixed. While small-file performance still isn't quite as snappy as that when writing (for instance, large Quicktimes of a few dozen gigabytes each), it's well within the sort of range that could be explained by performance variations in the disk feeding the tape drive.

Purpose

LTO is a backup medium, designed to store large amounts of data securely for long periods. RAID is not backup; a RAID array will faithfully store the result of a human error, for instance. Equally, a stack of hard disks actually has a notoriously short shelf life. One might reasonably point out that LTO is the solution because there really isn't any other, but that overlooks the fact that LTO and the LTFS software is a rare example of standardisation and cooperation that has really worked well. One would not wish to see half a dozen competing formats; that's the point. LTO6, in particular, is a welcome speed and capacity boost.

As to value, packages including an LTO6 drive mechanism and SAS card can be had for under £2000. This can seem expensive if we work out that we would need to write nearly 70 terabytes of data in order to make LTO6 cheaper than just stacking up two-terabyte hard disks (currently around £60). This does overlook the poor archival qualities of hard disks, though, and with cameras producing original material of ever greater resolution and dynamic range, file sizes have already ballooned to the point where 70TB is an entirely plausible data load for even a modest feature film. It's a fair price, then, for a complex piece of electronic and mechanical engineering, although it's still more than most people would spend on a laptop. One would wish for the market to grow enough for it to become a little more accessible to very low budget shoots, where the need for backup is at least as great as the big stuff.

Ultimately, the continued success of LTO in film and television is down to how well it works and how reliable it is as an archive. To be fair, 35mm film only lasts well if it's given quite careful handling, but LTO has some big shoes to fill. At this point, the LTO series is only fifteen years old and, since a shelf life of 30 years is claimed, we're relying on accelerated ageing tests. A bigger problem may be the continued availability of machines to read the tapes, although the LTO group have sensibly decreed that any drive will read tapes written by the previous two generations (and write to the previous generation). So, an LTO6 drive will read LTO5 and LTO4 tapes, and LTO3 drives (which will read 2 and 1) are still available.

The drives themselves are built by HP and IBM, with the specs developed by them in collaboration with Quantum. As with much modern computer equipment, the hardware is built from second-sourced parts. While there may not actually be that many places doing low level manufacturing work for the drives, there shouldn't be a problem finding someone to implement the drawings, should catastrophe occur.

Continued availability of the required half-inch tape stock is assured by the involvement of several different media manufacturing companies, avoiding the issues faced by Sony when their sole half-inch tape manufacturing facility was destroyed by an earthquake-motivated tsunami (this event is widely credited with provoking the mass move to flash-based acquisition).

So, yes, from the viewpoints of both technical functionality and archival longevity, LTO would appear to work about as well as any imaginable format could. And yes, bigger and faster is on the horizon.