Archiving - what's the story?

It's important to archive

Archiving – If you are a smaller post house or production company what are your options and do you really need it?

The answer is probably yes, unless you never want to be able to go back to previous projects. You never know when you might need some of that old material.

A few years ago when we were shooting on tape we didn't need to worry too much about archiving, as the camera masters would be stored somewhere safe (hopefully!) and we could always go back to them and recapture.

But now as productions switch to tapeless workflows, archiving is becoming more and more important, in fact many would say it's a necessity, and it's going to become even more necessary as we move to 4K.

The memory cards or data packs used to capture from the camera are simply too expensive to leave sitting in the archive, they need to be formatted and sent back into the field. Indeed, for serious productions the data from the camera is archived on location, with the rushes being checked and copied to multiple formats by the data wrangler on set.

Faster equals expensive-er

Once the rushes arrive at the post house they can be loaded into the editing system and also copied to a backup medium, usually a hard drive. During the post-production process rushes, edits, graphics and other media will live on the fast "online" storage of the editing system, for instant access and real time playback. The problem with fast storage is it's very expensive and it fills up too quickly, so although the client and the editor would like to keep material on the system "just in case" the facility manager has other projects coming in and needs to clear that space.
In the short term media can be copied to "near-line" storage, not as fast as the "online" storage, but fast enough to bring it back in a couple of hours if needed. This can even be setup as an automated process, so that the work in progress is being transferred to the near-line storage in the background.
But once the project is complete and the final production has been delivered to the client we need to think about archiving. We need to choose a format and plan for the future.

Value your material and budget for archiving

I find it remarkable how many production companies don't budget for archiving and expect the post house to keep the material around for years at no cost to them. But this is gradually changing, partly as people realise the value of their material, partly because the post houses won't do it for free any more.
The typical archive solution for smaller companies is the removable hard disc. These have become so cheap that it costs very little to have terabytes of storage available. Unfortunately a cupboard full of hard disks is a disaster waiting to happen. Many people do not realise that if a removable disk is sitting in storage for too long, there is a good chance that it won't work again when it is connected to a computer. The problem is the lubrication used in the disk bearings, after a few years without any use the lubrication, can thicken or seep out of the bearings. The next time you try to use the disk, the motor cannot spin the disk and the disk fails. According to a study by Imation in July 2012 "disk drives should be powered up and used to perform a nominal file access procedure every 18 to 24 months in order to redistribute mechanical lubricants."

LTO

So what is a better option? LTO tapes have been used in the computer industry for many years now and recent advances in the technology have made them suitable for archiving of media. LTO stands for Linear tape-open and it is actually a spool of tape with the end available to be picked up by the drive. The drive has an empty spool so the tape can then be wound onto the spool in the drive and the data can be read by a multitrack head. For many years data had to be written to the tapes as a single compressed file but recently LTFS (Linear tape file system) means that we can treat the tape almost like a hard drive (albeit a very slow one).

The current generation of the technology, LTO 5, has tapes with a capacity of 1.5TB and a maximum speed of 140MB/s but we are now starting to see the first LTO 6 tapes. These have capacities up to 2.5TB and speeds of 160MB/s. Indeed LTO 7 and LTO 8 are planned with LTO 8 having cartridges with 12.8TB capacity and 472MB/s transfer speeds but there are no dates for when these may start shipping.
The only problem with LTO is that it is tape; with the data written to it magnetically, this means it is susceptible to magnetic interference, heat, dust, moisture and mechanical damage. The tapes need to be stored in a controlled environment and ideally the data should be transferred to new tapes every couple of years to guard against corruption of the recorded data or "bit rot", which is when the magnetic field of the recording gradually diminishes due to time and external influences until the data can no longer be reliably recovered.

Ideally archives should be written to two or more different types of media and should be kept in different physical locations to guard against fire or flood.

Recently there has been a lot of talk about "the cloud" as an option for storing media. Unfortunately with the current bandwidth and costs I don't think it is ideal for media as the file sizes are just too big, but it is an excellent solution for project files, especially if you want to work collaboratively or in different locations.

Adobes Creative Cloud option gives you access to 20GB of online storage included in the monthly cost and up to 100GB if you have the Cloud Team option. Enough for project files but not enough for all the media of a large video project.

 While archiving is not the sexiest subject on the planet, it is becoming one of the most important, so we're going to be looking at it in more detail in later articles.