How can we store our huge file sizes in 2018?

The mLogic mDrive Thunderbolt LTO-8 backup drive

Just what are our options for long term storage of today's huge file sizes?

Today's look at the state of technology in spring 2018 addresses storage at two ends of the scale: very big, very fast, very capable storage, and ways to keep things around for a long time.

Bandwidth. Lots of bandwidth.

Parallelism in action

Way, way down the back of NAB's south hall, where all of the really hardcore computer systems equipment lives, sits Storbyte's Eco-Flash device (a countdown page exists here), a 4U rack case full of dozens of modules, each of which mounts eight mSATA flash disks. On both sides.

That's a lot of mSATA flash disks.

It's conceptually simple – parallelism is not a new idea, but it is a powerful one, and the result is stunningly capable. Anyone interested in shooting 8K, 16-bit images at 120 frames per second will already be interested. The idea, apparently, is to create something that provides a lot of storage and a lot of bandwidth without requiring lots of expensive power and cooling. Flash will generally be cooler and less power-hungry than spinning metal, as well as physically much smaller. It'll also be faster, and technically, the total available bandwidth and storage capacity of Storbyte's new toy can be safely described as “a lot.”

As, inevitably, is the price tag. It's a system in the mid six-figures. But if you need it, it exists.

It is not, however, a long-term storage option. The question of what we do with material once we won't want it for a while has been troubling for some time and it's only going to get worse in the context of 4K, 8K, HDR, and potentially things like lightfield arrays in the future.

LTO

Just about the only answer to this at the moment, is LTO which is good and bad. It's good, because the recently-released eighth generation of LTO is fast and capacious, with a 12-terabyte native capacity (not including the optional compression, which generally won't work on video data.) It's bad because the issue LTO-8 has not improved upon – and the biggest issue facing this sort of backup – is cost. A new LTO-8 tape drive costs at least £3000, plus a SAS interface for the host system, and tapes are over £100 each. That's OK for a twelve-terabyte storage device, but the cost of entry discourages many people who could really do with LTO.

LTO-8 is fast and not fast. Specified to 360MB/s, it's theoretically capable of writing uncompressed, high-definition 10-bit RGB frames to tape in realtime at conventional frame rates. Conversely, the capacity is so huge that it would still take more than nine hours to fill a tape at that speed, but consider this: ten years ago, we were carrying around a fridge-sized box full of hard disks with a measly record capacity to record that sort of video. An LTO-8 deck is too noisy to have on set, so this is a thought experiment only, but make no mistake, it's a capable format.

There is some comfort in that each released generation makes the previous one cheaper and LTO-6 drives (2.5TB capacity, 160MB/s bandwidth) can be had on eBay for under £1000. Still, LTO should be used by more people than actually use it, and the reason it's expensive is probably that of a lack of volume. It's a circular problem, but it's still a very impressive technology.