SSDs will not be replacing HDDs as soon as you think

All future predictions thought that by now cheap solid state storage would be normal, and that traditional mechanical HDDs would be a thing of the past. How wrong they were. Phil Rhodes looks at why this is, and how traditional hard drives have seen a lot more development recently that you might have expected. 

2018 is the year that some particularly optimistic predictions, particularly a famous one by analysts Trendforce, suggested that the per-gigabyte cost of solid state storage would cross over with that of traditional hard disks. As anyone who's rebuilt a video storage RAID recently will have realised, that hasn't quite turned out to be the case.

Solid state is fast, quiet, small and robust - but it's still about twelve times more expensive per gigabyte than a hard disk

It's a calculation quite a lot of people do every couple of years, when an existing cluster of disks has become old enough to worry about, or when the number before the K has incremented again and the current storage space has started to feel inadequate. Surely, we wonder, flash storage has advanced to the point where we can reasonably build a big video storage device out of it?

Well, yes, you can; people with a particular interest in portability, low noise, physical robustness or low power consumption have been willing to do this for some time. In the film and TV world, this often means digital imaging technicians, who can often make more than enough money to pay the premium for a big storage array that makes no noise and doesn't mind being rolled around a field on a magliner all day.

Big storage arrays are (almost) invariably still made out of hard disks

The expense of SSDs

But for most of us, in early 2018, it's still a stretch. 256GB (that is, quarter of a terabyte) of of flash storage will currently cost roughly £80, give or take the choice of M.2 or SATA disks and what sizes of device are involved. A similar amount of money will buy a decent 3TB hard disk, representing twelve times more storage, albeit at rather lower performance. Some people might be happy to laugh in the face of danger and run solid-state devices without RAID protection, which might halve the amount of storage required, especially given that faster SSDs generally don't need the performance benefits of RAID redundancy. Even in that very permissive situation, though, the 6:1 space advantage is pretty hard to ignore.

It is, of course, possible to pay a lot more than that for hard disks, and some people would claim it's worthwhile. 4TB of storage can be had for as little as £90 (the Seagate Barracuda ST4000DM004) or as much as £202 (Western Digital's WD Gold “datacenter” disks, part number WD4002FYYZ.) Some of the differences between them are fairly clear – the Western Digital drive spins at 7,200 RPM, versus the Seagate's 5,400 RPM, with the higher number resulting in better performance. Furthermore, the Seagate has only 64MB of cache memory, against Western Digital's 128MB.

Digging deeper, we find that Western Digital specifies that the drive is designed for a workload of 550 terabytes per year, equivalent to completely filling the drive something like 137 times. Seagate's budget models don't specify this - though they have options which do.

Portable media such as these Atomos Shogun caddies have been made possible by solid state storage, but it's still a specialist application

Seagate Ironwolf Pro

The thing to consider is how much difference any of this will make. The benefits of that cache memory will depend on the workload. Access to multiple small files will benefit far more than long, streaming accesses of large files, which is more representative of a typical video workload (assuming fragmentation on the disk is reasonable). The higher spindle speed will be noticeable, although it's possible to buy Seagate's Barracuda Pro line, achieve the same spindle speed, and still pay only £150 per disk. A fairer comparison with the WD Gold might be Seagate's Ironwolf Pro series, which, like the WD Gold, are intended for use in big arrays to perform big data storage. Ironwolf Pro disks at 4TB are still rather cheaper than Western Digital's datacenter disks, but the intent is largely the same.

Buying these kinds of disks, those specifically intended for high-workload array applications, might seem to make a lot of sense for video work. They often boast specially-tuned firmware and clearly stated yearly throughput specifications which can be easy to hit with bulk data such as video. The issue is how much this really helps. If a disk fails before it's hit that throughput limit, the company might replace that disk, but in many cases the data on the disk will be worth much more than the hardware. Naturally, we all have backups, don't we? But even then, the downtime caused by a broken disk can be the real issue, especially close to a deadline or with a client in the room, and those factors are certainly not covered by the warranty on a hard disk.

RAID is not backup. A slip of the mouse can still delete the wrong file, no matter how reliable the disk is

Some might point out that buying low-cost disks beckons more failures, but a quick visual inspection suggests that many higher-end disks are physically more or less identical to their lower-end counterparts, and that the reason for the higher price is sometimes intended to cover the fact that the tighter specifications will simply result in more warranty claims for that model. In a huge datacentre, this might make sense, where failures among hundreds of drives might be common. If the data loss and the downtime is a bigger issue than the hardware, and among less than a dozen disks, there's a question mark over how worthwhile it is to pay for what's basically an extended warranty. That's not the case with all higher-end disks, but sometimes it is.

Much of what was said five years ago about specifying storage for film and TV work remains valid. Particularly, it's a good idea to be a little cautious about advice coming from people used to setting up server farms or storage for conventional corporate environments, which may use roughly same sort of hardware but which have very different goals and requirements and do things very differently. When solid state storage is cheap enough to entirely displace hard disks, reliability and speed are likely to be so high that many of these concerns will become more or less irrelevant. As it is, traditional hard disks have enjoyed more development than many commentators expected, so they're likely to be with us for a while longer, and we should know how to get the best out of them.

Title image courtesy of Shutterstock.