How do filmmakers deal with growing amounts of data?

A LaCie 12TB RAID - not scratching the surface

Data, data, everywhere: Guest Author Andrew Johnstone on the growing problem of data storage for filmmakers.

I admit to feeling pretty pleased with myself. I have just delivered a new film to one of my main broadcast clients and my series editor was as happy as a kid in a sandpit. With a happy client and the prospect of more work, we all feel pretty epic. But now comes the tricky bit – what to do with all the data?

Earlier this year I spent a decent amount of cash on a new LaCie 12TB Raid for archiving rushes and a new G-Tech 3Tb Thunderbolt drive as my working project drive. On top of these two new workhorses, I have a 6TB Lacie USB3 Raid as old project storage, an older 3TB Firewire scratch disk, various portable drives for taking out on assignment and several older 1TB drives that I have now decommissioned. I had hoped that this might be the end of it for some time, but alas I was wrong – for various reasons.

I was wrong because I’ve been busy working and I now have a lot of content to store. I was wrong because while I am only shooting HD at the moment, 4k is round the corner and then my data storage requirements will double. I was wrong because while I am running out of digital storage space fast in my office, I am also not dealing with the issue of how to create a further backup of all my data offsite, in case of flood, fire or Martian attack. I was also wrong because I had believed naively that my various storage solutions would all play nicely together, which they don’t and I am in serious need of some sort of switch that will allow me to interface Thunderbolt with USB3.0 and Firewire and thereby transfer my data between devices with ease. Something like the Belkin Thunderbolt hub might be just the ticket – well provided I can afford it.

And this is actually the real problem here. As an independent filmmaker, none of my clients actually pays me to archive rushes, render files and all manner of data on my system. The cost of storing all this data comes down to me. If I asked clients for a contribution towards this service, they would probably cough up their cornflakes, but I know very well that the minute I choose to delete any content, they’ll come back and ask cheerily: “Do you remember Shot X? Could you just dig it out?”

Deleting old content is one solution to this data storage problem, but I am not convinced it is the right one. Amongst all the dross there will be shots that are genuinely worth keeping and if I ever need to recreate an edit, I would need all the other material to go with it.

There is an argument for simply archiving the shots you have used or partly used (a function that Final Cut Pro can do for you), but this is not ideal either because often, as any camera person will tell you, the director leaves some of the best material ‘on the cutting room floor’, so any project that is produced without, say, a skateboarding penguin shot will see the unused images deleted. This is not ideal when, a few days, weeks or months later you are commissioned to produce a new piece on the “Secret Life of Penguins” and the pictures of the amusing antics of the amiable Antarctic avifauna are gone.

For those of us who do not inhabit this data intense world, cloud storage is a very decent solution. It provides not only data storage, but crucially off-site data storage. It comes with other issues, such as the data security issues inherent in the cloud, but its premise is tantalising. However, cloud storage requires superfast broadband solutions which not every one has access to, I have calculated that to store 1TB of data in the cloud on my current broadband upload connection speed of 0.33mbps will take me 3 Days, 0 Hours, and 22 Minutes, 41 Seconds. And even if you can access the cloud, storage is not cheap. 1TB of data stored with Google will cost you £79/year, according to Which. Of course we all expect these costs to decrease, but in the meantime if you want to store 10TB of data, you’ll be paying £790 per year while they do.

There are cloud services that have been specifically tailored for film and television such as A-Frame, but these are aimed at large producers and broadcasters, not at the independent filmmaker.

A NAS storage solution is not hideously expensive. There are NAS storage devices that start at around £350, but there is the issue of longevity to consider, and even though most NAS drives offer hot-swappable replacement disks, these still need to be bought & paid for. Red Shark has already flagged one innovation in data storage that addresses the longevity issue. Norwegian company piql has developed a special film that offers, ironically, to store data for 500 years, but it is not clear how much this will cost or that this service is really aimed at small businesses.

Although a NAS drive it affordable, it does not resolve the issue of how to ensure that a back up of everything is stored off-site when you work from a home office. So what to do? I recall a conversation with a photographer friend who told me that at the end of play each day he would walk to the shed at the end of the garden with a USB back-up drive and lock it away overnight. Perhaps this really is the only safe and economical solution?