Replay: The technological revolution has created great opportunities for new film-makers everywhere, but has in its wake created a new challenge: where are we going to put all this stuff? Peter J Haas investigates An Affordable Middle Ground To Preserving Your Film’s Digital Assets.
We’re not talking about just making backups of your data here. Backing up implies a live / active copy of your materials to use just in case something happens with the copy you’re working on. What we’re going to discuss here is archiving, the long term storage of our materials.
We all know that digital archiving is a problem. Mediums tend to be unstable over time, formats evolve and change (ask anyone who has a pile of floppy or zip discs sitting around) and, some software manufacturers will go out of business (I still miss WordStar) and others will end of life your documents and project files (I’m looking at you Apple).
Complicating the situation is the exponential increase in the amount of data we are creating, particularly in digital cinema. We’ve gone from the days of DVCAM where 13GB would hold somewhere around an hour, to formats such as CinemaDNG where around three minutes would occupy roughly the same amount of space.
This is not to complain about these formats. Implementing cameras like the Black Magic Pocket, Cinema Camera and Digital Bolex has allowed small companies and truly independent film-makers (like the company I work for, Left Turn Productions) to produce works on par with that of much larger, fully funded studios. Yet, even with advances in digital cinema and computer technology, the bigger, less glamorous question still remains: “what do I do with my materials once I’m finished with my film?”
There is no simple magic bullet for solving the digital dilemma. In their aptly titled reports The Digital Dilemma Parts I and II the Academy of Motion Picture Sciences outlines the fact that even large Hollywood Productions are hitting a wall when trying to solve the same problems as the little guys. Even masters of the digital film domain such as George Lucas (who is arguably most responsible for pushing the industry towards digital for the last thirty or so years) has claimed on multiple occasions that the solutions aren’t there yet, but the issues of digital archiving will be solved by “the market” at “some point” in the future.
More importantly, as the Digital Dilemma points out, most film-makers are not technology people and are more concerned with “moving onto the next project,” than trying to find a complicated solution to seemingly overwhelming problem. This is worrisome, as there is an entire generation of digital media that is in severe danger of being lost.
This raises the next questions: “Why bother archiving?”, “Is there any reason to save our master materials beyond the final product?”, “As media creators, what are our obligations to the materials we are creating?”, and “Who am I even archiving this for anyway?”
It’s impossible to predict the cultural significance of the work that we are doing while we are working on it. Every recording that you make is a slice of history. Regardless of the content, be it documentary or fiction, the work that you are doing is representative of something at a particular period in time.
Outside the scope of greater society, you’re spending a great deal of time, and of course money, on creating something. At the very least, you should be concerned enough to preserve these materials for yourself! We want to treat our projects the best we can. We go through great effort to make them look great, and we want to maintain that throughout the process. Yet, after doing some extensive research, I’ve found that doing right by your film doesn’t always come down to strictly maintaining the highest numbers.
Taking a look at what’s available on the current market there are generally three routes you can take towards archiving your film: hard discs, optical (Blu-ray), and tape (LTO). Each format has it’s own series of pros and cons, making them the logical choice for different scenarios.
What we’re going to be looking at here is a cross section of reliability, convenience , and cost. An ideal archiving solution should be easy to use, require minimal up keep over the course of a decade, and still show some promise of being compatible in the future.
Archiving to hard discs is relatively affordable, but creates an issue of exponential growth. The more you shoot and edit, the more drives you’re going to need. Eventually having a pile of discs sitting around on your desktop is going to start eating up more space than you or your office-mates are comfortable with dealing with. Since hard discs only have a life-span of around three to five years you will find yourself migrating to new discs more frequently than other mediums.
A good, albeit more expensive, solution in this case would be a large RAID mirrored disc array such as a DROBO, LACIE BIG DISK, or TK, which provides large storage capacity while providing a “smart” backup system. With these RAID setups, it becomes easy to replace a disc that is full or failing with a new drive. Simply eject the drive from the chassis, insert the new drive, and the RAID knows what data to repopulate the disc with.
When attached to a network, a RAID solution can act like a slower, less elaborate version of the more corporate shared storage solutions from Avid or EditShare. This makes archiving to hard disc great for venues that produce a lot of television or programming that involves quickly accessing materials for reuse.
If you’re working on mostly one-off projects, you might not need (or be able to depending on your contracts) to access all previous materials from past projects. Keeping it active on hard discs that are constantly spinning costs money in the form of your utility bill, and might not be worth the long term cost. It also adds significantly to your carbon footprint!
Additionally, after a few projects you’re going to find you’ll max out your RAID and need to start another chassis. Soon you’ll have a collection of RAIDs, piling up the same way those external drives had been and you’ll begin feeling like you’re running a datacenter, which honestly, as a film-maker, was neither my intention nor an idea I’m keen on.
Pros
Cons
There was an article on RedShark a while back about tape being the current, and future golden standard for archiving (Tape has the last laugh: Sony breakthrough squeezes 180 TB onto a single tape). Frankly, Phil is correct. Talk to any IT professional and they’ll tell you that the most secure home for your data is going to be LTO / tape drives. It’s a long-standing institution for storing digital information with lots of infrastructure to support it.
While LTO is great for large businesses, well funded start-ups and people with access to the funds, it’s not great for indies on a budget. The great limiting factor around LTO is the cost-structure of the hardware and future migration. Although you can purchase the most recent iterations of LTO drives for around $3,000, to get a complete archiving packages that include tape drives, cables, software, hardware adapters and cards you can expect to pay upwards of $6,000 to $10,000. On top of this you’re paying around $45 for every 3TB tape cartridge (which is actually a really great deal if you break down the cost per terabyte!).
Additionally, if we’re talking really long term, current LTO 6 devices have the additionally ability to read LTO 4, and read/write LTO 5. This means that your tape drives are going to have some staying power.
Pros
Cons
I’m going to be upfront, LTO would be my ideal choice for archiving my films... if I had the budget for it. The startup costs for setting up an LTO storage system makes it outside the budget of most indie films and small companies.
When I start talking about Blu-ray I see a lot of people’s eyes glaze over. I’m not sure if it’s because of the contentious battle waged between HD-DVD and Blu-ray or the fact that most people associate it with movie delivery. I think Blu-ray is great. It’s affordable, burners and readers are commonplace, media is plentiful, and professional quality software can be found for around $60.
It’s more likely that this will be around for a while. Look at the fact that I can still use CD, DVD and Blu-ray discs in my current drives. Optical formats are more prevalent in popular consumer electronics. It’s more likely that as optical technologies evolve we’re going to be able to continue using our old discs. This might not always be the case, but if the current state of optical compatibility is a signal for the future (which it isn’t always, see why this is so frustrating?) Then a format like Blu-ray is a great option for archiving.
Like anything, there are downsides to using Blu-ray. The base discs only hold 25GB, and although there are currently 50GB, 100GB and a new 300GB model coming soon, the larger the discs the more expensive they become. I frequently find myself spanning a day’s shoot over several discs. When you want data to be accessed long term, you typically need to burn the discs at a low speed, making them slow to write. Disc life is uncertain, but from personally experience I was recently able to access my burned CD-R “mix-tapes” from the early 2000s along with all my college papers (that I had migrated from floppies), so I’m feeling pretty confident in the life span of optical media if it’s treated well.
Pros
Cons
Okay, so there is a fourth option, one that I’m really hesitant about talking about here, but let’s go ahead and jump into cloud computing. I’m not going to spend much time talking about this, and you’ll see why. I’ve seen that there are a number of really great editing-in-the-cloud products coming out from various companies both big and small. I’m still not convinced that the future of archiving large files is in the cloud, and I defiantly know we aren’t there at the current time.
First off, ISP service, especially here in the States, is terrible: slow downloads and even slower uploads. With new laws and corporate mergers on the horizon that are going to restrict our internet access even further, baring any major changes in policy, I don’t foresee the internet being a satisfactory or convenient place to storage our digital materials. The price point is unacceptable; a monthly fee for accessing the internet, an additional fee for renting a modem from the only internet company in town, and then having to pay to host the kind of space you’re going to need to store all the materials. Additionally, there are all sorts of privacy concerns!
So, unless something seriously changes over the next five or so years that eliminates all the restrictions currently imposed upon the “cloud,” online storage isn’t going to be a logical solution to the “Digital Dilemma.”
David recently wrote a fantastic article about using all the bandwidth in a format such as HD [How to make HD look like 4K], and how the results can still be stunning. As an example, recently our documentary UNDER THE BUS was screened using a 2K DCP. The film was shot on a (unhacked) GH2 and I was blown away by the quality of the DCP on the huge screen. The image was sharp yet still maintained that creamy cinema magic.
What this experience told me fell very much inline with what David was talking about. We had amazing results with this little AVCHD camera format. After doing some research which involved investigating broadcast standards for the BBC, PBS, Al Jazeera and DCP requirements, I’ve come to a conclusion that might make me sound a little crazy:
Okay, well it’s okay in most cases. Ideally, we could all keep everything uncompressed all the time. Yet, when working as an indie film-maker, practical considerations such as budgetary constraints don’t always allow for it. You can’t always have your cake and eat it too. The trick here is to find a format that fits in with your delivery requirements as well as all the practical considerations.
I’ve found that there are two formats that fit really well into that middle ground: Apple ProRes and DnxHD115. They both meet the AVCHD Intra 100 data-rate requirements for most broadcast deliverables and look fantastic projected, even on large theatre screens. Both of these formats have been around for a while and seem as though they will be around in the foreseeable future.
We’ve integrated converting our materials to DNxHD115 (or ProRes for our 2K productions) into our workflow and after three productions of smooth sailing, I’ve confident that we’ve found the correct solution for our company. We might still be burning a spindle or two of Blu-rays to save everything, but because we’ve found a good middle ground we’re not losing weeks to the process of archiving, nor are we sacrificing much quality-wise.
Even after coming to a decision about our current workflows there is always the nagging thought about how things are going to change in the future. Technology is rapidly expanding and changing, but that doesn’t necessarily mean that one should be distracted by it. As indie film-makers we find ourselves having to make more immediate and practical decisions than the large studios. We can’t be waiting around tending to mountains of CinemaDNG filled drives, hoping that the market will magically solve the problem one day. We need simple, straightforward, and affordable solutions to these problems so we can focus on what we should be doing: making movies.